Connection to 3dGroupInCorr is broken

AFNI Message Board
1

Hi! After talking to Rick we decided that I should post this.

Some of our researchers are using 3dGroupInCorr and the AFNI viewer for trying to optimize DMN seed locations. They have experienced some issues with 3dGroupInCorr closing the connection randomly. This can occur after just 1 click or 20 clicks and might happen 1-2 times a day. AFNI is installed on a remote server and accessed via ssh -X. This has, to our knowlage, never happend on a local AFNI session. This is the terminal output:


afni -NIML &

 3dGroupInCorr -setA ALL.grpincorr.niml

++ 3dGroupInCorr: AFNI version=AFNI_16.3.04 (Oct 19 2016) [64-bit]

++ Authored by: Cox, the Mad Correlator

++ GIC: data file ALL.grpincorr.data found with 11,413,107,472 bytes of data

++ GIC: -setA opened, contains 46 datasets, 862408 time series, 11,413,107,472 bytes

++ GIC: page faulting (reading) data into memory

 + setA:0123456789.0123456789.0123456789.0123456789.0123456789.!

++ GIC: total .data bytes input = 11,413,107,472 (about 11 billion [giga])

++ GIC: --- Be sure to start AFNI with the '-niml' command line option

 +      ---  [or press the NIML+PO button if you forgot '-niml']

++ Opening NIML socket 'tcp:localhost:53213' to AFNI. Connected!

++ NIML connection opened from 127.0.0.1 (tcp:host:53213,AFNI_GroupInCorr_NIML)

++ 3dGroupInCorr stands ready to do thy bidding, O Master :-) !!

 + GIC: OpenMP thread count = 15

++ Added 3dGroupInCorr dataset 'A_GRP_ICORR+tlrc' to controller [A]

 + 46 datasets in set A

 + ----------------------------------------------------

 + ----- AFNI is now connnected to 3dGroupInCorr! -----

 + ..... Use 'InstaCorr Set' to pick a seed voxel .....

 + ..... (Mouse-right-click menu in image viewer) .....

 + ----------------------------------------------------

++ AFNI GIC: AFNI_gicor_setref called --> ijk=538613

++ AFNI GIC: sending command to 3dGroupInCorr

++ AFNI GIC: called from xyz = 0.00 56.00 29.00 (DICOM), seed radius 0

++ GIC: Received command SETREF_ijk from AFNI

 + GIC:  loaded seed vectors: elapsed=1 ms

 + GIC:  finished correlation-izing: elapsed=1198 ms

 + GIC:  finished t-test-izing: elapsed=258 ms

++ GIC: Reconnecting to AFNI with shared memory channel shm:GrpInCorr_53213:1M+4K

 + GIC:  SHM reconnection *ACTIVE* :-) !!!

 + GIC:  sent results to AFNI: elapsed=208 ms  bytes=6,899,382

 + GIC:  Total elapsed time = 1665 msec

++ Potentially more regions could be found than the 9 reported.

Set the environment variable AFNI_WHEREAMI_MAX_FIND to higher

than 9 if you desire a larger report.

It behooves you to also checkout AFNI_WHEREAMI_MAX_SEARCH_RAD

and AFNI_WHEREAMI_NO_WARN. See whereami -help for detail.

*+ WARNING: GIC: Connection to AFNI broken - trying to restart

reading MNIa_caez_pmaps_18+tlrc(328 million [mega] bytes).........done

++ NIML connection closed from shm:GrpInCorr_53213:1M+4K (shm:GrpInCorr_53213:1M+4K,ZERO)

 + GIC: TCP/IP restart is good :-)

++ NIML connection opened from 127.0.0.1 (tcp:host:53213,AFNI_GroupInCorr_NIML)

*+ WARNING: AFNI GIC: connection to 3dGroupInCorr is broken

It seems like it does re-establish the connection but the time between “+ GIC: TCP/IP restart is good :-)” and “*+ WARNING: AFNI GIC: connection to 3dGroupInCorr is broken” is probably very short.

Any insights?

Best,
Robin

2

Obviously, I’ve seen this before, which is why I programmed in the re-connection attempt.

What you are seeing is 2 different types of inter-process communication. First you get the messages

[b]Opening NIML socket ‘tcp:localhost:53213’ to AFNI. Connected!

++ NIML connection opened from 127.0.0.1 (tcp:host:53213,AFNI_GroupInCorr_NIML)[/b]

The first one is from 3dGIC, saying it is opening a connection to AFNI (think of it as making a telephone call – AFNI is listening, 3dGIC is calling). The second message is from AFNI, saying it just got an incoming connection.

Later you get the messages

[b]++ GIC: Reconnecting to AFNI with shared memory channel shm:GrpInCorr_53213:1M+4K

  • GIC: SHM reconnection ACTIVE smiling smiley !!![/b]

Which says that 3dGIC has dropped the TCP/IP connection in favor of a SHM (shared memory) connection – which is much faster, but should only work on the same computer.

At this point I’m confused, because you said this was a remote session.

Even later you get the messages

[b]*+ WARNING: GIC: Connection to AFNI broken - trying to restart

++ NIML connection closed from shm:GrpInCorr_53213:1M+4K (shm:GrpInCorr_53213:1M+4K,ZERO)

  • GIC: TCP/IP restart is good smiling smiley

++ NIML connection opened from 127.0.0.1 (tcp:host:53213,AFNI_GroupInCorr_NIML)[/b]

These indicate that the SHM connection went bad – I’ve certainly seen this happen before, and spent a great deal of time trying to figure out why – and gave up. At this point, the 3dGIC program re-opens the TCP/IP connection and will use that for the rest of the session.

If it continues to work for you, then I’d just ignore this problem. I suppose it is a little annoying, but in my experience it all happens pretty quickly and so has no real downside. If you want to avoid the annoyance, then use the -NOshm option with 3dGroupInCorr and then it will only use TCP/IP – which seems to be more reliable.

By the way, the reason TCP/IP is slower when two programs on the same computer are talking is that the data must pass through the operating system, so it gets touched by more “hands”. With shared memory, the data passes directly from one program to another, which is 3-4 times faster for large amounts of data – which is often the case when passing entire datasets around. But computers are pretty fast these days, so the extra overhead from using TCP/IP may not be noticeable for most users.

3

Thanks for the very detailed reply!

Yes, that is confusing… The used computer is a Windows computer that does not even have AFNI. It is running MobaXterm to connect to our redhat server with AFNI installed. How is it even possible to run an SHM connection there?

We will try witth the suggested flag (-NOshm)!

Thanks again.