3dLMEr creates main BRIK results output but segfaults before outputting residuals BRIK

mpasternak · October 4, 2024, 6:18pm

All runs well, with the program eventually outputting the typical "Smallest FDR q ...." messages. But these are unfortunately followed by:

 *** caught segfault ***
address 0x78f5e6d81da0, cause 'memory not mapped'

Traceback:
 1: write.c.AFNI(filename, dset = brk, label = label, space = space,     note = note, origin = origin, delta = delta, orient = orient,     idcode = idcode, defhead = defhead, verb = verb, maskinf = maskinf,     scale = scale, addFDR = addFDR, statsym = statsym, view = view,     com_hist = com_hist, type = type, TR = TR, overwrite = overwrite)
 2: write.AFNI(lop$resid, Stat[, , , (lop$NoBrick + 1):(lop$NoBrick +     (!is.null(lop$resid)) * nrow(lop$dataStr)), drop = FALSE],     label = 0:(dim(lop$dataStr)[1] - 1), defhead = head, idcode = newid.AFNI(),     com_hist = lop$com_history, type = "MRI_float", scale = FALSE)
An irrecoverable exception occurred. R is aborting now ...
Segmentation fault (core dumped)

I am running this on a 128GB machine and while there is a point near the start of the 3dMEr program where it reaches 110GB usage, it settles down to ~70GB of use as it processes the slices.

System check is as follows:

-------------------------------- general ---------------------------------
architecture:         64bit ELF
cpu type:             x86_64
system:               Linux
release:              6.8.0-45-generic
version:              #45~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Sep 11 15:25:05 UTC 2
distribution:         ubuntu 22.04 Jammy Jellyfish
number of CPUs:       24
apparent login shell: bash
shell RC file:        .bashrc (exists)

--------------------- AFNI and related program tests ---------------------
which afni           : /home/mpasternak/afni/abin/afni
afni version         : Precompiled binary linux_ubuntu_16_64: Feb 22 2024 
                     : AFNI_24.0.08 'Caracalla'
AFNI_version.txt     : AFNI_24.0.08, linux_ubuntu_16_64, Feb 22 2024, official
which python         : /usr/bin/python
python version       : 3.10.12
which R              : /usr/bin/R
R version            : R version 4.4.1 (x86_64-pc-linux-gnu)

instances of various programs found in PATH:
    afni    : 1   (/home/mpasternak/afni/abin/afni)
    R       : 1   (/usr/bin/R)
    python  : 1   (/usr/bin/python3.10)
    python2 : 1   (/usr/bin/python2.7)
    python3 : 1   (/usr/bin/python3.10)


testing ability to start various programs...
    afni                 : success
    suma                 : success
    3dSkullStrip         : success
    3dAllineate          : success
    3dRSFC               : success
    SurfMesh             : success
    3dClustSim           : success
    uber_subject.py      : success
    3dMVM                : success

------------------------ dependent program tests -------------------------
checking for dependent programs...

which tcsh           : /usr/bin/tcsh
which Xvfb           : /usr/bin/Xvfb

checking for R packages...
    rPkgsInstall -pkgs ALL -check : success

R RHOME : /usr/lib/R

------------------------------ python libs -------------------------------

++ module loaded: matplotlib.pyplot
   module file : /usr/lib/python3/dist-packages/matplotlib/pyplot.py
   matplotlib version : 3.5.1

++ module loaded: flask
   module file : /usr/lib/python3/dist-packages/flask/__init__.py
   flask version : 2.0.1

++ module loaded: flask_cors
   module file : /usr/lib/python3/dist-packages/flask_cors/__init__.py
   flask_cors version : 3.0.9

-------------------------------- env vars --------------------------------
PATH                       = /home/mpasternak/mrtrix3/bin:/home/mpasternak/.local/share/pnpm:/home/mpasternak/.pyenv/plugins/pyenv-virtualenv/shims:/home/mpasternak/.pyenv/shims:/home/mpasternak/.pyenv/bin:/home/mpasternak/.local/bin:/home/mpasternak/MATLAB/R2020b/bin:/home/mpasternak/freesurfer/bin:/home/mpasternak/ants/bin:/home/mpasternak/fsl/share/fsl/bin:/home/mpasternak/fsl/share/fsl/bin:/home/mpasternak/.cargo/bin:/home/mpasternak/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/usr/local/go/bin:/home/mpasternak/go/bin:/home/mpasternak/afni/abin

PYTHONPATH                 = 
R_LIBS                     = 
LD_LIBRARY_PATH            = 
DYLD_LIBRARY_PATH          = 
DYLD_FALLBACK_LIBRARY_PATH = 
CONDA_SHLVL                = 
CONDA_DEFAULT_ENV          = 

----------------------------- eval dot files -----------------------------

----------- AFNI $HOME files -----------

    .afnirc                   : found
    .sumarc                   : found
    .afni/help/all_progs.COMP : found

--------- shell startup files ----------

   -- good: .tcshrc seems to contain 'source .cshrc'
   -- considered operations: path, apsearch
   
   -- note: followers should not need edits, so edit flags should be 0
      (have 1 follower(s), which can be ignored)
   
   dot file test : want 1 modifications across 3 files:
   
      file             path  flatdir  apsearch        follower
      ---------------  ----  -------  --------        --------
      .cshrc           0     0        1               0     
      .tcshrc          0     0        0               1     
      .bashrc          0     0        0               0     
   
------------------------------ data checks -------------------------------
data dir : found AFNI_data6   under $HOME/afni (467851M Avail)
           top history: ... [gangc]: redo 0.BGA as 1.bga and update run.MSS.txt
data dir : found AFNI_demos   under $HOME/afni
           top history: ...ct 2020 [taylorp]: updated scripts under FATCAT_DEMO
data dir : found suma_demo    under $HOME/afni
           top history: ...s_New/data/Build_tmp on Mon Mar  4 11:56:45 EST 2013
data dir : found afni_handouts under $HOME/afni
atlas    : found TT_N27+tlrc  under /home/mpasternak/afni/abin

------------------------------ OS specific -------------------------------
which apt-get        : /usr/bin/apt-get
apt-get version      : apt 2.4.13 (amd64)

have Ubuntu system: ubuntu 22.04 Jammy Jellyfish
have Ubuntu afni  : Precompiled binary linux_ubuntu_16_64: Feb 22 2024 

=========================  summary, please fix:  =========================
*  just be aware: login shell 'bash', but our code examples use 'tcsh'
*  dot file test : want 1 modifications across 3 files:

Gang · October 5, 2024, 2:49am

The error message, *** caught segfault *** address 0x78f5e6d81da0, cause 'memory not mapped', typically indicates a memory access issue. This could mean the program is trying to access memory that isn't available or has run out of available memory. You may want to try running the analysis on a computer with more memory. However, if your goal is to output residuals to estimate local spatial relatedness for addressing multiple testing, consider exploring an alternative reporting approach, as discussed in this AFNI blog post.

Gang Chen

mpasternak · October 11, 2024, 10:43pm

Hi Gang,

Appreciate the link.

I sincerely doubt the issue is with a lack of memory, as this same error arose when I attempted the analysis on a machine with 256GB of memory, with the program never coming close to any threshold. Apparently, I am not the only one to have experienced this - fortunately the solution proposed in this thread did the trick.

Since this thread appears high on Google, I will briefly summarize the steps:

Split your input images, masks, etc. using 3dZcutup into subsets to process individually. As of October 2024, I observe that this memory issue occurs around 1200 NIFTI1 images that are each of float32 dtype and 7.5MB when not gzipped. For my purposes and with a personal workstation of 128GB of memory, doing a lower half and an upper half split for 1400 volumes was sufficient to avoid these memory issues. Your mileage may vary.
Process each split with 3dLMEr separately. Assign unique names to the statistical image and residuals output so that they don't get overwritten.
Recreate the intended statistics and residuals files by concatenating these splits along the Z-axis together using 3dZcat. You can sanity-check yourself if you happen to have a copy of the statistics file from an earlier run that failed to procure the residuals file due to this memory address-related error (i.e. check that all the voxels are essentially the same between the two cases using nibabel and numpy with Python, for example).

I can appreciate that the root cause of this may be very difficult to pinpoint - is it AFNI? R? Some shared library or dependency that has some imposed upper memory ceiling?

Given the emergence of more "big-data" endeavors, this issue may begin to slowly prop up more often and at large enough scan numbers even this z-axis split trick may begin to fail.

In the spirit of FOSS, let me know if there is a way I can help.

Gang · October 14, 2024, 2:18pm

Thanks for sharing your experience and solution. If memory limitations become a bottleneck due to the size of the input or the need to output residuals, breaking the analysis into smaller chunks or slices is indeed a practical solution. This approach has been effectively used for many years.

Gang Chen