afni_proc.py Killed after tedana: Computing T2* map

taliwe · August 2, 2025, 6:11am

linux_ubuntu_16_64, AFNI_25.2.06

I have 2 scans, SHORT scan (~480 TRs), and LONG scan (~730 TRs).
The scan parameters are idetical, just the number of TR differ.
For all participants, afni_proc for the SHORT is working great, but the LONG is Killed after tedana:tedana_workflow:609 Computing T2* map
The next step should be:
combine:make_optcom:242 Optimally combining data with voxel-wise T2* estimates

afni_proc.py -subj_id $f.visual.blur6 \
        -script $mypath/$f/afni/proc.$f.visual.blur6.MNI -scr_overwrite                  \
        -blocks despike tshift align tlrc volreg mask combine blur scale regress \
        -despike_opts_3dDes -localedit   \
        -tshift_interp             -wsinc9                               \
        -radial_correlate_blocks tcat volreg   \
        -copy_anat $mypath/$f/afni/anatSS.T1W.nii                                \
        -anat_follower  anat_w_skull anat $mypath/$f/afni/t1+orig      \
        -anat_has_skull no                                       \
        -dsets_me_echo $mypath/$f/epi_run2_visual_echo_001+orig.HEAD       \
        -dsets_me_echo $mypath/$f/epi_run2_visual_echo_002+orig.HEAD       \
        -dsets_me_echo $mypath/$f/epi_run2_visual_echo_003+orig.HEAD       \
        -dsets_me_echo $mypath/$f/epi_run2_visual_echo_004+orig.HEAD       \
        -tcat_remove_first_trs   4   \
        -echo_times 10.6 22.92 35.24 47.56                 \
        -reg_echo 2                                              \
        -mask_epi_anat yes                           \
        -align_opts_aea -giant_move        \
        -align_opts_aea -cost lpc+ZZ -check_flip           \
        -volreg_align_to MIN_OUTLIER -volreg_align_e2a           \
        -volreg_warp_final_interp wsinc5           \
        -volreg_tlrc_warp                                        \
        -blur_to_fwhm -blur_size 6                               \
        -blur_in_mask yes                                        \
        -tlrc_base MNI152_2009_template_SSW.nii.gz               \
        -tlrc_NL_warp     \
        -tlrc_NL_warped_dsets $mypath/$f/afni/anatQQ.T1W.nii $mypath/$f/afni/anatQQ.T1W.aff12.1D $mypath/$f/afni/anatQQ.T1W_WARP.nii \
        -combine_method 'm_tedana'                               \
        -combine_opts_tedana --fittype curvefit --tedpca kic --lowmem --n-threads 4 \
        -test_stim_files no   \
        -regress_stim_times $prtpath/$f.visual.txt  \
        -regress_stim_labels odor                        \
        -regress_basis        'BLOCK(2,1)'        \
        -regress_opts_3dD     -jobs 4                          \
                              -gltsym 'SYM: odor'          \
                              -glt_label 1 odor                    \
        -regress_motion_per_run                                         \
        -regress_apply_mot_types   demean deriv                         \
        -regress_censor_motion    0.5                                   \
        -regress_censor_outliers  0.05                                   \
        -regress_3dD_stop                                               \
        -regress_reml_exec                                              \
        -regress_compute_fitts                                          \
        -regress_make_ideal_sum   sum_ideal.1D                          \
        -regress_est_blur_epits                                         \
        -regress_est_blur_errts                                         \
        -regress_run_clustsim     no                                    \
        -html_review_style pythonic
                    
tcsh -xef $mypath/$f/afni/proc.$f.visual.blur6.MNI 2>&1 | tee $mypath/$f/afni/output.proc.$f.visual.blur6.MNI

ptaylor · August 2, 2025, 12:37pm

Hi-

The term Killed is actually a technical one for computations. It means your operating system ran out of memory, and terminated the process to not freeze up.

So, it sounds like the tedana processing step is running out of memory. What is the RAM on that computer? I'm not sure if there are tricks to help with that (closing other running programs---Chrome is generally a memory hog, for example?). Maybe @handwerkerd would have a suggestion?

--pt

taliwe · August 2, 2025, 1:18pm

Thank you!

#
              total        used        free      shared  buff/cache   available
Mem:           62Gi       1.9Gi        59Gi       584Mi       1.4Gi        59Gi
Swap:         2.0Gi       878Mi       1.1Gi

ptaylor · August 3, 2025, 11:28am

Howdy-

So, "total" is 62Gi, which is sizeable, while "used" (that which is used by your system) is pretty small and only 1.9Gi. That is a big difference between the numbers. Is that a virtual machine or something like that?

--pt

ptaylor · August 3, 2025, 11:32am

Well, I guess the output of "free" has changed across different versions, so perhaps having a smaller "used" value isn't so abnormal. What OS are your running on?

--pt

ptaylor · August 3, 2025, 11:34am

And I guess since it looks like your computer does have sizeable memory available, what is your FMRI data's voxel size and number of time points in the two cases?

--pt

handwerkerd · August 3, 2025, 1:33pm

What version of tedana are you using? We identified and fixed some unnecessary memory usage in this step, in version 23.0.2 (released November 2023). If you're using an older version, I'd update to the newest version and see if that corrects this problem.

You're also using --fittype curvefit which should be computationally more intensive, but I don't think it would use significantly more memory. By the time Computing T2* map appears in the log, the fMRI data are already loaded into memory.

Instead of rerunning the entire afni_proc statement, maybe you can just run the call to tedana from the script that afni_procoutputs to confirm this is crashing there. Then it would also be easier to compare total memory usage for the short and long datasets within just the tedana step.

Dan

taliwe · August 4, 2025, 12:34pm

linux_ubuntu_16_64, not virtual machine.
isovoxel 2.4 mm, TR= 1.5 sec, 700 time points.

I updated to tedana v24.0.2, now I get this error:

/home/taliw/.local/lib/python3.8/site-packages/nilearn/_utils/niimg_conversions.py:325: UserWarning: Data array used to create a new image contains 64-bit ints. This is likely due to creating the array with numpy and passing `int` as the `dtype`. Many tools such as FSL and SPM cannot deal with int64 in Nifti images, so for compatibility the data has been converted to int32.
  niimg = new_img_like(niimg, data, niimg.affine)
Traceback (most recent call last):
  File "/home/taliw/.local/bin/tedana", line 8, in <module>
    sys.exit(_main())
  File "/home/taliw/.local/lib/python3.8/site-packages/tedana/workflows/tedana.py", line 1077, in _main
    tedana_workflow(**kwargs, tedana_command=tedana_command)
  File "/home/taliw/.local/lib/python3.8/site-packages/tedana/workflows/tedana.py", line 1029, in tedana_workflow
    reporting.static_figures.plot_adaptive_mask(
  File "/home/taliw/.local/lib/python3.8/site-packages/tedana/reporting/static_figures.py", line 760, in plot_adaptive_mask
    discrete_cmap = cmap.resampled(3)  # colors matching the mask lines in the image
AttributeError: 'LinearSegmentedColormap' object has no attribute 'resampled'

handwerkerd · August 4, 2025, 1:24pm

We recently set the minimum version of nilearn to 0.10.3 because they were changing options and we were seeing issues with older versions (see: minimum nilearn to 0.10.3 by handwerkerd · Pull Request #1094 · ME-ICA/tedana · GitHub ). If you're using an older version of nilearn, could you try upgrading?

FWIW, we also recently dropped support for python 3.8 because Python designated it as no longer supported (see: Status of Python versions ) and we started to see hints of issues developing. I'm not sure this is a specific cause of the above problem, but it might be worth switching to a newer version of python. Newer python versions should also be better with processing efficiency and memory management.

Dan

taliwe · August 23, 2025, 8:40am

Hi Dan,

I order and install 4 new DDR (4*32 Gi, 128 Gi in total).

linux_ubuntu_16_64: Version AFNI_25.2.06 'Gordian I'
Python 3.8.10
nilearn Version: 0.10.4

I get the same error...

/home/taliw/.local/lib/python3.8/site-packages/nilearn/_utils/niimg_conversions.py:325: UserWarning: Data array used to create a new image contains 64-bit ints. This is likely due to creating the array with numpy and passing `int` as the `dtype`. Many tools such as FSL and SPM cannot deal with int64 in Nifti images, so for compatibility the data has been converted to int32.
  niimg = new_img_like(niimg, data, niimg.affine)
Traceback (most recent call last):
  File "/home/taliw/.local/bin/tedana", line 8, in <module>
    sys.exit(_main())
  File "/home/taliw/.local/lib/python3.8/site-packages/tedana/workflows/tedana.py", line 1077, in _main
    tedana_workflow(**kwargs, tedana_command=tedana_command)
  File "/home/taliw/.local/lib/python3.8/site-packages/tedana/workflows/tedana.py", line 1029, in tedana_workflow
    reporting.static_figures.plot_adaptive_mask(
  File "/home/taliw/.local/lib/python3.8/site-packages/tedana/reporting/static_figures.py", line 760, in plot_adaptive_mask
    discrete_cmap = cmap.resampled(3)  # colors matching the mask lines in the image
AttributeError: 'LinearSegmentedColormap' object has no attribute 'resampled'

taliwe · August 25, 2025, 5:19pm

[quote="taliwe, post:10, topic:8984"]

 'LinearSegmentedColormap' object has no attribute 'resampled'

I updated Matplotlib version, now it is working

ptaylor · August 25, 2025, 5:40pm

Fascinating! Do you happen to know both the prior and current Matplotlib versions, in case other folks have an issue with this?

thanks,
pt