3dDeconvolve hangs?

DeLab · April 1, 2021, 2:30am

Running a script generated by afni_proc.py called proc.s3. Goes up to 3dDeconvolve, runs 3dDeconvolve, in about 4 hrs generates the stats.s3 file, then appears to hang. The 3dDeconvolve process is running 6 hrs after saving the stats file but nothing seems to be happening no new files being generated in 6 hours. The last few lines of output.proc.s3 file are

…
++ Smallest FDR q [74 LP+LN+MP+MN-2AP-2AN_GLT#0_Tstat] = 6.6892e-08
++ Smallest FDR q [75 LP+LN+MP+MN-2AP-2AN_GLT_Fstat] = 6.68918e-08
++ Wrote bucket dataset into ./stats.s3+tlrc.BRIK

created 51 FDR curves in bucket header

The fitts and errts files are not generated and appears to stall. The 3dDecon command in proc.s3 file is

------------------------------

run the regression analysis

3dDeconvolve -input pb04.$subj.r*.scale+tlrc.HEAD
-censor censor_${subj}_combined_2.1D
-ortvec ROI.CSFe_rall.1D ROI.CSFe
-ortvec mot_demean.r01.1D mot_demean_r01
-ortvec mot_demean.r02.1D mot_demean_r02
-ortvec mot_demean.r03.1D mot_demean_r03
-ortvec mot_demean.r04.1D mot_demean_r04
-ortvec mot_demean.r05.1D mot_demean_r05
-ortvec mot_demean.r06.1D mot_demean_r06
-ortvec mot_demean.r07.1D mot_demean_r07
-ortvec mot_demean.r08.1D mot_demean_r08
-ortvec mot_demean.r09.1D mot_demean_r09
-polort 3
-num_stimts 8
-stim_times 1 stimuli/abstract_negative_timing.1D ‘SPMG1(2.5)’
-stim_label 1 abstract_negative_timing
-stim_times 2 stimuli/abstract_positive_timing.1D ‘SPMG1(2.5)’
-stim_label 2 abstract_positive_timing
-stim_times 3 stimuli/filler_negative_timing.1D ‘SPMG1(2.5)’
-stim_label 3 filler_negative_timing
-stim_times 4 stimuli/filler_positive_timing.1D ‘SPMG1(2.5)’
-stim_label 4 filler_positive_timing
-stim_times 5 stimuli/literal_negative_timing.1D ‘SPMG1(2.5)’
-stim_label 5 literal_negative_timing
-stim_times 6 stimuli/literal_positive_timing.1D ‘SPMG1(2.5)’
-stim_label 6 literal_positive_timing
-stim_times 7 stimuli/metaphor_negative_timing.1D ‘SPMG1(2.5)’
-stim_label 7 metaphor_negative_timing
-stim_times 8 stimuli/metaphor_positive_timing.1D ‘SPMG1(2.5)’
-stim_label 8 metaphor_positive_timing
-jobs 4
-num_glt 17
-gltsym ‘SYM: abstract_positive_timing -abstract_negative_timing’
-glt_label 1 AP-AN
-gltsym ‘SYM: literal_positive_timing -literal_negative_timing’
-glt_label 2 LP-LN
-gltsym ‘SYM: metaphor_positive_timing -metaphor_negative_timing’
-glt_label 3 MP-MN
-gltsym ‘SYM: literal_positive_timing -abstract_positive_timing’
-glt_label 4 LP-AP
-gltsym ‘SYM: metaphor_positive_timing -abstract_positive_timing’
-glt_label 5 MP-AP
-gltsym ‘SYM: literal_negative_timing -abstract_negative_timing’
-glt_label 6 LN-AN
-gltsym ‘SYM: metaphor_negative_timing -abstract_negative_timing’
-glt_label 7 MN-AN
-gltsym ‘SYM: literal_positive_timing -metaphor_positive_timing’
-glt_label 8 LP-MP
-gltsym ‘SYM: literal_negative_timing -metaphor_negative_timing’
-glt_label 9 LN-MN
-gltsym ‘SYM: literal_positive_timing +metaphor_positive_timing
+abstract_positive_timing -literal_negative_timing
-metaphor_negative_timing -abstract_negative_timing’
-glt_label 10 LP+MP+AP-LN-MN-AN
-gltsym ‘SYM: literal_positive_timing -literal_negative_timing
-abstract_positive_timing +abstract_negative_timing’
-glt_label 11 LP-LN-AP+AN
-gltsym ‘SYM: metaphor_positive_timing -metaphor_negative_timing
-abstract_positive_timing +abstract_negative_timing’
-glt_label 12 MP-MN-AP+AN
-gltsym ‘SYM: literal_positive_timing -literal_negative_timing
+metaphor_positive_timing -metaphor_negative_timing
-2abstract_positive_timing +2abstract_negative_timing’
-glt_label 13 LP-LN+MP-MN-2AP+2AN
-gltsym ‘SYM: literal_positive_timing -literal_negative_timing
-metaphor_positive_timing +metaphor_negative_timing’
-glt_label 14 LP-LN-MP+MN
-gltsym ‘SYM: literal_positive_timing +literal_negative_timing
-abstract_positive_timing -abstract_negative_timing’
-glt_label 15 LP+LN-AP-AN
-gltsym ‘SYM: metaphor_positive_timing +metaphor_negative_timing
-abstract_positive_timing -abstract_negative_timing’
-glt_label 16 MP+MN-AP-AN
-gltsym ‘SYM: literal_positive_timing +literal_negative_timing
+metaphor_positive_timing +metaphor_negative_timing
-2abstract_positive_timing -2abstract_negative_timing’
-glt_label 17 LP+LN+MP+MN-2AP-2AN
-fout -tout -x1D X.xmat.1D -xjpeg X.jpg
-x1D_uncensored X.nocensor.xmat.1D
-fitts fitts.$subj
-errts errts.${subj}
-bucket stats.$subj

3dDecon process itself is running without taking any CPU resources. There is no error, no flag, and it is not out of disk space. Afni version is the latest from complied on March 24 ‘Titus’. Running on OSX Catalina 10.15.7. Thanks!

DeLab · April 1, 2021, 12:53pm

Sorry one correction. I am running OSX version 10.13.6, not 15.7

rhdesai% sw_vers
ProductName: Mac OS X
ProductVersion: 10.13.6
BuildVersion: 17G14042
rhdesai% afni -ver
Precompiled binary macos_10.12_local: Mar 24 2021 (Version AFNI_21.0.21 ‘Titus’)

rickr · April 1, 2021, 3:50pm

I must expect that you are running out of RAM.

The next operation would be to actually output smallest FDR q-values for each t-/F-stat volume.

Consider adding -regress_compute_fitts to the AP command, which will save RAM in 3dDeconvolve by computing the fitts dataset via 3dcalc rather than via 3dDeconvolve (which uses the same RAM as for the linear regression).

rick

DeLab · April 2, 2021, 5:33pm

Thanks. For now I am using -regress_no_fitts and that seems to be working. Is there a similar option for -regress_no_errts if you don’t want to add -errts to 3dDecon? Could not find it in afni_proc.py help.

Now there is another strange problem. This may be a minor silly scripting thing, not sure. Most of the proc.s1 script seems to have run correctly, and all the corresponding files are generated. Then it never seems to finish. Last files were generated 14 hrs ago, but the script is still running for some reason. The last few lines of the proc.s1 script are

if the basic subject review script is here, run it

(want this to be the last text output)

if ( -e @ss_review_basic ) then
./@ss_review_basic |& tee out.ss_review.$subj.txt

# generate html ss review pages
# (akin to static images from running @ss_review_driver)
apqc_make_tcsh.py -review_style basic -subj_dir . \
    -uvar_json out.ss_review_uvars.json
tcsh @ss_review_html |& tee out.review_html
apqc_make_html.py -qc_dir QC_$subj

echo "\nconsider running: \n\n    afni_open -b $subj.results/QC_$subj/index.html\n"

endif

return to parent directory (just in case…)

cd …

echo “execution finished: `date`”

All of the commands seem to have run. The @ss_review_basic is run and there is output in the output.proc.s1 file. There are out.review_html and out.ss_review_uvars.json files generated 14 hrs ago. The QC_s1 folder was also generated, and there are files inside indicating apqc_make_html.py ran fine. But the “consider running:…” message is never printed. “execution finished” is also never printed. The last few lines of output.proc.s1 are

TSNR average : 206.017
global correlation (GCOR) : 0.0374245
anat/EPI mask Dice coef : 0.909886
anat/templ mask Dice coef : 0.953539
maximum F-stat (masked) : 216.554
blur estimates (ACF) : 0.644382 5.29497 13.4468
blur estimates (FWHM) : 0 0 0

which as printed 14 hrs ago. After this, nothing.

‘ps’ still shows that proc.s1 is still running, ‘tee out.review_html’ is still running, doing what I am not sure. Any thoughts on why would the script not finish?

rickr · April 9, 2021, 9:19pm

Sorry, there is no -regress_no_errts option. Such an option would not be advisable though, since the errts dataset is used for things like TSNR, blur estimates and global correlation.

The hung process is a little odd. It suggests that @ss_review_html is stuck. You could terminate the process if you want to (even ctrl-c), and try re-running those last 2 lines, just to see. I expect you did that already.

tcsh @ss_review_html |& tee out.review_html
apqc_make_html.py -qc_dir QC_$subj

Is the file system full or something?

df -h .

rick