afni_proc.py: (partial) rerun after modification of GLM

AFNI Message Board
1

AFNI version info (afni -ver):
Precompiled binary linux_openmp_64: Jun 29 2023 (Version AFNI_23.1.10 'Publius Helvius Pertinax')

Hi all,

I have been using afni_proc.py for a long time, and often I find myself wanting to change something in the regression model after having run the analysis the first time (e.g. adding some confound regressors). I have tried to edit the script that afni_proc.py outputs (proc_...) and comment out the sections of the processing that do not need to be rerun. However, there is still quite a bit of code in that script after the 3dDeconvolve command, and some commands need files that have been removed after running the original script (e.g. "rm.*" files) or some files that have been "covertly" produced by the preceding script lines, so the task is not so trivial.

Since this is arguably a very common situation (i.e. revising the regression model), I wonder whether there is some more streamlined way to update an analysis without having to rerun the full modified afni_proc.py script. Furthermore, if one wants to compare the results of the two models in a group analysis, it is very resource expensive (CPU time and disk space) to rerun the full processing pipeline for every subject. Ideally, it would be sufficient for each subject to add the new stats.* datasets for the new analysis to the original directory, to be then used as input for the new group analysis.

As an example, I am listing below two afni_proc.py scripts: the original one I used, and the modified one that I would like to run.

Thanks for any suggestions!

PS: the AFNI site seems unreachable since yesterday, is it a known issue?

ORIGINAL
#!/bin/tcsh

# This is a customised version of the afni_proc.py
# to be run within the selected subject's directory
#
# The nonlinear warping of the T1 is supposed to have
# been done previously (see @warp_t1_to_mni)

set topdir = `echo $cwd`

# Location of the nonlinear T1 template
set basedset = MNI152_2009_template_SSW.nii.gz
set tpath = `@FindAfniDsetPath $basedset`
if( "$tpath" == '' ) then
    echo "***** @SSwarper -- Failed to find $basedset :("
    exit 1
endif
set basedset = $tpath/$basedset

# Directory with the T1 warped files
set warpdir = ${topdir}/anat_warped

# Directory with the stimulus timing files
set stimdir = ${topdir}/onsets
afni_proc.py \
   -subj_id subj_anaticor \
   -copy_anat anat_warped/anatSS.subj.nii \
   -anat_has_skull no \
   -anat_follower t1_with_skull anat t1+orig \
   -dsets ep?+orig.HEAD \
   -blocks tshift align tlrc volreg mask blur scale regress \
   -radial_correlate_blocks tcat volreg \
   -tcat_remove_first_trs 0 \
   -align_unifize_epi local \
   -align_opts_aea -cost lpc+ZZ -giant_move -check_flip \
   -tlrc_base $basedset \
   -tlrc_NL_warp \
   -tlrc_NL_warped_dsets \
   $warpdir/anatQQ.subj.nii.gz \
   $warpdir/anatQQ.subj.aff12.1D \
   $warpdir/anatQQ.subj_WARP.nii.gz \
   -volreg_align_to MIN_OUTLIER \
   -volreg_align_e2a \
   -volreg_tlrc_warp \
   -volreg_warp_dxyz 2.0 \
   -volreg_compute_tsnr yes \
   -mask_epi_anat yes \
   -mask_segment_anat yes \
   -mask_segment_erode yes \
   -blur_size 4.0 \
   -regress_anaticor_fast \
   -regress_stim_times \
       ${stimdir}/onset_PosL.txt ${stimdir}/onset_NegL.txt \
       ${stimdir}/onset_PosV.txt ${stimdir}/onset_NegV.txt \
       ${stimdir}/onset_ScrL.txt ${stimdir}/onset_ScrV.txt \
       ${stimdir}/onset_PosFoil.txt ${stimdir}/onset_NegFoil.txt ${stimdir}/onset_ScrFoil.txt \
       -regress_stim_labels PosL NegL PosV NegV ScrL ScrV PosFoil NegFoil ScrFoil \
   -regress_basis 'BLOCK(2,1)' \
   -regress_opts_3dD -jobs 32 \
   -num_glt 12 \
   -gltsym 'SYM: .25*PosL .25*PosV .25*NegL .25*NegV -.50*ScrV -.50*ScrL' \
    -glt_label 1 FACES-SCR.bpress \
   -gltsym 'SYM: .50*PosFoil .50*NegFoil -1*ScrFoil' \
     -glt_label 2 FACES-SCR.foil \
   -gltsym 'SYM: .50*NegV .50*NegL -.50*PosV -.50*PosL' \
      -glt_label 3 FACES.neg-pos \
   -gltsym 'SYM: PosL -NegL' \
      -glt_label 4 PosL-NegL \
   -gltsym 'SYM: PosV -NegV' \
     -glt_label 5 PosV-NegV \
   -gltsym 'SYM: PosV -PosL' \
     -glt_label 6 PosV-PosL  \
   -gltsym 'SYM: NegV -NegL' \
     -glt_label 7 NegV-NegL \
   -gltsym 'SYM: ScrV -ScrL' \
     -glt_label 8 ScrV-ScrL \
   -gltsym 'SYM: PosFoil -NegFoil' \
     -glt_label 9 Foil.pos-neg \
   -gltsym 'SYM: .25*PosV .25*PosL .25*NegV .25*NegL -.50*PosFoil -.50*NegFoil' \
     -glt_label 10 FACES.bpress-foil\
   -gltsym 'SYM: .50*ScrV .50*ScrL -ScrFoil' \
     -glt_label 11 SCR.bpress-foil\
   -gltsym 'SYM: .50*PosL -.50*NegL -.50*PosV .50*NegV' \
     -glt_label 12 Pos-Neg.L.gt.V\
    -regress_motion_per_run \
    -regress_censor_motion 0.3 \
    -regress_censor_outliers 0.05 \
    -regress_3dD_stop \
    -regress_fout no \
    -regress_reml_exec \
    -regress_compute_fitts \
    -regress_make_ideal_sum sum_ideal.1D \
#    -regress_est_blur_epits \
    -regress_est_blur_errts \
    -regress_run_clustsim no \
    -html_review_style pythonic \
    -execute 


# Wrote QC HTML.  To check, consider running:
#
#   afni_open -b QC_subj/index.html
#   afni_open -b subj.results/QC_subj/index.html

-----------------

MODIFIED
#!/bin/tcsh

# This is a customised version of the afni_proc.py
# to be run within the selected subject's directory
#
# The nonlinear warping of the T1 is supposed to have
# been done previously (see @warp_t1_to_mni)

set topdir = `echo $cwd`

# Location of the nonlinear T1 template
set basedset = MNI152_2009_template_SSW.nii.gz
set tpath = `@FindAfniDsetPath $basedset`
if( "$tpath" == '' ) then
    echo "***** @SSwarper -- Failed to find $basedset :("
    exit 1
endif
set basedset = $tpath/$basedset

# Directory with the T1 warped files
set warpdir = ${topdir}/anat_warped

# Directory with the stimulus timing files
set stimdir = ${topdir}/onsets
afni_proc.py \
   -subj_id subj_anaticor_csf \
   -copy_anat anat_warped/anatSS.subj.nii \
   -anat_has_skull no \
   -anat_follower t1_with_skull anat t1+orig \
   -dsets ep?+orig.HEAD \
   -blocks tshift align tlrc volreg mask blur scale regress \
   -radial_correlate_blocks tcat volreg \
   -tcat_remove_first_trs 0 \
   -align_unifize_epi local \
   -align_opts_aea -cost lpc+ZZ -giant_move -check_flip \
   -tlrc_base $basedset \
   -tlrc_NL_warp \
   -tlrc_NL_warped_dsets \
   $warpdir/anatQQ.subj.nii.gz \
   $warpdir/anatQQ.subj.aff12.1D \
   $warpdir/anatQQ.subj_WARP.nii.gz \
   -volreg_align_to MIN_OUTLIER \
   -volreg_align_e2a \
   -volreg_tlrc_warp \
   -volreg_warp_dxyz 2.0 \
   -volreg_compute_tsnr yes \
   -mask_epi_anat yes \
   -mask_segment_anat yes \
   -mask_segment_erode yes \
   -mask_import Tvent ../../ventricle_mask_2mm/template_ventricle_2mm_mni+tlrc \
   -mask_intersect Svent CSFe Tvent \
   -blur_size 4.0 \
   -regress_ROI_PC Svent 3 \
   -regress_ROI_PC_per_run Svent \
   -regress_anaticor_fast \
   -regress_stim_times \
       ${stimdir}/onset_PosL.txt ${stimdir}/onset_NegL.txt \
       ${stimdir}/onset_PosV.txt ${stimdir}/onset_NegV.txt \
       ${stimdir}/onset_ScrL.txt ${stimdir}/onset_ScrV.txt \
       ${stimdir}/onset_PosFoil.txt ${stimdir}/onset_NegFoil.txt ${stimdir}/onset_ScrFoil.txt \
       -regress_stim_labels PosL NegL PosV NegV ScrL ScrV PosFoil NegFoil ScrFoil \
   -regress_basis 'BLOCK(2,1)' \
   -regress_opts_3dD -jobs 32 \
   -num_glt 12 \
   -gltsym 'SYM: .25*PosL .25*PosV .25*NegL .25*NegV -.50*ScrV -.50*ScrL' \
    -glt_label 1 FACES-SCR.bpress \
   -gltsym 'SYM: .50*PosFoil .50*NegFoil -1*ScrFoil' \
     -glt_label 2 FACES-SCR.foil \
   -gltsym 'SYM: .50*NegV .50*NegL -.50*PosV -.50*PosL' \
      -glt_label 3 FACES.neg-pos \
   -gltsym 'SYM: PosL -NegL' \
      -glt_label 4 PosL-NegL \
   -gltsym 'SYM: PosV -NegV' \
     -glt_label 5 PosV-NegV \
   -gltsym 'SYM: PosV -PosL' \
     -glt_label 6 PosV-PosL  \
   -gltsym 'SYM: NegV -NegL' \
     -glt_label 7 NegV-NegL \
   -gltsym 'SYM: ScrV -ScrL' \
     -glt_label 8 ScrV-ScrL \
   -gltsym 'SYM: PosFoil -NegFoil' \
     -glt_label 9 Foil.pos-neg \
   -gltsym 'SYM: .25*PosV .25*PosL .25*NegV .25*NegL -.50*PosFoil -.50*NegFoil' \
     -glt_label 10 FACES.bpress-foil\
   -gltsym 'SYM: .50*ScrV .50*ScrL -ScrFoil' \
     -glt_label 11 SCR.bpress-foil\
   -gltsym 'SYM: .50*PosL -.50*NegL -.50*PosV .50*NegV' \
     -glt_label 12 Pos-Neg.L.gt.V\
    -regress_motion_per_run \
    -regress_censor_motion 0.3 \
    -regress_censor_outliers 0.05 \
    -regress_3dD_stop \
    -regress_fout no \
    -regress_reml_exec \
    -regress_compute_fitts \
    -regress_make_ideal_sum sum_ideal.1D \
#    -regress_est_blur_epits \
    -regress_est_blur_errts \
    -regress_run_clustsim no \
    -html_review_style pythonic \
    -execute  


# Wrote QC HTML.  To check, consider running:
#
#   afni_open -b QC_subj/index.html
#   afni_open -b subj.results/QC_subj/index.html
2

Hi @gpagnon,

If only the GLM options are modified in the new afnI_proc.py script, try adding -write_3dD_script with an output script name. That will generate a single 3dDeconvolve command that can be run in the results directory to produce new stats. You can set a prefix for the output datasets in the script.

It will not handle all of the post regression code though, that is harder. Though if really needed, it might be possible to write an afni_proc.py command with only the regress block. I have not tried that. But still, even that would not produce all of the QC.

Anyway, give -write_3dD_script a try and see if that is sufficient.

  • rick
3

Hello Rick,

thank you so much for the lightning fast reply :-). I think that -write_3dD_script is just what I needed!

with best wishes
giuseppe

4

And just to note, we are aware the main webpage is down, but thanks for mentioning, just in case. We have to replace and update some hardware, and will hopefully be back up again soon.

--pt

1 Like