AFNI version info (afni -ver): AFNI_24.0.12 'Caracalla'
Hi AFNI experts!
I recently had to make some changes to my afni_proc.py script and now when I run the script, it runs, but does not produce a stats file for the subject I'm working on. Some subjects in my dataset have a missing.1D file that contains the times for any responses they missed during the task and others do not have a missing.1D file at all. Because of this, I tried to structure the script so that way if the subject has one of these files, it'll adjust the code accordingly. However, adjusting this script has caused some sort of error in the code.
Here is my afni_proc.py script:
#!/usr/bin/env tcsh
# File containing a list of subjects and scan numbers, one per line
set input_file = "subj_scan_list.txt"
foreach line("`cat $input_file`")
set subj = `echo $line | awk '{print $1}'`
set scan_num = `echo $line | awk '{print $2}'`
#data directories
set nifti_dir = "/Users/ajcoope9/Dropbox/2023-LC_fMRI/data/${subj}/mri/${scan_num}"
set outputs_dir = "/Users/ajcoope9/Dropbox/2023-LC_fMRI/data/${subj}/SSwarper_outputs/${scan_num}"
set stim_dir = "/Users/ajcoope9/Dropbox/2023-LC_fMRI/ParticipantFiles/${subj}/Oddball/${scan_num}"
# set subject and group identifiers
set group_id = "LC_PROJECT"
# statement that checks if missing.1D exists. If it does, the path to the file is stored in the variable missing_1D. If it doesn't exist, missing_1D is set to an empty string. The afni_proc.py command then includes this variable in the -regress_stim_times line. If missing_1D is an empty string, it ignores it
set stim_times = "$stim_dir/oddball.1D $stim_dir/default.1D"
if (-e "$stim_dir/missing.1D") then
set stim_times = "$stim_times $stim_dir/missing.1D"
endif
set stim_types = "AM1 AM1"
if (-e "$stim_dir/missing.1D") then
set stim_types = "$stim_types times"
endif
set stim_labels = "oddball default"
if (-e "$stim_dir/missing.1D") then
set stim_labels = "$stim_labels missing"
endif
if (-e "$stim_dir/missing.1D") then
set regress_basis_command = "-regress_basis_multi 'dmUBLOCK(-1)' 'dmUBLOCK(-1)' 'BLOCK(3,1)'"
else
set regress_basis_command = "-regress_basis 'dmUBLOCK(-1)'"
endif
afni_proc.py \
-subj_id $subj \
-copy_anat $outputs_dir/anatSS.${subj}.nii \
-anat_has_skull no \
-align_epi_strip_method 3dSkullStrip \
-anat_follower anat_w_skull anat \
$nifti_dir/MPRAGE+orig.HEAD \
-dsets $nifti_dir/oddball_run1+orig.HEAD \
$nifti_dir/oddball_run2+orig.HEAD \
$nifti_dir/oddball_run3+orig.HEAD \
$nifti_dir/oddball_run4+orig.HEAD \
-blocks tshift align tlrc volreg mask \
blur scale regress \
-radial_correlate_blocks tcat volreg regress \
-align_opts_aea -cost lpc+ZZ \
-check_flip \
-tlrc_base MNI152_2009_template_SSW.nii.gz \
-tlrc_NL_warp \
-tlrc_NL_warped_dsets $outputs_dir/anatQQ.${subj}.nii \
$outputs_dir/anatQQ.${subj}.aff12.1D \
$outputs_dir/anatQQ.${subj}_WARP.nii \
-volreg_align_to MIN_OUTLIER \
-volreg_align_e2a \
-volreg_tlrc_warp \
-volreg_compute_tsnr yes \
-mask_epi_anat yes \
-blur_size 3.0 \
-regress_stim_times $stim_times \
-regress_stim_types $stim_types \
-regress_stim_labels $stim_labels \
$regress_basis_command \
-regress_opts_3dD -jobs 8 \
-gltsym 'SYM: oddball -default' \
-glt_label 1 oddball-default \
-gltsym 'SYM: oddball default' \
-glt_label 2 oddball+default \
-regress_motion_per_run \
-regress_censor_motion 0.3 \
-regress_censor_outliers 0.05 \
-regress_3dD_stop \
-regress_reml_exec \
-regress_compute_fitts \
-regress_make_ideal_sum "sum_ideal.1D" \
-regress_est_blur_epits \
-regress_est_blur_errts \
-regress_run_clustsim no \
-html_review_style pythonic \
-execute
end
Here is the end of the terminal output for this script, I am not seeing any error messages, but am likely missing something:
1d_tool.py -infile dfile_rall.1D -set_run_lengths 123 125 122 127 -demean -write motion_demean.1D
1d_tool.py -infile dfile_rall.1D -set_run_lengths 123 125 122 127 -derivative -demean -write motion_deriv.1D
1d_tool.py -infile motion_demean.1D -set_run_lengths 123 125 122 127 -split_into_pad_runs mot_demean
1d_tool.py -infile dfile_rall.1D -set_run_lengths 123 125 122 127 -show_censor_count -censor_prev_TR -censor_motion 0.3 motion_NMLC133
total number of censored TRs (simple form) = 42
1deval -a motion_NMLC133_censor.1D -b outcount_NMLC133_censor.1D -expr a*b
1d_tool.py -infile censor_NMLC133_combined_2.1D -show_trs_uncensored space
set ktrs = 1dcat out.keep_trs_rall.txt
Badly placed (.
-- template = 'MNI152_2009_template_SSW.nii.gz', exists = 1
-- will use min outlier volume as motion base
-- including default: -find_var_line_blocks tcat
-- tcat: reps is now 123
++ updating polort to 3, from run len 307.5 s
-- importing NL-warp datasets
-- volreg: using base dset vr_base_min_outlier+orig
++ volreg: applying volreg/epi2anat/tlrc xforms to isotropic 2 mm tlrc voxels
-- applying anat warps to 1 dataset(s): MPRAGE
++ mask: using epi_anat mask in place of EPI one
-- masking: group anat = 'MNI152_2009_template_SSW.nii.gz', exists = 1
-- have 1 ROI dict entries ...
++ applying 2 stim types: ['AM1', 'AM1']
++ will compute regress TSNR stats for dsets: brain
-- classes have multiple regressors, so not making ideals
-- using default: will not apply EPI Automask
(see 'MASKING NOTE' from the -help for details)
--> script is file: proc.NMLC133
to execute via tcsh:
tcsh -xef proc.NMLC133 |& tee output.proc.NMLC133
to execute via bash:
tcsh -xef proc.NMLC133 2>&1 | tee output.proc.NMLC133
end
This is a minor scripting note to start. It might be easier to bundle several of the initial variables in then just use a single if condition, like this:
# compactify pre-AP stuff
set stim_times = "$stim_dir/oddball.1D $stim_dir/default.1D"
set stim_types = "AM1 AM1"
set stim_labels = "oddball default"
set regress_basis_command = "-regress_basis 'dmUBLOCK(-1)'"
if (-e "$stim_dir/missing.1D") then
set stim_times = "$stim_times $stim_dir/missing.1D"
set stim_types = "$stim_types times"
set stim_labels = "$stim_labels missing"
set regress_basis_command = "-regress_basis_multi 'dmUBLOCK(-1)' 'dmUBLOCK(-1)' 'BLOCK(3,1)'"
endif
Line 9 of the second (terminal output) code block shows this error message:
Badly placed (.
Can you add -verb 2 to your afni_proc.py command, and see if it outputs more details that help to troubleshoot?
Another thing we sometimes do is to not use -execute, but to dump the afni_proc.py command to a text file and then execute that, which gives an opportunity to see that all variables were filled in what what was expected. (The bottom of the proc file contains a copy of the execute afni_proc.py command, too, for reference.) To do this, you could do (PT: slightly edited now to run the AP command after dumping it to a file):
#!/usr/bin/env tcsh
# File containing a list of subjects and scan numbers, one per line
set input_file = "subj_scan_list.txt"
foreach line("`cat $input_file`")
set subj = `echo $line | awk '{print $1}'`
set scan_num = `echo $line | awk '{print $2}'`
#data directories
set nifti_dir = "/Users/ajcoope9/Dropbox/2023-LC_fMRI/data/${subj}/mri/${scan_num}"
set outputs_dir = "/Users/ajcoope9/Dropbox/2023-LC_fMRI/data/${subj}/SSwarper_outputs/${scan_num}"
set stim_dir = "/Users/ajcoope9/Dropbox/2023-LC_fMRI/ParticipantFiles/${subj}/Oddball/${scan_num}"
# set subject and group identifiers
set group_id = "LC_PROJECT"
# statement that checks if missing.1D exists. If it does, the path to the file is stored in the variable missing_1D. If it doesn't exist, missing_1D is set to an empty string. The afni_proc.py command then includes this variable in the -regress_stim_times line. If missing_1D is an empty string, it ignores it
set stim_times = "$stim_dir/oddball.1D $stim_dir/default.1D"
if (-e "$stim_dir/missing.1D") then
set stim_times = "$stim_times $stim_dir/missing.1D"
endif
set stim_types = "AM1 AM1"
if (-e "$stim_dir/missing.1D") then
set stim_types = "$stim_types times"
endif
set stim_labels = "oddball default"
if (-e "$stim_dir/missing.1D") then
set stim_labels = "$stim_labels missing"
endif
if (-e "$stim_dir/missing.1D") then
set regress_basis_command = "-regress_basis_multi 'dmUBLOCK(-1)' 'dmUBLOCK(-1)' 'BLOCK(3,1)'"
else
set regress_basis_command = "-regress_basis 'dmUBLOCK(-1)'"
endif
# dump AP command to a text file, to look at.
# NB: do not indent where the closing "EOF" is
cat <<EOF >! ap_cmd_${subj}.tcsh
afni_proc.py \
-subj_id $subj \
-copy_anat $outputs_dir/anatSS.${subj}.nii \
-anat_has_skull no \
-align_epi_strip_method 3dSkullStrip \
-anat_follower anat_w_skull anat \
$nifti_dir/MPRAGE+orig.HEAD \
-dsets $nifti_dir/oddball_run1+orig.HEAD \
$nifti_dir/oddball_run2+orig.HEAD \
$nifti_dir/oddball_run3+orig.HEAD \
$nifti_dir/oddball_run4+orig.HEAD \
-blocks tshift align tlrc volreg mask \
blur scale regress \
-radial_correlate_blocks tcat volreg regress \
-align_opts_aea -cost lpc+ZZ \
-check_flip \
-tlrc_base MNI152_2009_template_SSW.nii.gz \
-tlrc_NL_warp \
-tlrc_NL_warped_dsets $outputs_dir/anatQQ.${subj}.nii \
$outputs_dir/anatQQ.${subj}.aff12.1D \
$outputs_dir/anatQQ.${subj}_WARP.nii \
-volreg_align_to MIN_OUTLIER \
-volreg_align_e2a \
-volreg_tlrc_warp \
-volreg_compute_tsnr yes \
-mask_epi_anat yes \
-blur_size 3.0 \
-regress_stim_times $stim_times \
-regress_stim_types $stim_types \
-regress_stim_labels $stim_labels \
$regress_basis_command \
-regress_opts_3dD -jobs 8 \
-gltsym 'SYM: oddball -default' \
-glt_label 1 oddball-default \
-gltsym 'SYM: oddball default' \
-glt_label 2 oddball+default \
-regress_motion_per_run \
-regress_censor_motion 0.3 \
-regress_censor_outliers 0.05 \
-regress_3dD_stop \
-regress_reml_exec \
-regress_compute_fitts \
-regress_make_ideal_sum "sum_ideal.1D" \
-regress_est_blur_epits \
-regress_est_blur_errts \
-regress_run_clustsim no \
-html_review_style pythonic \
EOF
# execute the new AP command, with logging
tcsh -xef ap_cmd_${subj}.tcsh |& tee output.ap.cmd.${subj}
end
Sorry, I accidentally introduced a blank line someone in the AP command, and that means that "blur" is viewed as a new command, incorrectly. Could you try this one:
#!/usr/bin/env tcsh
# File containing a list of subjects and scan numbers, one per line
set input_file = "subj_scan_list.txt"
foreach line("`cat $input_file`")
set subj = `echo $line | awk '{print $1}'`
set scan_num = `echo $line | awk '{print $2}'`
#data directories
set nifti_dir = "/Users/ajcoope9/Dropbox/2023-LC_fMRI/data/${subj}/mri/${scan_num}"
set outputs_dir = "/Users/ajcoope9/Dropbox/2023-LC_fMRI/data/${subj}/SSwarper_outputs/${scan_num}"
set stim_dir = "/Users/ajcoope9/Dropbox/2023-LC_fMRI/ParticipantFiles/${subj}/Oddball/${scan_num}"
# set subject and group identifiers
set group_id = "LC_PROJECT"
# statement that checks if missing.1D exists. If it does, the path
# to the file is stored in the variable missing_1D. If it doesn't
# exist, missing_1D is set to an empty string. The afni_proc.py
# command then includes this variable in the -regress_stim_times
# line. If missing_1D is an empty string, it ignores it
# compactify pre-AP stuff for maybe having "missing" stim
set stim_times = "$stim_dir/oddball.1D $stim_dir/default.1D"
set stim_types = "AM1 AM1"
set stim_labels = "oddball default"
set regress_basis_command = "-regress_basis 'dmUBLOCK(-1)'"
if (-e "$stim_dir/missing.1D") then
set stim_times = "$stim_times $stim_dir/missing.1D"
set stim_types = "$stim_types times"
set stim_labels = "$stim_labels missing"
set regress_basis_command = "-regress_basis_multi 'dmUBLOCK(-1)' 'dmUBLOCK(-1)' 'BLOCK(3,1)'"
endif
# dump AP command to a text file, to look at.
# NB: do not indent where the closing "EOF" is
cat <<EOF >! ap_cmd_${subj}.tcsh
afni_proc.py \
-subj_id $subj \
-copy_anat $outputs_dir/anatSS.${subj}.nii \
-anat_has_skull no \
-align_epi_strip_method 3dSkullStrip \
-anat_follower anat_w_skull anat \
$nifti_dir/MPRAGE+orig.HEAD \
-dsets $nifti_dir/oddball_run1+orig.HEAD \
$nifti_dir/oddball_run2+orig.HEAD \
$nifti_dir/oddball_run3+orig.HEAD \
$nifti_dir/oddball_run4+orig.HEAD \
-blocks tshift align tlrc volreg mask \
blur scale regress \
-radial_correlate_blocks tcat volreg regress \
-align_opts_aea -cost lpc+ZZ \
-check_flip \
-tlrc_base MNI152_2009_template_SSW.nii.gz \
-tlrc_NL_warp \
-tlrc_NL_warped_dsets $outputs_dir/anatQQ.${subj}.nii \
$outputs_dir/anatQQ.${subj}.aff12.1D \
$outputs_dir/anatQQ.${subj}_WARP.nii \
-volreg_align_to MIN_OUTLIER \
-volreg_align_e2a \
-volreg_tlrc_warp \
-volreg_compute_tsnr yes \
-mask_epi_anat yes \
-blur_size 3.0 \
-regress_stim_times $stim_times \
-regress_stim_types $stim_types \
-regress_stim_labels $stim_labels \
$regress_basis_command \
-regress_opts_3dD -jobs 8 \
-gltsym 'SYM: oddball -default' \
-glt_label 1 oddball-default \
-gltsym 'SYM: oddball default' \
-glt_label 2 oddball+default \
-regress_motion_per_run \
-regress_censor_motion 0.3 \
-regress_censor_outliers 0.05 \
-regress_3dD_stop \
-regress_reml_exec \
-regress_compute_fitts \
-regress_make_ideal_sum "sum_ideal.1D" \
-regress_est_blur_epits \
-regress_est_blur_errts \
-regress_run_clustsim no \
-html_review_style pythonic \
EOF
# execute the new AP command, with logging
tcsh -xef ap_cmd_${subj}.tcsh |& tee output.ap.cmd.${subj}
end
Great, and I don't see an error output. Did it run?
One thing I notice is your 2nd GLT:
-gltsym 'SYM: oddball default' \
There is no operator between the labels. I am not certain that this is a problem (@rickr or @Gang ?), but I might have expected this:
-gltsym 'SYM: oddball +default' \
for example.
Note that in our Bootcamp s05* script, we calculate the average of the two stimuli in the task; I wonder if that might be of interest, rather than the sum, although it is obviously just a factor of 2 difference. So, that would look like:
Unfortunately, it didn't run. I edited the second GLT to include the "+" in between oddball and default and the script failed pretty quickly as well. I'll keep investigating to try and figure out what the culprit could be, but if you see anything that jumps out to you, let me know!
OK, that runs the AP command to generate the proc script; then you should execture the proc script that was created with one of these recommended commands:
to execute via tcsh:
tcsh -xef proc.NMLC133 |& tee output.proc.NMLC133
to execute via bash:
tcsh -xef proc.NMLC133 2>&1 | tee output.proc.NMLC133
What is the output from that? (it will be captured into the output.proc.* file)
Well, as part of the process of checking the command for runtime errors here, I had removed -execute and instead did this:
# dump AP command to a text file, then execute separately
cat << EOF >> ap_cmd_${subj}.tcsh
afni_proc.py \
...
EOF
# execute the new AP command, with logging
tcsh -xef ap_cmd_${subj}.tcsh |& tee output.ap.cmd.${subj}
Was that second piece of executing the AP command (line 10 here) in your script?
Importantly, afni_proc.py doesn't just process your data. Even better, it creates a script that fully informs you (with comments!) about how it will process your data, and then you can run that. This gives you detailed provenance/history of your processing from start to finish, so you don't have to wonder about what is happening. This detailed more in this recent (and long awaited!) draft article:
Wow, you two were busy. So the first error was shown by:
set ktrs = 1dcat out.keep_trs_rall.txt
Badly placed (.
That suggests the line after "set ktrs" failed, which is 3dDeconvolve, and the place that you were using () in that was in the basis functions. And that use of $regress_basis_command looks suspicious from the point of view of shell syntax
But later on the script was changed to redirecting this to a text file and running that. I think this actually fixed the issue, as it changed the application of the multiple quotes.
Plus, debugging is easier, since you can review that command script, which removes one level of syntax nesting.
The
National Institute of Mental Health (NIMH) is part of the National Institutes of
Health (NIH), a component of the U.S. Department of Health and Human
Services.