afni_proc.py ** failed to open 1D file

AFNI version info ('Septimius Severus'):

Hello,

I am trying to run this afni_proc.py script below and I am getting an error that it is not able to open the 1D files:

# control variables
set final_dxyz = 1.75
set blur_size = 6
set cen_motion = 0.3
set cen_outlier = 0.05


afni_proc.py -subj_id "${subj}"                                                          \
             -script ${subj}.${task}.${ses}.data_prep -scr_overwrite                     \
             -out_dir ${top_dir}/${task}_task_output                                     \
             -copy_anat ${subj_dir}/sswarp/${ses}/anatSS.${subj}.nii                     \
             -anat_has_skull no                                                          \
             -anat_follower anat_w_skull anat ${niix_dir}/*_t1_mprage_tra_p2_iso.nii.gz  \
             -dsets ${niix_dir}/*_${task}_R*.nii.gz                                      \
             -volreg_base_dset "${volreg_min_outlier_dset}"                              \
             -blocks tcat despike tshift align tlrc volreg mask blur scale regress       \
             -radial_correlate_blocks tcat volreg regress                                \
             -tshift_opts_ts -tpattern alt+z2                                            \
             -align_opts_aea -cost lpc+ZZ                                                \
             				 -giant_move                                                 \
             				 -check_flip                                                 \
             -tlrc_base MNI152_2009_template_SSW.nii.gz                                  \
             -tlrc_NL_warp                                                               \
             -tlrc_NL_warped_dsets ${subj_dir}/sswarp/${ses}/anatQQ.${subj}.nii          \
								   ${subj_dir}/sswarp/${ses}/anatQQ.${subj}.aff12.1D	 \
								   ${subj_dir}/sswarp/${ses}/anatQQ.${subj}_WARP.nii	 \
#            -volreg_align_to MIN_OUTLIER                                                \
             -volreg_align_e2a                                                           \
             -volreg_tlrc_warp                                                           \
             -volreg_compute_tsnr yes                                                    \
             -volreg_warp_dxyz "${final_dxyz}"                                           \
             -mask_epi_anat yes                                                          \
             -blur_size "${blur_size}"                                                   \
             -regress_stim_times                                                         \
             	"${stims1}"                                                              \
             	"${stims2}"                                                              \
             	"${stims3}"                                                              \
             	"${stims4}"                                                              \
             	"${stims5}"                                                              \
             	"${stims6}"                                                              \
             -regress_stim_labels                                                        \
             	"${stim_labels1}"                                                        \
             	"${stim_labels2}"                                                        \
             	"${stim_labels3}"                                                        \
             	"${stim_labels4}"                                                        \
             	"${stim_labels5}"                                                        \
             	"${stim_labels6}"                                                        \
             -regress_basis                                                              \
             	"${regress_funct}"                                                       \
             -regress_opts_3dD -local_times                                              \
                               -jobs 4                                                   \
                               -gltsym "${sym1}" -glt_label 1 ${label1}                  \
                               -gltsym "${sym2}" -glt_label 2 ${label2}                  \
                               -gltsym "${sym3}" -glt_label 3 ${label3}                  \
             -regress_apply_mot_types demean deriv                                       \
#              -test_stim_files no                                                         \
             -regress_censor_motion "${cen_motion}"                                      \
             -regress_censor_outliers "${cen_outlier}"                                   \
             -regress_reml_exec                                                          \
             -regress_compute_fitts                                                      \
             -regress_make_ideal_sum sum_ideal.1D                                        \
             -regress_est_blur_epits                                                     \
             -regress_est_blur_errts                                                     \
             -regress_run_clustsim no                                                    \
             -html_review_style pythonic                                                 
                                
             tcsh -xef ${subj}.${task}.${ses}.data_prep

I am getting the error for all of our tasks with this script. Below is the error that I am getting:

** failed to open 1D file '/labs/sim_lab/test/fmri_data/subjects/26GLE/analyses/regressors/session_1/alcohol_set_*.1D'
** A1D: failed to read data file /labs/sim_lab/test/fmri_data/subjects/26GLE/analyses/regressors/session_1/alcohol_set_*.1
D

Any thoughts on this would be greatly appreciated!

Thank you

-Garrett

What is the output of:

ls /labs/sim_lab/test/fmri_data/subjects/26GLE/analyses/regressors/session_1/alcohol_set_*.1D

?

--pt

Hi Garrett,

In addition to Paul's verification of the path, note that the filename in question is not shown in the afni_proc.py command, meaning it is presumably set as a variable. And the command variables are being passed in quotes, such as:
-regress_stim_times "${stims1}" ...

If $stims1 equals .../alcohol_set_*.1D, then the shell will never expand the wildcard, since it is in quotes.

So you should probably remove the quotes around most of your variables.

  • rick

Hello Paul and Rick,

The output from the following is:

ls /labs/sim_lab/test/fmri_data/subjects/26GLE/analyses/regressors/session_1/alcohol_set_*.1D

alcohol_set_a.1D

I removed the double quotes around the variables within my afni_proc.py script and that seems to have fixed my error. Thank you both for your replies and helping me with this!

Is there a general rule on when we need to have the double quotes around the variable in afni_proc.py? I was able to remove them from most of my variables in afni_proc.py without issue, except for the volreg_base_dset and the gltsym names.

-Garrett

Hi, Garrett-

The issues of when to put quotes or not is not an AFNI/afni_proc.py, but a shell thing. The same would apply for any command line tools/program/software.

There can be some subtlety, but I think one place to start would be that if you use an asterisk within quotes when defining a variable:

set my_var = "/some/path/to/dataset/name_*.ext"

... then you would not include quotes when using it:

prog -opt $my_var

Using quotes in the variable definition is "protecting" the asterisk from being intepreted and expanded as a wildcard; if you use quotes when applying that variable in the option usage, that would again protect against allowing that wildcard to be applied as a wildcard, and so the program opt literally sees the character * in the filename to try to interpret---which is not what should be happening.

Actually, never say never, but I think I would generally not put wildcards into quotes when defining a variable. It might be better to do this:

set my_var = ( /some/path/to/dataset/name_*.ext )

... to allow the $my_var variable to hold a list of the files that get found, as if running ls /some/path/to/dataset/name_*.ext, even if there is only 1 file there in reality. Then, you can also query whether $my_var has zero, one or more elements in its list---whatever is appropriate.

Also, "normal" file names typically don't need quotes. Those tend to be useful when special characters are involved for special functionality. For example, to use AFNI subbrick selectors [...], you would need quotes to protect the shell from interpreting those in a shell-variable-syntax way. So, you would use:

set my_var = FILE"[0]"
3dcalc -a "$my_var" ...

... so that the shell never gets to see the square brackets to try to interpret how it wants---using quotes all the time allows AFNI to see them and interpret them how it wants. If you don't use subbrick selectors or other odd characters (like eeeevil spaces) in filenames, then you will likely be OK without so many quotes. It's certainly better to never put spaces in filenames, anyways.

--pt

Hi Paul,

Thank you so much, for explaining the quotes for me that is very helpful. I was able to run my script again without most of the double quotes around the variables. I have gotten the following error in 3dDeconvolve:

++ Wrote matrix values to file X.nocensor.xmat.1D
++ ----- Signal+Baseline matrix condition [X] (223x26):  3.14585  ++ VERY GOOD ++
ESC[7m*+ WARNING:ESC[0m !! in Signal+Baseline matrix:
 * Largest singular value=1.90196
 * 2 singular values are less than cutoff=1.90196e-07
 * Implies strong collinearity in the matrix columns! 
++ Signal+Baseline matrix singular values:
   8.70521e-09   2.96095e-08      0.192188      0.259312      0.346414
      0.369094      0.395232      0.481212      0.556685      0.579813
      0.730691      0.754634       0.80302      0.888847      0.899808
      0.964907       1.00331       1.14366       1.21512       1.27832
        1.3227       1.37487       1.41682       1.60925       1.74563
       1.90196
++ ----- Signal-only matrix condition [X] (223x6):  1.3612  ++ VERY GOOD ++
ESC[7m*+ WARNING:ESC[0m !! in Signal-only matrix:
 * Largest singular value=1.60661
 * 2 singular values are less than cutoff=1.60661e-07
 * Implies strong collinearity in the matrix columns! 
++ Signal-only matrix singular values:
             0    2.2546e-08      0.867095       1.05612       1.24562
       1.60661
++ ----- Baseline-only matrix condition [X] (223x20):  2.99531  ++ VERY GOOD ++
++ ----- stim_base-only matrix condition [X] (223x12):  2.16199  ++ VERY GOOD ++
++ ----- polort-only matrix condition [X] (223x8):  1.10645  ++ VERY GOOD ++
ESC[7m*+ WARNING:ESC[0m +++++ !! Matrix inverse average error = 0.00724412  ** BEWARE **
++ Matrix setup time = 0.87 s
ESC[7m** ERROR:ESC[0m !! 3dDeconvolve: Can't run past 3 matrix warnings without '-GOFORIT 3'
ESC[7m** ERROR:ESC[0m !!                Currently at -GOFORIT 0
ESC[7m** ERROR:ESC[0m !! See file 3dDeconvolve.err for all WARNING and ERROR messages !!
ESC[7m** ERROR:ESC[0m !! Be sure you understand what you are doing before using -GOFORIT !!
ESC[7m** ERROR:ESC[0m !! If in doubt, consult with someone or with the AFNI message board !!
ESC[7m** FATAL ERROR:ESC[0m !! 3dDeconvolve (regretfully) shuts itself down !!

I received this for both our Alcohol and Food task runs, but not the VIA task runs. Any help with this would with this or helping explain it would be greatly appreciated! I'd be happy to give more information about our data if needed.

Thank you
-Garrett

A post was split to a new topic: apqc_make_tcsh.py error