afni_proc.py error: +orig vs. +tlrc mismatch

AFNI Message Board
1

Hi AFNI gurus,

I’m running afni_proc.py after fMRIPrep. It works fine for one participant but fails for another, even though I’m using the exact same parameters.

Here’s the error where it stops (slurm logs copied in full at the end):

3dresample -master full_mask.sNTD194+tlrc \
           -input sub-NTD194_ses-2_space-MNI152NLin6Asym_res-2_desc-preproc_T1w_brain+orig \
           -prefix rm.resam.anat

** invalid input dataset <sub-NTD194_ses-2_space-MNI152NLin6Asym_res-2_desc-preproc_T1w_brain+orig>

For the participant that worked, the equivalent line was:

3dresample -master full_mask.sNTD193+tlrc \
           -input sub-NTD193_ses-2_space-MNI152NLin6Asym_res-2_desc-preproc_T1w_brain+tlrc \
           -prefix rm.resam.anat

So the key difference appears to be:

  • Failure case: anatomical dataset labeled +orig
  • Working case: anatomical dataset labeled +tlrc

In the failing participant’s folder, the file is actually +tlrc. It seems AFNI is trying to find an +orig version (which doesn’t exist), and then stops.

Do you know why one participant would be assigned +orig while another is correctly assigned +tlrc?

As always, thanks very much for your advice and time!

Best,
Lauren

2

Hi, Lauren-

Let's check the header info of the input files you are using. Particularly, it is the sform_code and qform_code values that label what space a dset is in: original or template, along with potentially some more detailed info about which template it is; and most dreadedly, there is an "ambiguous" value.

One way to check these is to run nifti_tool on the datasets in question, honing in on the fields of interest here

nifti_tool \
    -disp_hdr -field sform_code -field qform_code \
    -infiles DSET1 DSET2 DSET3 ...

A what version of AFNI do you have? It is also possible to use gtkyd_check.py to make a useful spreadsheet of all properties (and this can be followed up with gen_ss_review_table.py to programmatically check for consistency of header properties).

--pt

3

Hi Paul,

Thanks for your reply!

Here are some additional details:

  • AFNI version on our cluster:
    Precompiled binary linux_openmp_64: Jan 11 2024 (Version AFNI_24.0.01 'Caracalla')

It looks like this version might not include gtkyd_check.py?

Results from nifti_tool:

Working subject:

N-1 header file 'sub-NTD193_ses-2_space-MNI152NLin6Asym_res-2_desc-preproc_T1w_brain.nii', num_fields = 2
  name                offset  nvals  values
  ------------------- ------  -----  ------
  sform_code           254      1    4
  qform_code           252      1    4

Failed subject:

N-1 header file 'sub-NTD194_ses-2_space-MNI152NLin6Asym_res-2_desc-preproc_T1w_brain.nii', num_fields = 2
  name                offset  nvals  values
  ------------------- ------  -----  ------
  sform_code           254      1    4
  qform_code           252      1    4

These appear identical to me.

Results from 3dinfo -space -av_space:
For both files:

MNI  tlrc  LPI  0 -2 -2 -2

So far, everything looks the same between the “working” and “failed” subjects. I’m not sure what to check next—any suggestions would be greatly appreciated.

Thanks so much for your help!
Best,
Lauren

4

Hi Lauren,

The final space will actually come from the EPI, not the anat. What is this information for the input EPI datasets?

Thanks,

-rick

5

Hi Rick - I just double checked the information for the input epi datasets (denoised_func_data_nonaggr.nii.gz) and the information is identical to the above and identical between both the subject that worked and the one that failed.
Best,
Lauren

6

How about just looking for all datasets from each subject?

cd ..
nifti_tool -disp_hdr -field sform_code -field qform_code -infiles */*.nii*

What does this show for both of those subjects?

Thanks,

-rick

7

To clarify, do you mean all nifti files from the afni_proc.py output? If so, I think they also seem the same.

**Subject that FAILED:

[llebois@mickey NTD194_afni_proc_fmriprep_wm_csf_3cond] nifti_tool -disp_hdr -field sform_code -field qform_code -infiles /.nii*

N-1 header file 'vlines.pb00.tcat/mask.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/proj.min.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/proj.r01.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/proj.r02.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/proj.r03.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/tmp.mask.col.count.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/tmp.mask.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/var.0.orig.r01.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/var.0.orig.r02.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/var.0.orig.r03.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/var.1.scale.r01.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/var.1.scale.r02.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/var.1.scale.r03.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4
[llebois@mickey NTD194_afni_proc_fmriprep_wm_csf_3cond] cd ../NTD193_afni_proc_fmriprep_wm_csf_3cond/

Subject that worked*

[llebois@mickey NTD193_afni_proc_fmriprep_wm_csf_3cond] nifti_tool -disp_hdr -field sform_code -field qform_code -infiles /.nii*

N-1 header file 'vlines.pb00.tcat/mask.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/proj.min.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/proj.r01.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/proj.r02.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/proj.r03.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/tmp.mask.col.count.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/tmp.mask.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/var.0.orig.r01.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/var.0.orig.r02.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/var.0.orig.r03.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/var.1.scale.r01.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/var.1.scale.r02.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4

N-1 header file 'vlines.pb00.tcat/var.1.scale.r03.nii.gz', num_fields = 2
name offset nvals values

sform_code 254 1 4
qform_code 252 1 4
[llebois@mickey NTD193_afni_proc_fmriprep_wm_csf_3cond]

8

Sorry no, please run that on all inputs to afni_proc.py. It is almost certainly coming from the inputs somewhere.

An alternative method would be to add -show_tracked_files ALL to the afni_proc.py command, though that is messy, since it would want to overwrite the proc script.

Anyway, it might help to post the actual afni_proc.py command. Then we could give exact input files (basically any dataset input to afni_proc.py).

Thanks,

-rick

9

Got it! Here is the afniproc.py command:

oreach subj ($subjects_included)

    # Determine scanner type from subject ID
    set numpart = `echo $subj | sed 's/[^0-9]//g'`
    if ( $numpart >= 167 ) then
        set stim_offset = 1.167841   # Prisma
    else
        set stim_offset = 1.153182   # Trio
    endif

    # -------------------------------------------------------------------
    # Auto-locate anat and epi directories
    # -------------------------------------------------------------------
    set anat_dir = `ls -d /data/ddtrp/TP2/${subj}/${subj}_2/*/derivatives/fmriprep/sub-${subj}/ses-2/anat | head -n 1`
    set epi_dir  = `ls -d /data/ddtrp/TP2/${subj}/${subj}_2/*/ica_aroma/out | head -n 1`

    if ( "$anat_dir" == "" || "$epi_dir" == "" ) then
        echo "ERROR: Missing anat or epi dir for subject $subj"
        continue
    endif    
    
## -------------------------------------------------------------------
    # Run afni_proc.py
    # -------------------------------------------------------------------

    afni_proc.py -subj_id s{$subj}                                    					    \
         -script proc.s{$subj}						  										\
	  	 -scr_overwrite                                     								\
#	 -execute							     												\
	 -out_dir $top_dir/{$subj}_afni_proc_fmriprep_wm_csf_3cond								\
     -blocks mask scale regress           														\
     -copy_anat "$anat_dir/sub-${subj}_ses-2_space-MNI152NLin6Asym_res-2_desc-preproc_T1w_brain.nii.gz" \
	 -anat_has_skull no																		\
     -dsets $epi_dir/notme1/denoised_func_data_nonaggr.nii.gz            	                \
    	   $epi_dir/notme2/denoised_func_data_nonaggr.nii.gz         	                    \
    	   $epi_dir/notme3/denoised_func_data_nonaggr.nii.gz         	                    \
     -tcat_remove_first_trs 0                                         						\
	 -mask_epi_anat yes						    											\
	 -regress_stim_times_offset $stim_offset												\
     -regress_stim_times $stim_dir/s{$subj}_*Female_times.1D           					    \
     -regress_stim_labels Famous_Female_times Other_Female_times Self_Female_times   		\
     -regress_basis 'BLOCK(23.6,1)'                   					                    \
	 -regress_make_cbucket yes					    										\
	 -regress_extra_ortvec /data/ddtrp/TP2/Nuisance_Reg_WM_CSF/{$subj}_csf_notme_all.1D		\
	 					   /data/ddtrp/TP2/Nuisance_Reg_WM_CSF/{$subj}_wm_notme_all.1D		\
	 -regress_extra_ortvec_labels CSF WM													\
	 -regress_3dD_stop						      											\
     -regress_reml_exec                                                    					\
     -regress_opts_3dD						      											\
	    -jobs 2							      												\
	    -bout                                                	      						\
	    -num_glt 3							      											\
            -gltsym 'SYM: Self_Female_times -Other_Female_times'	      					\
		-glt_label 1 Self_F-Other_F            			      								\
            -gltsym 'SYM: Self_Female_times -Famous_Female_times'             				\
		-glt_label 2 Self_F-Famous_F           			      								\
            -gltsym 'SYM: Famous_Female_times -Other_Female_times'	      					\
		-glt_label 3 Famous_F-Other_F			              								\
    -regress_compute_fitts                                                					\
    -regress_est_blur_errts						      										\
	-regress_local_times
end
10

And not to muddy the conversation here, but to interject on an earlier point: Indeed, that earlier version of AFNI probably doesn't have the gtkyd_check.py in it. We have also made some useful recent updates to it.

We have some more updates coming to afni_proc.py's QC HTML coming in near future, in time for the upcoming Bootcamp, so it might be worth pinging the HPC folks about updating AFNI there. Also, you can always have a local copy of AFNI binaries that you control, using the same package flavor of the HPC, so all the dependencies are present, and then you can put your version at the start of your $PATH, and update it as you wish (which would probably be more regularly than the HPC-moderated version).

-pt

11

It appears the inputs for all input files in the working and failed participant are the same as well?

WORKING

N-1 header file 'sub-NTD193_ses-2_space-MNI152NLin6Asym_res-2_desc-preproc_T1w_brain.nii', num_fields = 2
  name                offset  nvals  values
  ------------------- ------  -----  ------
  sform_code           254      1    4
  qform_code           252      1    4

llebois@mickey notme1] nifti_tool -disp_hdr -field sform_code -field qform_code -inflies denoised_func_data_nonaggr.nii.gz 

N-1 header file 'denoised_func_data_nonaggr.nii.gz', num_fields = 2
  name                offset  nvals  values
  ------------------- ------  -----  ------
  sform_code           254      1    4
  qform_code           252      1    4
[llebois@mickey notme1] 3dinfo -space -av_space denoised_func_data_nonaggr.nii.gz 
MNI	+tlrc
[llebois@mickey notme1] 3dinfo -orient -obliquity -d3 denoised_func_data_nonaggr.nii.gz 
LPI	0.000	-2.000000	-2.000000	2.000000
[llebois@mickey notme1] cd ../notme2
[llebois@mickey notme2] nifti_tool -disp_hdr -field sform_code -field qform_code -inflies denoised_func_data_nonaggr.nii.gz

N-1 header file 'denoised_func_data_nonaggr.nii.gz', num_fields = 2
  name                offset  nvals  values
  ------------------- ------  -----  ------
  sform_code           254      1    4
  qform_code           252      1    4
[llebois@mickey notme2] 3dinfo -space -av_space denoised_func_data_nonaggr.nii.gz
MNI	+tlrc
[llebois@mickey notme2] 3dinfo -orient -obliquity -d3 denoised_func_data_nonaggr.nii.gz
LPI	0.000	-2.000000	-2.000000	2.000000
[llebois@mickey notme2] cd ../notme3
[llebois@mickey notme3] nifti_tool -disp_hdr -field sform_code -field qform_code -inflies denoised_func_data_nonaggr.nii.gz

N-1 header file 'denoised_func_data_nonaggr.nii.gz', num_fields = 2
  name                offset  nvals  values
  ------------------- ------  -----  ------
  sform_code           254      1    4
  qform_code           252      1    4

FAILED participant

N-1 header file 'sub-NTD194_ses-2_space-MNI152NLin6Asym_res-2_desc-preproc_T1w_brain.nii', num_fields = 2
  name                offset  nvals  values
  ------------------- ------  -----  ------
  sform_code           254      1    4
  qform_code           252      1    4

[llebois@mickey TP2] cd NTD194/NTD194_2/E41609522/ica_aroma/out/notme1
[llebois@mickey notme1] nifti_tool -disp_hdr -field sform_code -field qform_code -inflies denoised_func_data_nonaggr.nii.gz

N-1 header file 'denoised_func_data_nonaggr.nii.gz', num_fields = 2
  name                offset  nvals  values
  ------------------- ------  -----  ------
  sform_code           254      1    4
  qform_code           252      1    4
[llebois@mickey notme1] 3dinfo -space -av_space denoised_func_data_nonaggr.nii.gz
MNI	+tlrc
[llebois@mickey notme1] 3dinfo -orient -obliquity -d3 denoised_func_data_nonaggr.nii.gz
LPI	0.000	-2.000000	-2.000000	2.000000
[llebois@mickey notme1] cd ../notme2
[llebois@mickey notme2] nifti_tool -disp_hdr -field sform_code -field qform_code -inflies denoised_func_data_nonaggr.nii.gz

N-1 header file 'denoised_func_data_nonaggr.nii.gz', num_fields = 2
  name                offset  nvals  values
  ------------------- ------  -----  ------
  sform_code           254      1    4
  qform_code           252      1    4

[llebois@mickey notme2] 3dinfo -space -av_space denoised_func_data_nonaggr.nii.gz
MNI	+tlrc
[llebois@mickey notme2] cd ../notme3
[llebois@mickey notme3] nifti_tool -disp_hdr -field sform_code -field qform_code -inflies denoised_func_data_nonaggr.nii.gz

N-1 header file 'denoised_func_data_nonaggr.nii.gz', num_fields = 2
  name                offset  nvals  values
  ------------------- ------  -----  ------
  sform_code           254      1    4
  qform_code           252      1    4
12

Hmm. It is hard to see how these two cases are different.

Just to be sure that nothing odd happened in scripting, are you taking these dataset values that you are checking from the proc* itself? In the proc script, the command to copy the input anatomical is probably a 3dcopy call just after this comment:

# copy anatomy to results dir
3dcopy ...

... and the copying of the input EPI datasets happens at the start of the tcat block, which looks like this:

# ============================ auto block: tcat ============================

I'm grasping at straws a little bit, to see what is happening here.

Were these runs processed recently and concurrently, so there is no chance that an earlier version of a script or path or anything changed between them?

--pt

13

Just to follow up on this for anyone else reading, we zoomed a bit, and it seems like there might have been some network instability while generating the proc scripts. So many were good, but then it stopped being able to find some of the anatomical datasets, and ended up using the default +orig view for the script in those cases, which failed later during execution of the proc scripts.

If these failures happen again, it might be necessary to just wait a bit and try again.

Another option would be to see when afni_proc.py whines about finding the files, and then terminate the processing immediately.

-rick

14

Blame the network! Sounds good to me.

Thanks for resolving this. It was pretty perplexing.

--pt