Running 3ddeconvolve on fMRIPEP gifti outputs

eriklee · January 18, 2024, 10:14pm

Hello AFNI Experts,

I have used the fMRIPREP container (version 23.1.4) to run some minimal fMRI preprocessing, and now I want to use AFNI's 3ddeconvolve/remlfit commands to do task analysis on the output. I am using the AFNI container (version 23.1.10) to run processing, and I have some code setup that is giving reasonable results when ran on the fMRIPREP nifti outputs. However when I adapt that code to run on the fMRIPREP gifti outputs, most of the final files aren't produced. I was wondering if you had any suggestions? If 3ddeconvolve does not support gifti inputs, would it be possible for me to temporarily stuff my gifti data into a nifti file and run 3ddeconvolve on that (if so, what dimensions would 3ddeconvolve be expecting)?

For reference here is the command I am running inside the AFNI container :

3dDeconvolve -force_TR 1.8 -input /func_dir/sub-msc01_ses-01_task-FT4_rec-NoNoiseVols_run-04_hemi-R_space-fsnative_bold.func.gii \
-stim_times_subtract 0.9 \
-polort 2 -ok_1D_text \
-censor /stimuli_dir/custom_censoring_arr_InitialScans_FD0P5AndNeighbors.txt \
-num_stimts 15 \
-stim_times 1 /stimuli_dir/task-FT4_cond-pinky.1D 'BLOCK(12,1)' \
-stim_label 1 pinky \
-stim_times 2 /stimuli_dir/task-FT4_cond-thumb.1D 'BLOCK(12,1)' \
-stim_label 2 thumb \
-stim_times 3 /stimuli_dir/task-FT4_cond-index.1D 'BLOCK(12,1)' \
-stim_label 3 index \
-stim_file 4 /stimuli_dir/trans_x.txt'[0]' -stim_base 4 -stim_label 4 trans_x \
-stim_file 5 /stimuli_dir/trans_y.txt'[0]' -stim_base 5 -stim_label 5 trans_y \
-stim_file 6 /stimuli_dir/trans_z.txt'[0]' -stim_base 6 -stim_label 6 trans_z \
-stim_file 7 /stimuli_dir/trans_x_derivative1.txt'[0]' -stim_base 7 -stim_label 7 trans_x_derivative1 \
-stim_file 8 /stimuli_dir/trans_y_derivative1.txt'[0]' -stim_base 8 -stim_label 8 trans_y_derivative1 \
-stim_file 9 /stimuli_dir/trans_z_derivative1.txt'[0]' -stim_base 9 -stim_label 9 trans_z_derivative1 \
-stim_file 10 /stimuli_dir/rot_x.txt'[0]' -stim_base 10 -stim_label 10 rot_x \
-stim_file 11 /stimuli_dir/rot_y.txt'[0]' -stim_base 11 -stim_label 11 rot_y \
-stim_file 12 /stimuli_dir/rot_z.txt'[0]' -stim_base 12 -stim_label 12 rot_z \
-stim_file 13 /stimuli_dir/rot_x_derivative1.txt'[0]' -stim_base 13 -stim_label 13 rot_x_derivative1 \
-stim_file 14 /stimuli_dir/rot_y_derivative1.txt'[0]' -stim_base 14 -stim_label 14 rot_y_derivative1 \
-stim_file 15 /stimuli_dir/rot_z_derivative1.txt'[0]' -stim_base 15 -stim_label 15 rot_z_derivative1 \
-fout -tout -x1D /out_dir/X.xmat.1D -xjpeg /out_dir/X.jpg \
-fitts /out_dir/fitts -errts /out_dir/errts -bucket /out_dir/stats \
-fout -tout

And this is what I get for outputs:
** DA[0] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
** DA[1] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
** DA[2] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
** DA[3] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
.....MANY MORE LINES OF THIS.....
++ 3dDeconvolve: AFNI version=AFNI_23.1.10 (Jun 30 2023) [64-bit]
++ Authored by: B. Douglas Ward, et al.
** DA[0] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
** DA[1] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
** DA[2] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
** DA[3] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
.....MANY MORE LINES OF THIS.....
++ loading dataset /func_dir/sub-msc01_ses-01_task-FT4_rec-NoNoiseVols_run-04_hemi-R_space-fsnative_bold.func.gii
** DA[0] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
** DA[1] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
** DA[2] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
** DA[3] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
.....MANY MORE LINES OF THIS.....
++ forcibly using TR=1.8000 seconds for -input dataset
++ Skipping check for initial transients
*+ WARNING: Input polort=2; Longest run=430.2 s; Recommended minimum polort=3
++ -stim_times using TR=1.8 s for stimulus timing conversion
++ -stim_times using TR=1.8 s for any -iresp output datasets
++ [you can alter the -iresp TR via the -TR_times option]
++ ** -stim_times NOTE ** guessing GLOBAL times if 1 time per line; LOCAL otherwise
++ Subtracted 0.90 sec from 5 times for -stim_times 1
++ ** GUESSED ** -stim_times 1 using GLOBAL times
++ Subtracted 0.90 sec from 5 times for -stim_times 2
++ ** GUESSED ** -stim_times 2 using GLOBAL times
++ Subtracted 0.90 sec from 5 times for -stim_times 3
++ ** GUESSED ** -stim_times 3 using GLOBAL times
++ Number of time points: 239 (before censor) ; 218 (after)

Number of parameters: 18 [15 baseline ; 3 signal]
++ Memory required for output bricks = 285,763,040 bytes (about 286 million)
++ Wrote matrix image to file /out_dir/X.jpg
++ Wrote matrix values to file /out_dir/X.xmat.1D
++ ========= Things you can do with the matrix file =========
++ (a) Linear regression with ARMA(1,1) modeling of serial correlation:

3dREMLfit -matrix /out_dir/X.xmat.1D -input /func_dir/sub-msc01_ses-01_task-FT4_rec-NoNoiseVols_run-04_hemi-R_space-fsnative_bold.func.gii
-fout -tout -Rbuck /out_dir/stats_REML -Rvar /out_dir/stats_REMLvar
-Rfitts /out_dir/fitts_REML -Rerrts /out_dir/errts_REML -verb

++ N.B.: 3dREMLfit command above written to file /out_dir/stats.REML_cmd
++ (b) Visualization/analysis of the matrix via ExamineXmat.R
++ (c) Synthesis of sub-model datasets using 3dSynthesize
++ ==========================================================
++ ----- Signal+Baseline matrix condition (218x18): 4.25072 ++ VERY GOOD ++
++ ----- Signal-only matrix condition (218x3): 1 ++ VERY GOOD ++
++ ----- Baseline-only matrix condition (218x15): 4.0087 ++ VERY GOOD ++
++ ----- stim_base-only matrix condition (218x12): 3.6512 ++ VERY GOOD ++
++ ----- polort-only matrix condition (218x3): 1.03814 ++ VERY GOOD ++
++ +++++ Matrix inverse average error = 7.421e-14 ++ VERY GOOD ++
++ Matrix setup time = 0.39 s
++ Calculations starting; elapsed time=4.797
++ voxel loop:0123456789.0123456789.0123456789.0123456789.0123456789.
++ Calculations finished; elapsed time=12.474
** DA[0] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
** DA[1] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
** DA[2] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
** DA[3] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
.....MANY MORE LINES OF THIS.....
++ Smallest FDR q [0 Full_Fstat] = 2.33837e-14
++ Smallest FDR q [2 pinky#0_Tstat] = 5.70543e-14
++ Smallest FDR q [3 pinky_Fstat] = 5.70566e-14
++ Smallest FDR q [5 thumb#0_Tstat] = 3.15297e-13
++ Smallest FDR q [6 thumb_Fstat] = 3.15312e-13
++ Smallest FDR q [8 index#0_Tstat] = 5.49435e-14
++ Smallest FDR q [9 index_Fstat] = 5.49435e-14
** failed to open 'stats.gii' for gifti write
++ Wrote bucket dataset into /out_dir/stats.gii

created 7 FDR curves in bucket header
** DA[0] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
** DA[1] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
** DA[2] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
** DA[3] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
.....MANY MORE LINES OF THIS.....
** failed to open 'fitts.gii' for gifti write
++ Wrote 3D+time dataset into /out_dir/fitts.gii
** DA[0] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
** DA[1] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
** DA[2] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
.....MANY MORE LINES OF THIS.....
** failed to open 'errts.gii' for gifti write
++ Wrote 3D+time dataset into /out_dir/errts.gii
++ #Flops=3.78844e+10 Average Dot Product=31.4392

And then in the output directory only the following three files are created:
X.xmat.1D
X.jpg
stats.REML_cmd

Thanks in advance for any suggestions!

Best,
Erik

AFNI version info (23.1.0)

rickr · January 26, 2024, 2:39pm

Hi Erik,

Sorry for the delay. Those intent code warnings were throwing me off. But it looks like you are simply trying to write files to /out_dir, presumably instead of $out_dir. That is to say, you have a '/' instead of a '$' before out_dir, so the program is trying to write under the root directory and failing.

Warnings aside, does it work if you fix that?

rick

eriklee · January 26, 2024, 3:17pm

Hi Rick,

Thanks for your reply! I should have also included how I initialized the AFNI container to get the paths that are used when I am running 3ddeconvolve.

As a first step before running the 3ddeconvolve command, I "shell" into the AFNI container using the code below. This tells the container that the long path I have for the output directory can just be referred to as "/out_dir" within the container.

Since I have various volumetric (nifti) and gifti outputs from fMRIPREP I have tried running the command posted above with both nifti and gifti fMRI files, where the only difference is the file I provide to "-input". That command seems to work totally fine with the MNI/nifti files I have tested, but for some reason most of the outputs aren't created when I use gifti inputs.

Any idea why this might be?

singularity shell \
-B /home/lewi1538/shared/motor_map_from_cristian/AFNI_Out_Dir/3dDeconvolve/FT4:/out_dir \
-B /home/lewi1538/shared/motor_map_from_cristian/AFNI_Out_Dir/stim/sub-msc01_ses-01_task-FT4_rec-NoNoiseVols_run-04_desc-confounds_timeseries:/stimuli_dir \
-B /home/lewi1538/shared/motor_map_from_cristian/fmriprep_output_erik/sub-msc01/ses-01/func:/func_dir \
-B /home/lewi1538/shared/motor_map_from_cristian/AFNI_Out_Dir:/script_dir \
/home/lewi1538/shared/containers/afni_make_build_AFNI_23.1.10.sif

rickr · January 26, 2024, 6:01pm

Note that I can run 3dDeconvolve with GIFTI inputs with no problems or warnings. While your warnings are a bit troubling (time series datasets do no usually even have coordinate systems in them), it does not seem like that is the problem.

What is striking are the error messages like "** failed to open 'errts.gii' for gifti write". That happens when a file fails to open for writing. It really has nothing to do with GIFTI specifically, though a GIFTI file is the purpose of that write. It means the code could not even write "hello there" to the file if it wanted to. That suggests some sort of difficulty with the container paths.

You could also set the environment variable AFNI_GIFTI_VERB to 2 (or even 3) to get more details, but it seems likely that you will mostly just see an extra message like this one before the "failed to open" message.

writing gifti image (with data) to 'test.fitts.FT.surf.lh.gii' (450 DA elements = 62 MB)

To be sure, did 3dDeconvolve produce files like X.xmat.1D in that /out_dir directory?

rick

eriklee · January 26, 2024, 7:00pm

Hi Rick,

I can confirm that I am seeing the X.xmat.1D file, stats.REML_cmd file, and X.jpg file in the output directory (these are the only files that are getting created).

Also the following output is the end of what I see when I set AFNI_GIFTI_VERB to 3. I am not sure what to be looking for here but I didn't notice anything that seemed like obvious errors:

** DA[238] has coordsys with intent NIFTI_INTENT_TIME_SERIES (should be NIFTI_INTENT_POINTSET)
++ UPB, alloc 61440 bytes (from 585580, 0) for buff
++ creating extra zdata for zlib extraction
++ UPB, alloc 591447 bytes (from 591447, 1) for buff
-- uncompressing 490536 bytes into 585580
-- uncompressed buffer (83.77% of 585580 bytes)
-- darray no swap needed : 146395 sets of 4 bytes
==================================================
pop: gifti_image struct
    version    = 1.0
    numDA      = 239
gifti_image struct
    swapped    = 0
    compressed = 1
 -- darray totals: 133 MB
==================================================
++ data was compressed
-- have gifti image '/func_dir/sub-msc01_ses-01_task-FT4_rec-NoNoiseVols_run-04_hemi-R_space-fsnative_bold.func.gii', (239 DA elements = 133 MB)
++ creating gifti_image with 239 DA elements
     intent[2001] = NIFTI_INTENT_TIME_SERIES, dtype[16] = NIFTI_TYPE_FLOAT32,
     alloc_data = 0, ndim = 1, dims: 146395 0 0 0 0 0
++ writing gifti image (with data) to 'errts.gii' (239 DA elements = 133 MB)
-- user opts: gxml_data :
   verb        : 3
   dstore      : 1
   indent      : 3
   buf_size    : 61440
   b64_check   : 5
   zlevel      : -1
   perm_by_iord: 1
   da_len      : 0
++ setting GIFTI MD: gifticlib-version to gifti library version 1.17, 19 Feb, 2021
++ gifti image, numDA = 239, size = 133 MB
++ write 0 ex_atr's
-- freeing gifti_image
++ Wrote 3D+time dataset into /out_dir/gifti/errts.gii
++ #Flops=3.78844e+10  Average Dot Product=31.4392

And as another test I tried using the AFNI installation provided by my university's computing cluster (version 19.3.14) instead of the containerized version that I got from Docker Hub. When I try running the same command on this alternative form of AFNI the results I got are the same.

Thanks again,
Erik

rickr · January 29, 2024, 9:13pm

Hi Erik,

Would it be possible for me to get my hands on one of the GIFTI datasets, just to test locally? If that would be possible, I will send you a private message with details.

Thanks,

rick

rickr · February 6, 2024, 2:46am

Hi Erik,

Thanks for the data and the reminder. There was indeed a problem specific to writing GIFTI datasets, where the output directory was ignored (it was attempting to write to the execution directory). This doesn't have anything to do with your data in particular, but with writing GIFTI in general.

This should be fixed now, with a new build running. Binaries should be ready in a couple of hours. I am not sure whether an update docker container will be built that you can use. We will have to check on that.

Thanks a lot for bringing this up!

rick

rickr · February 6, 2024, 2:38pm

Note that if it is difficult to update your docker container, your current version will work fine if you run it from the output directory. You would include paths to the inputs, but then not use /out_dir as a prefix for the output. Does that seem reasonable?

rick

eriklee · February 6, 2024, 3:47pm

Hi Rick,

Thanks for looking into this! When I go to the output directory before running 3ddeconvolve all the gifti files get created just like you would expect. It looks like this simple solution will work fine for my processing workflow.

Thanks again!
Erik

rickr · February 6, 2024, 4:04pm

That is great!

With the update, there will only be one of those "intent NIFTI_INTENT_TIME_SERIES" warnings per program execution, too. We don't need to see one for every volume. :)

rick