Help transfering individual functions to afni_proc.py

e.baldelomar · August 28, 2024, 3:42pm

AFNI version info (afni -ver): Version AFNI_24.2.01 'Macrinus'

Hello,

I'd like to move my previous pre- processing protocol to afni_proc.py, if possible. Below are my current functions. Ideally, I do not want it to do any auto-masking as I've found that our application requires us to manually segment our areas of interest.

Let me know any other information I can provide to help clarify.

3dDespike \
-overwrite \
-nomask \
-localedit \
-cut 2.5 5 \
-ignore 2 \
-prefix $AnimalID/$FolderID/${AnimalID}_${FolderID}.DS.Box \
$AnimalID/$FolderID/${AnimalID}_${FolderID}.Box+orig

align_epi_anat.py \
-overwrite \
-epi $AnimalID/$FolderID/${AnimalID}_${FolderID}.DS.Box+orig \
-anat $AnimalID/$FolderID/${AnimalID}_${FolderID}.Box.Mean+orig \
-epi2anat \
-epi_base "$NBASE0" \
-epi_strip None \
-volreg_method 3dvolreg \
-volreg_base median \
-volreg_opts  "-heptic -weight COPY_tempab2D+orig'[0]'" \
-anat_has_skull no \
-output_dir "$AnimalID/$FolderID/" \
-cost ls

# have not validated this below function works. Just want to give an idea that I'd like to also include this kind of regressing protocol
#
afni_proc.py                                  \
-subj_id                 $AnimalID  \
-dsets                    $AnimalID/$FolderID/${AnimalID}_${FolderID}.DS.Box_al+orig \
-blocks                  regress \
-anat_follower_ROI mMask  epi   $AnimalID/$FolderID/MMASK_${AnimalID}_${FolderID}.Box.Mean+orig \
-regress_ROI       mMask

Any help is greatly appreciated!

rickr · August 28, 2024, 5:41pm

Hello,

While the option order does not matter, let me set up a command that specifies the EPI (and any related into), the anat (and related), then the -blocks and block options. Changes that will be included:

include a scale block
assume 2 pre-steady state TRs (-ignore 2 suggests it)
align EPI to MIN_OUTLIER rather than median or $NBASE0
I do not currently think there is a way to apply 3dvolreg -weight
(such a thing could possibly be added)
regression will include motion (you could consider censoring)
some additional QC will be added

This is tcsh syntax, so you may need to translate.

set anat = $AnimalID/$FolderID/${AnimalID}_${FolderID}.Box.Mean+orig
set epis = ( $AnimalID/$FolderID/${AnimalID}_${FolderID}.Box+orig )
set ROI  = $AnimalID/$FolderID/MMASK_${AnimalID}_$FolderID.Box.Mean+orig

afni_proc.py                                                           \
    -subj_id                  $AnimalID                                \
    -dsets                    $epis                                    \
    -copy_anat                $anat                                    \
    -anat_has_skull           no                                       \
    -anat_follower_ROI        mMask epi $ROI                           \
    -blocks                   despike align volreg mask scale regress  \
    -radial_correlate_blocks  tcat regress                             \
    -tcat_remove_first_trs    2                                        \
    -despike_opts_3dDes       -nomask -localedit -cut 2.5 5 -ignore 2  \
    -align_epi_strip_method   None                                     \
    -align_opts_aea           -cost ls                                 \
    -volreg_interp            -heptic                                  \
    -volreg_align_to          MIN_OUTLIER                              \
    -mask_epi_anat            yes                                      \
    -regress_motion_per_run                                            \
    -regress_ROI              mMask                                    \
    -regress_est_blur_errts                                            \
    -html_review_style        pythonic

Ponder that a bit, give it a try, and we can go from there.

rick

e.baldelomar · August 28, 2024, 6:13pm

Thanks so much for this Rick. I will give it a try.

An initial ponder leads me to the question:

If we realize our application does need the weight for optimal co-registration, is there a way to break up the afni_proc.py command to do essentially the same thing overall, but keeping use of 'align_epi_anat' instead?
EDIT: And using the a 1D file of motion parameters for the regress_motion part?

Currently I convert the motion parameters as follows:

1d_tool.py \
-overwrite \
-infile $AnimalID/$FolderID/dfile_VolReg.1D \
-volreg2allineate \
-write $AnimalID/$FolderID/dfile_VolReg_aff12.1D

rickr · August 28, 2024, 7:16pm

Because you are leaving the EPI in original space, there is no harm in breaking the process into pieces, one up through volreg, one after.

Or, you could pass the needed files (.BRIK, .HEAD or .nii.gz) via -copy_files, and edit the proc script to use that file for a volreg weight.

Additionally, you could get me to add such an option to the program. I won't guarantee doing that right away, but it seems like it would be reasonable to add. But note that using an external weight volume probably means you should be using an external registration base (which aligns with the weight) via -volreg_base_dset.

rick

e.baldelomar · October 10, 2024, 9:01pm

Hi Rick,

Moving forward with breaking up the process into pieces after doing volreg independently: I can't figure out how to appropriately use my motion parameter files with afni_proc.py. Currently I output .1D files from 3dvold reg. and I also convert it using the following function:

1d_tool.py \
-overwrite \
-infile $AnimalID/$FolderID/dfile_VolReg3.1D \
-volreg2allineate \
-write $AnimalID/$FolderID/dfile_VolReg3_aff12.1D

I've gone through the documentation and I see 'regress_stim_times' and 'regress_make_ideal_sum' as options that allow text or 1d inputs but they don't seem like the correct options. I also see '-regress_apply_mot_types' but again not sure how to input a file with it, nor if it would do the regressing correctly. What do you recommend?

Very much appreciate all your help!

e.baldelomar · October 11, 2024, 6:25pm

After looking through the documentation again, I think I found something to help me, but am still getting an error:

Current code is:

afni_proc.py \
-subj_id $AnimalID \
-dsets $epis \
-copy_anat $anat \
-anat_has_skull no \
-anat_follower_ROI mMask epi $ROI \
-blocks mask scale regress \
-radial_correlate_blocks tcat regress \
-tcat_remove_first_trs 2 \
-mask_epi_anat yes \
-regress_apply_mot_types basic \
-regress_motion_file dfile_VolReg.1D \
-regress_ROI mMask \
-regress_est_blur_errts \
-html_review_style pythonic \
-execute

But I get the following error:

++ 3dTproject: AFNI version=AFNI_24.2.01 (Jul 16 2024) [64-bit]
++ Authored by: Cox the Algebraic (Linear)
** FATAL ERROR: Can't open dataset 'RH2401_C09.results/errts.RH2401_C09.tproject+orig'

I thought it might have to do with the line '-regress_est_blur_errts' but I tried without that and still got the same error.

Any insight to move forward would be greatly appreciated.

Cheers,
Edwin

p.s. For this I did not run '-blocks despike align' in a separate algorithm, I thought I could just do it these through '3dDespike' and '3dVolreg'...

e.baldelomar · October 15, 2024, 1:40pm

Hi All,

Just following up to see if anyone had a solution to the above question. Any guidance is very much appreciated.

Cheers,
Edwin

rickr · October 15, 2024, 5:58pm

Hi Edwin,

I tried out your exact command, but with datasets from our class data, and it worked fine. That makes me wonder if there is some NIFTI dataset that is input and is showing up as in standard space, while the rest of the data is in original space.
What is the output from (fill in the variables):

3dinfo -av_space -prefix $epis $anat $ROI

It seems most likely that the $ROI dataset might not be orig.

-rick

e.baldelomar · October 18, 2024, 8:20pm

Hi Rick,

Thanks so much for your time into this. I tried your command and each are listed as '+orig'.

Because this isn't typical brain data, I did interpret what I should put in for anat, epis, and ROI. epis is standard, so I imagine no issue there. But for anat, I use an average 3D image calculated from epis. I imagine that's probably different than normal use, so not sure if that would cause an issue. And then for ROI, I use the plugin 'Draw Dataset' to make the mask.

Also curious if my input 1D file is an issue. It only has 6 columns. Not sure if it needs more (like all 12 potential 3dvolreg degrees of freedom)

Not sure if that information is helpful but just trying to provide any information that may be helpful since you said you use datasets from your class data and it worked.

Cheers,
Edwin

e.baldelomar · October 18, 2024, 8:54pm

Update:

I tried the following and got it to run:

afni_proc.py \
-subj_id $AnimalID \
-dsets $epis \
-anat_follower_ROI mMask epi $ROI \
-blocks regress \
-radial_correlate_blocks tcat regress \
-tcat_remove_first_trs 2 \
-regress_apply_mot_types demean deriv \
-regress_motion_file $dfile \
-regress_ROI mMask \
-html_review_style pythonic \
-execute

I did see the following at the end which I'm not sure means it's still missing anything:

gen_ss_review_scripts.py -exit0 -init_uvars_json out.ap_uvars.json -write_uvars_json out.ss_review_uvars.json
** failed to guess at any stats dset, resting state?
   (else X-matrix file "X.xmat.1D" may not apply)
** failed to find volreg dset, continuing...
** failed to find motion enorm dset
** failed to init basics...

Also, looking into the '.results' folder. I see a 'errts.*.tproject' dataset. Is this my 'regressed' dataset?

rickr · October 24, 2024, 3:08pm

Hi Edwin,

Trying to catch up on some threads here...

Yes, the errts*.tproject dataset should be the one with everything projected out.
I am a little surprised about it failing to find a motion enorm dataest, given that you pass the motion file. I will try to test that out...

-rick

rickr · October 25, 2024, 5:26pm

Yes, apparently you need either a volreg block or regression censoring to get that file, which might be needed to get the QC.

I will ponder that, but at least you should have the results.

rick

e.baldelomar · October 29, 2024, 11:46pm

Hi Rick,

No worries. I imagine it takes a lot to go through all the threads. I'm super appreciative for your time and help.

Also, can you confirm a few things for me, please:

which file would be like a final dataset to use moving forward? It looks like if I use other blocks like scale, the 'errts*' dataset doesn't change but the 'allruns*' seems to change. And there's a 'pb01*scale' file that comes up with scale. What is that?
Do I still need to detrend the final dataset determined in (1)?

Just trying to confirm necessary steps before calculating power spectra. Thank you again!

Cheers,
Edwin

rickr · October 30, 2024, 9:16pm

Hi Edwin,

Sorry, but would you please post the afni_proc.py command that this refers to, or if it is already above, say which one? Is this the "Current code is:" version?

Thanks,

-rick

e.baldelomar · October 30, 2024, 9:55pm

Hi Rick,

Correct, but I'll repost it below:

afni_proc.py \
-subj_id $AnimalID \
-dsets $epis \
-anat_follower_ROI mMask epi $ROI \
-blocks scale regress \
-radial_correlate_blocks tcat regress \
-tcat_remove_first_trs 2 \
-regress_apply_mot_types demean deriv \
-regress_motion_file $dfile \
-regress_ROI mMask \
-html_review_style pythonic \
-execute

Thanks!
Edwin

rickr · October 31, 2024, 1:28am

Hi Edwin,

Note that all_runs is the concatenation of the datasets input to 3dDeconvolve (though maybe you have only one run), while the errts is the residuals output be 3dDeconvolve. If you are using the regression to remove signals of no interest, then the errts dataset will be used going forward.

Using -blocks mask scale regress, the scale block would be the only "preprocessing" of the data before the linear regression, leaving pb01*scale datasets as being input to 3dDeconvolve (review this in the proc script).

The 3dDeconvolve command in the proc script should show you what polynomial is being used to model the slow drift/trend (-polort VALUE).

rick

e.baldelomar · November 12, 2024, 5:56pm

Thank you very much Rick! I'm really grateful for all your help!