afni_proc.py example 11b

osman · April 20, 2022, 11:25pm

Hi i try to do resting-state fmri analysis for a 25 subject. when i run the example of afni_proc.py 11b i get this output:

osman@osman-VirtualBox:~/sub08224$ afni_proc.py \

    -subj_id FT.11b.rest                                      \
    -blocks despike tshift align tlrc volreg blur mask scale  \
        regress                                               \
    -copy_anat anat+orig                                      \
    -dsets func+orig.HEAD                                     \
    -tcat_remove_first_trs 2                                  \
    -align_opts_aea -cost lpc+ZZ                              \
    -tlrc_base TT_N27+tlrc                                    \
    -tlrc_NL_warp                                             \
    -volreg_align_to MIN_OUTLIER                              \
    -volreg_align_e2a                                         \
    -volreg_tlrc_warp                                         \
    -volreg_warp_dxyz 2.5                                     \
    -blur_size 4                                              \
    -mask_segment_anat yes                                    \
    -mask_segment_erode yes                                   \
    -mask_import Tvent template_ventricle_2.5mm+tlrc          \
    -mask_intersect Svent CSFe Tvent                          \
    -mask_epi_anat yes                                        \
    -regress_motion_per_run                                   \
    -regress_ROI_PC Svent 3                                   \
    -regress_ROI_PC_per_run Svent                             \
    -regress_make_corr_vols WMe Svent                         \
    -regress_anaticor_fast                                    \
    -regress_censor_motion 0.2                                \
    -regress_censor_outliers 0.05                             \
    -regress_apply_mot_types demean deriv                     \
    -regress_est_blur_epits                                   \
    -regress_est_blur_errts                                   \
    -regress_run_clustsim yes

++ applying anaticor without label, using 3dSeg mask
warning: removing first 2 TRs from beginning of each run
--> the stimulus timing files must reflect the removal of these TRs
-- will use min outlier volume as motion base
cannot evaluate view of missing input: template_ventricle_2.5mm+tlrc
in/out view change: NO-DSET -> +orig: mask_import_Tvent
-- tcat: reps is now 195
++ updating polort to 3, from run len 390.0 s
-- using non-linear template alignment
-- volreg: using base dset vr_base_min_outlier+orig
++ volreg: applying volreg/epi2anat/tlrc xforms to isotropic 2.5 mm tlrc voxels
-- applying anat warps to 1 dataset(s): anat
++ mask: using epi_anat mask in place of EPI one
-- masking: group anat = 'TT_N27+tlrc', exists = 1
3dinfo -d3: cannot get val list from NO-DSET NO-DSET NO-DSET, for dset template_ventricle_2.5mm+tlrc
bad dims for -mask_import dataset:
template_ventricle_2.5mm+tlrc
import dims = None, analysis dims = [2.5, 2.5, 2.5]
script creation failure for block 'mask'

** failed command (create_script):

afni_proc.py -subj_id FT.11b.rest -blocks despike tshift align tlrc volreg
blur mask scale regress -copy_anat anat+orig -dsets func+orig.HEAD
-tcat_remove_first_trs 2 -align_opts_aea -cost lpc+ZZ -tlrc_base
TT_N27+tlrc -tlrc_NL_warp -volreg_align_to MIN_OUTLIER
-volreg_align_e2a -volreg_tlrc_warp -volreg_warp_dxyz 2.5 -blur_size 4
-mask_segment_anat yes -mask_segment_erode yes -mask_import Tvent
template_ventricle_2.5mm+tlrc -mask_intersect Svent CSFe Tvent
-mask_epi_anat yes -regress_motion_per_run -regress_ROI_PC Svent 3
-regress_ROI_PC_per_run Svent -regress_make_corr_vols WMe Svent
-regress_anaticor_fast -regress_censor_motion 0.2
-regress_censor_outliers 0.05 -regress_apply_mot_types demean deriv
-regress_est_blur_epits -regress_est_blur_errts -regress_run_clustsim
yes

ptaylor · April 21, 2022, 1:55pm

Hi, Osman-

I suggest you copy+paste the following text (which is your command, from above, just spaced out fairly uniformly) into a text file, say one called “do_ap.tcsh”, so it is easier to fix and rerun, and generalize to process more subjects over time (the first line, which starts with odd characters, should be the very first thing in the new text file created):


#!/bin/tcsh

afni_proc.py                                                                 \
    -subj_id                  FT.11b.rest                                    \
    -blocks                   despike tshift align tlrc volreg blur mask     \
                              scale regress                                  \
    -copy_anat                anat+orig                                      \
    -dsets                    func+orig.HEAD                                 \
    -tcat_remove_first_trs    2                                              \
    -align_opts_aea           -cost lpc+ZZ                                   \
    -tlrc_base                TT_N27+tlrc                                    \
    -tlrc_NL_warp                                                            \
    -volreg_align_to          MIN_OUTLIER                                    \
    -volreg_align_e2a                                                        \
    -volreg_tlrc_warp                                                        \
    -volreg_warp_dxyz         2.5                                            \
    -blur_size                4                                              \
    -mask_segment_anat        yes                                            \
    -mask_segment_erode       yes                                            \
    -mask_import              Tvent template_ventricle_2.5mm+tlrc            \
    -mask_intersect           Svent CSFe Tvent                               \
    -mask_epi_anat            yes                                            \
    -regress_motion_per_run                                                  \
    -regress_ROI_PC           Svent 3                                        \
    -regress_ROI_PC_per_run   Svent                                          \
    -regress_make_corr_vols   WMe Svent                                      \
    -regress_anaticor_fast                                                   \
    -regress_censor_motion    0.2                                            \
    -regress_censor_outliers  0.05                                           \
    -regress_apply_mot_types  demean deriv                                   \
    -regress_est_blur_epits                                                  \
    -regress_est_blur_errts                                                  \
    -regress_run_clustsim     yes

Then, when you want to process this subject, you can type the following single line in the terminal:


tcsh do_ap.tcsh

Looking at your warnings and error messages, let’s pick out a few to unpack:

This one with double asterisks can be fine—it is just reminding you to make sure that any associated stimulus timing files are also adjusted to match this removal of two initial time points (and since your run is like a resting state processing, with no stim files, then this should not be an issue to even need to verify):


** warning: removing first 2 TRs from beginning of each run
--> the stimulus timing files must reflect the removal of these TRs

this suggests that this file “template_ventricle_2.5mm+tlrc” does not exist on your system:


** cannot evaluate view of missing input: template_ventricle_2.5mm+tlrc
** in/out view change: NO-DSET -> +orig: mask_import_Tvent

This is not a file distributed with AFNI, so it would have to exist on your computer; how one could make this file is actually described at the top of this example 11b section here in the AP help:
https://afni.nimh.nih.gov/pub/dist/doc/htmldoc/programs/afni_proc.py_sphx.html#example-11b-similar-to-11-but-without-freesurfer

these error message also relates to that same issue (a mask can’t be made because the file doesn’t exist):


** 3dinfo -d3: cannot get val list from NO-DSET NO-DSET NO-DSET, for dset template_ventricle_2.5mm+tlrc
** bad dims for -mask_import dataset:
template_ventricle_2.5mm+tlrc
import dims = None, analysis dims = [2.5, 2.5, 2.5]
** script creation failure for block 'mask'

To solve the above issue, you could follow the instructions at the top of the help section, which I noted above and copy+paste here:


AFNI currently does not have a good program to extract ventricles.
But it can make a CSF mask that includes them.  So without FreeSurfer,
one could import a ventricle mask from the template (e.g. for TT space,
using TT_desai_dd_mpm+tlrc).  For example, assuming Talairach space
(and a 2.5 mm^3 final voxel grid) for the analysis, one could create a
ventricle mask as follows:

       3dcalc -a ~/abin/TT_desai_dd_mpm+tlrc                       \
              -expr 'amongst(a,152,170)' -prefix template_ventricle
       3dresample -dxyz 2.5 2.5 2.5 -inset template_ventricle+tlrc \
              -prefix template_ventricle_2.5mm

Running that 3dcalc and 3dresample command should create template_ventricle_2.5mm, which you should then be able to use in your AP command.

–pt

osman · April 21, 2022, 9:06pm

Thank you ptylor
i did the preprocessing for one subject but when underlay the final anatomic data and overlay the error time series “errts.FT.11b.rest.fanaticor” i got a data match error as shown in the picture. so I can’t go to the next step which is creating correlation maps.

ptaylor · April 21, 2022, 9:12pm

Hi, Osman-

Hmm, something might have gone wrong with one of hte alignment steps? There should be an automatically-generated QC HTML in the output results directory.

Can you see if there is a directory there starting with “QC_”, and then run:


afni_open -b QC_*/index.html &

? That should open the QC HTML in an available browser.

More description on the main QC sections are provided here (though this page is a bit out of date…):
https://afni.nimh.nih.gov/pub/dist/doc/htmldoc/tutorials/apqc_html/main_toc.html
But specifically, check the ve2a (volumetric EPI-to-anatomical alignment) and va2t (volumetric anatomical-to-template alignment), and see how each of those look. My guess is that the EPI-to-anatomical alignment might not have gone well?

Also, what version of AFNI are you using? What is “afni -ver”?

–pt

osman · April 21, 2022, 9:48pm

i foun a folder in name of QC_FT.11b.rest in the subject results directory and after running the command above “afni_open -b QC_*/index.html &”
QC HTML web page was opend i printed this web page into a pdf file which including informations about the allignment and preprocessing stage but i did not understand what to do ; i sent the file by the e-mail
my afni version is: Precompiled binary linux_ubuntu_16_64: Mar 10 2022 (Version AFNI_22.0.17 ‘Hadrian’)
thank you

ptaylor · April 22, 2022, 1:28am

Hi, Osman-

Thanks for sending a copy of some of the APQC HTML. Attached is the critical part that I suggested looking at—the ve2a, which shows the EPI-anatomical alignment. Because your initial EPI and anatomical are a bit far apart (shown in the vorig block of the APQC), I suggest adding the option “-giant_move” to your afni_proc.py command in the ‘-align_opts_aea …’ part of controlling alignment of EPI and anatomical (= AEA), like this:


#!/bin/tcsh

afni_proc.py                                                                 \
    -subj_id                  FT.11b.rest                                    \
    -blocks                   despike tshift align tlrc volreg blur mask     \
                              scale regress                                  \
    -copy_anat                anat+orig                                      \
    -dsets                    func+orig.HEAD                                 \
    -tcat_remove_first_trs    2                                              \
    -align_opts_aea           -cost lpc+ZZ   -giant_move                                \
    -tlrc_base                TT_N27+tlrc                                    \
    -tlrc_NL_warp                                                            \
    -volreg_align_to          MIN_OUTLIER                                    \
    -volreg_align_e2a                                                        \
    -volreg_tlrc_warp                                                        \
    -volreg_warp_dxyz         2.5                                            \
    -blur_size                4                                              \
    -mask_segment_anat        yes                                            \
    -mask_segment_erode       yes                                            \
    -mask_import              Tvent template_ventricle_2.5mm+tlrc            \
    -mask_intersect           Svent CSFe Tvent                               \
    -mask_epi_anat            yes                                            \
    -regress_motion_per_run                                                  \
    -regress_ROI_PC           Svent 3                                        \
    -regress_ROI_PC_per_run   Svent                                          \
    -regress_make_corr_vols   WMe Svent                                      \
    -regress_anaticor_fast                                                   \
    -regress_censor_motion    0.2                                            \
    -regress_censor_outliers  0.05                                           \
    -regress_apply_mot_types  demean deriv                                   \
    -regress_est_blur_epits                                                  \
    -regress_est_blur_errts                                                  \
    -regress_run_clustsim     yes

Also, I don’t see the motion plots in your QC. I was a bit surprised by that. What is the output of:


afni_system_check.py -check_all

?

–pt

osman · April 28, 2022, 4:15pm

Hi ptaylor. I add the option “-giant_move” to my afni_proc.py command and run the afni_proc.py command as you suggest above in the last replay. then i underlayed the final anatomic data and overlayed the error time series “errts.s01.fanaticor” like shown in the picture attached below. i applied a thershhold value to the error time series. is my preprocessing step is alright? because i see correlation in every where except the brain!. am i alright to continue to the next steps?
and i sent a copy of the resulted APQC HTML to the e-mail
i also attached the output of afni_system_check.py -check_all
thank yot ptaylor.

ptaylor · April 28, 2022, 4:42pm

Hi, Osman-

To check the alignment specifically before looking at stats, I looked at the ve2a QC images in the APQC HTML you sent. It looks quite good now (see attached).

It looks like you are performing a resting state analysis, because you don’t have any stimulus files/GLTs in your afni_proc.py command. There is some QC for that in the APQC HTML as well—the seedbased correlation (DMN seed) image is attached—it looks OK; the correlation might be a bit low in some places, but with the transparent thresholding we see “orange” in places we would expect.

I thiiiink your shared image here is of the actual time series values of one EPI, just one of the EPI volumes. Am I understanding this correctly? Because I don’t think it is really “correlation” being shown, which you commented about in your question. With scaling on, I think one might expect the noise outside the brain to have large values like that above some kind of threshold. I don’t think that is really a worry or sign of badness. Using something like seedbased correlation would be a better test of data quality.

–pt

rickr · April 28, 2022, 5:06pm

Indeed, as Paul notes, this is the errts dataset, volume #0. These are not correlations but regression residual values, and the the ones outside the brain tend to be larger. Do not judge alignment from this.

rick

osman · April 28, 2022, 11:27pm

Hi ptaylor;
actually i said correlation but i didn’t mean it exactly . what i did so far is the preprocessing step using the afni_proc.py . then i underlayed the resulted anatomic data and overlayed the resulted fanaticor error time series and applied a thershhold value. so it is not a correlation it is kind of signal seen in every where except the brain. so i wonderd if i did the preprocessing step correctly? so i can pass to the next step which it is creating roi and correlation maps.

osman · April 28, 2022, 11:30pm

can you sir explain what is the nature of this signals which seen in every where except the brain?

ptaylor · April 29, 2022, 12:49pm

Hi, Osman-

This is a pretty large topic, and I think the best place to start would be with some of the underlying theory of FMRI modeling, which we have some AFNI Academy videos about here:
https://www.youtube.com/c/AFNIBootcamp/playlists?view=50&sort=dd&shelf_id=4
Somewhat more focusedly, in the afni_proc.py start-to-finish videos:
https://www.youtube.com/watch?v=XNPNAgpmjxI&list=PL_CD549H9kgpGgC8xWFexw-8R95HZokAr
… Rick goes through a single subject task-based FMRI analysis with afni_proc.py; resting state has some differences, but looking at the scaling description will be relevant to what is here (and subsequently):
https://www.youtube.com/watch?v=P83ZCuKPQ0o&list=PL_CD549H9kgpGgC8xWFexw-8R95HZokAr&index=12
Really, going through the theory and start-to-finish video would be quite useful.
Going through videos #30 and 31 here (from an AFNI Bootcamp we taught at MIT):
https://cbmm.mit.edu/afni
… would be useful for some resting-state specific processing issues.

As a short answer: there is data everywhere in the dataset field of view, both inside the brain and outside the brain. We measure ‘signal’ response in a general sense of brain, ventricle, skull, face, and air. Some of those areas are more useful to focus on for our scientific questions than others—but performing statistical modeling throughout the whole volume and looking at results everywhere is reeeeaaallly useful for fully understanding the acquired data and avoiding potential acquisition/processing pitfalls. Hence, we don’t mask the data at this stage.

The data outside the brain is reaaallly noisy, so it can have veeery large features in it, esp in the residual time series, which is what the errts* dataset is. Trying to gauge goodness/badness from just the values in the errts is not a good road to be going down, because of scaling in particular. Looking at seedbased correlation from a seed within the brain is better—that is where the final analyses will be based, anyways. So, understanding the FMRI-based modeling, the difference between fitts and errts time series (the former being more used in task-based FMRI and the latter in resting state FMRI), and how each can be used to answer specific research/clinical questions, would be good to aims. Those are larger topics than can be adequately typed about here—going through those videos would be useful for that, and subsequent questions could be addressed here.

-pt

osman · April 29, 2022, 2:23pm

Hello there;
what i am trying to say is it normal to see this signals outside of the brain in the error time series and what is this signals?

ptaylor · April 29, 2022, 6:52pm

Hi, Osman-

Both Rick’s reply and my own provide details on this.

In brief: yes, one will see signal in the unmasked errts file throughout the entire field of view.

In detail: please see those Bootcamp lecture videos for more details on why.

–pt