QC output question

annschoe · March 2, 2021, 8:58pm

Good afternoon,

I have two questions regarding the quality control report that is generated after running afni_proc.py, which I am using to preprocess my resting state fMRI data set. Please see below for my afni_proc.py


	afni_proc.py -subj_id ${subj} \
	-script proc.${subj} -scr_overwrite \
	-blocks despike tshift align tlrc volreg blur mask regress \
	-out_dir $outputDir \
	-copy_anat $sdir/${anatsubj}_SurfVol+orig \
	-anat_follower_ROI aaseg anat $sdir/aparc.a2009s+aseg.nii \
	-anat_follower_ROI aeseg epi  $sdir/aparc.a2009s+aseg.nii \
	-anat_follower_ROI FSVente epi $sdir/${anatsubj}_vent.nii \
	-anat_follower_ROI FSWe epi $sdir/${anatsubj}_WM.nii \
	-anat_follower_erode FSVente FSWe \
	-dsets $dataIn \
	-tshift_opts_ts -tpattern seq+z -TR 2s \
	-align_opts_aea -cost lpc+ZZ -ginormous_move \
	-tlrc_base MNI152_T1_2009c+tlrc \
	-tlrc_NL_warp \
	-volreg_align_to MIN_OUTLIER \
	-volreg_align_e2a \
	-volreg_tlrc_warp \
	-mask_segment_anat yes \
	-mask_segment_erode yes \
	-blur_size 6.0 \
	-regress_motion_per_run \
	-regress_ROI_PC FSVente 3 \
	-regress_make_corr_vols aeseg FSVente \
	-regress_anaticor_fast \
	-regress_anaticor_label FSWe \
	-regress_censor_motion 0.4 \
	-regress_censor_outliers 0.1 \
	-regress_bandpass 0.01 0.1 \
	-regress_apply_mot_types demean deriv \
	-regress_est_blur_epits \
	-regress_est_blur_errts \

Question #1: When I look at the QC report generated (QC/index.html), all looks good, except the ‘qc_10_regr_grayplot.jpg’ file is an empty file. However, if I run the @ss_review_driver, I can see the enorm, outlier, and censoring information without an issue. Would you let me know what may be going on?

Question #2: When I run the @ss_review_driver, I get the following error messages.


** ERROR: mri_read_ascii: can't read any valid data from file X.stim.xmat.1D
** FATAL ERROR: Can't read input file 'X.stim.xmat.1D' iarg=2

and


------------- 3dDeconvolve.err -------------
*+ WARNING: !! in Signal+Baseline matrix:
 * Largest singular value=2.51615
 * 4 singular values are less than cutoff=2.51615e-07
 * Implies strong collinearity in the matrix columns! 
*+ WARNING: !! in Baseline-only matrix:
 * Largest singular value=2.51615
 * 4 singular values are less than cutoff=2.51615e-07
 * Implies strong collinearity in the matrix columns! 
--------------------------------------------

Am I getting these messages because this is a resting state data, or there is something else (perhaps related to my Question #1?) going on?

Question #3: This has nothing to do with the last 2 questions, but I was wondering. Would you mind explaining what would adding the following code to my afni_proc.py script would do? I have read the help file multiple times, but I’m still having trouble understanding what the code is doing.


-tlrc_NL_warped_dsets anatQQ.FT.nii anatQQ.FT.aff12.1D  anatQQ.FT_WARP.nii

Thank you in advance for your help,
Ann

ptaylor · March 3, 2021, 3:09pm

Hi, Ann-

What is your AFNI version (“afni -ver”)?

One first thing to note about your afni_proc.py command the QC: using “-html_review_style pythonic” will give you much nicer looking line plots (assuming you have Python plus the Matplotlib module installed on your computer). You can re-run the APQC in pythonic-mode once we sort the other issue(s), without having to rerun the full afni_proc.py. At the moment, the way to do this would be to run:


wget https://github.com/afni/afni/tree/master/src/ptaylor/supplement/redo_apqc.tcsh

… to download a file (that I should just add to the distribution). After that, you can run:


tcsh redo_apqc.tcsh AP_RESULTS_DIR

… where AP_RESULTS_DIR is a list of one or more results directory locations; this program will push any existing QC_${subj} dir to QC_${subj}_TIMESTAMP and create a new one.

Re. Q3 (because you did sneak a third question in!):
Adding these lines in won’t do anything for your code here:


-tlrc_NL_warped_dsets anatQQ.FT.nii anatQQ.FT.aff12.1D  anatQQ.FT_WARP.nii

… because these are for when you run a program called “@SSwarper” before afni_proc.py to do a combined skullstripping (SS) and nonlinear alignment (warping) to a template from your anatomical T1w volume. This program is run before afni_proc.py, and then you pass the results of the warping into the program—this line that you have highlighted passes the results in.

The benefit of using @SSwarper first is that 1) it tends to provide the best nonlinear alignment and skullstripping, and 2) you can check the results and re-run that part of the processing if necessary separate from the rest of things. Nonlinear warping can be slow, so doing this ahead of time is nice in case you decide to re-run afni_proc.py with other settings—you don’t have to re-do the nonlinear alignment part.

In your case, you could run this program as follows (not the different MNI template: @SSwarper requires a special, multi-brick version of the MNI template, which we distribute, more info here, if you are interested: https://afni.nimh.nih.gov/pub/dist/doc/htmldoc/template_atlas/sswarper_base.html):


@SSwarper                                     \
        -tmp_name_nice                            \
        -input   $sdir/${anatsubj}_SurfVol+orig                  \
        -base    MNI152_2009_template_SSW.nii.gz  \
        -subid   ${subj}                          \
        -odir    group/o.aw_${subj}

… The @SSwarper help tells you how to provide this info to afni_proc.py, then. Some pieces are:


    -copy_anat group/o.aw_${subj}/anatSS.${subj}.nii                               \
     -anat_has_skull no                                          \
....
      -volreg_tlrc_warp -tlrc_base MNI152_2009_template_SSW.nii.gz           \
....
      -tlrc_NL_warp                                               \
      -tlrc_NL_warped_dsets                                       \
         group/o.aw_${subj}/anatQQ.${subj}.nii                                       \
         group/o.aw_${subj}/anatQQ.${subj}.aff12.1D                                  \
         group/o.aw_${subj}/anatQQ.${subj}_WARP.nii

… where the exact paths might change.

Re. Q1: There should be a file called “out.review_html” in your *.results directory from afni_proc.py. That contains a log of all the APQC (afni_proc.py QC) generation. Could you email me that?

Re. Q2: That Xmat failure is odd… I am not sure—that is probably a Rick-level question, and I will ping him.

–pt

ptaylor · March 3, 2021, 4:24pm

Actually, Ann, what is your final degree of freedom (DF) count?

You should see it in a text block under the “regr” block in the APQC output. It would also appear in the 3rd text block here:
./@ss_review_basic

Could you please copy+paste either of those text blocks here?

–pt

annschoe · March 3, 2021, 5:26pm

Good afternoon pt,

Thank you so much for looking into the questions. Below are my responses.

-AFNI version (“afni -ver”)?
Precompiled binary macos_10.12_local: Jan 18 2021 (Version AFNI_21.0.04 ‘Titus’)

-One first thing to note about your afni_proc.py command the QC: using “-html_review_style pythonic”
I actually have this line of code in my afni_proc.py. I accidently omitted the line while copy-pasting. The QC-report I mentioned previously was generated via this line of code, and in the report, the “qc_10_regr_grayplot.jpg” file is not being generated correctly. I can see the enorm plots just fine when I run the @ssreview_driver.

-Re. Q3 (because you did sneak a third question in!)(<= I know… I did!)
Thank you so much for the explanation. It all makes sense now, and I will look to see how to use the line of code. Running the SSWarper before afni_proc.py makes a lot of sense indeed.

-Re. Q1: There should be a file called “out.review_html” in your *.results directory from afni_proc.py. That contains a log of all the APQC (afni_proc.py QC) generation. Could you email me that?
Will email right away after posting this.

Re. Q2: That Xmat failure is odd… I am not sure—that is probably a Rick-level question, and I will ping him.
Actually, Ann, what is your final degree of freedom (DF) count?


initial DF                   : 890 : 100.0%

DF used for regs of interest :   0 :   0.0%
DF used for censoring        :   0 :   0.0%
DF used for polort           :  14 :   1.6%
DF used for motion           :  24 :   2.7%
DF used for ROIPC            :   6 :   0.7%
DF used for bandpass         : 564 :  63.4%
total DF used                : 608 :  68.3%

final DF                     : 282 :  31.7%

ptaylor · March 3, 2021, 5:39pm

Hi, Ann-

Thanks for posting those, and I await that out*html file.

I will ping Rick about Q2 because your reply rules out the potential issue I wondered about. (But note how many DFs bandpassing purges… This is a processing choice note, see: file:///home/ptaylor/afni_doc/_build/html/programs/afni_proc.py_sphx.html#resting-state-note)

–pt

annschoe · March 3, 2021, 5:52pm

Hello, pt.

Thanks for pinging. And as for the bandpass issue - I did read the note, and I agree with you. However, the thought that restings state data should be band passed at 0.01-0.1Hz has been ingrained in people’s head so much so that if I say I didn’t do it, I am immediately hit back with a lot of skeptical looks and questions - something I don’t want to really deal with when I’m trying to publish. I’m curious as to if you’ve also experienced this and what your thoughts on the issue.

Thanks,
Ann

rickr · March 3, 2021, 8:30pm

Hi Ann,

Q2.1: We have waffled a bit regarding X.stim.xmat.1D, between having it full, empty and non-existent (for resting state data). Nothing makes everyone/thing happy. Right now, the file exists, but is empty, and afni (the GUI program itself) is whining about not finding valid contents. There is another similar message from 1dplot that I should change the driver to avoid.

Q2.2: For the collinearity warning, what is the output of:

1d_tool.py -show_cormat_warnings -infile X.xmat.1D

Thanks,

rick

annschoe · March 3, 2021, 8:44pm

Hello Rick,

Here is the output:


[anns-MacBook-Pro:06_PM_blur/02_afniBOLD_blur6/20190815jp] achoe2% 1d_tool.py -show_cormat_warnings -infile X.xmat.1D
-- no warnings for correlation matrix (cut = 0.400) --

Thank you,
Ann

rickr · March 3, 2021, 9:21pm

Hi Ann,

If there are no pairwise correlations, are there empty columns?
What does this show?

3dTstat -absmax -prefix - X.xmat.1D\'

rick

annschoe · March 4, 2021, 12:45am

Here is the result.


[anns-MacBook-Pro:06_PM_blur/02_afniBOLD_blur6/20190815jp] achoe2% 3dTstat -absmax -prefix - X.xmat.1D\'
++ 3dTstat: AFNI version=AFNI_21.0.04 (Jan 18 2021) [64-bit]
++ Authored by: KR Hammett & RW Cox
*+ WARNING: Input dataset is not 3D+time; assuming TR=1.0
 1
 1
 0.997748
 1
 0.997736
 1
 0.997717
 1
 1
 0.997748
 1
 0.997736
 1
 0.997717
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999844
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999845
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999845
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999846
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999845
 1
 0.999994
 1
 0.999995
 1
 0.999994
 1
 0.999994
 1
 0.999845
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999846
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999846
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999845
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999845
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999845
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999847
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999845
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999845
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999846
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999847
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999847
 1
 0.999994
 1
 0.999995
 1
 0.999994
 1
 0.999994
 1
 0.999845
 1
 0.999994
 1
 0.999994
 1
 0.999994
 0.999999
 0.999994
 1
 0.999846
 1
 0.999994
 1
 0.999994
 1
 0.951112
 1
 0.999995
 1
 0.999846
 1
 0.999994
 1
 0.999995
 1
 0.999994
 1
 0.999994
 1
 0.999847
 1
 0.999994
 1
 0.999994
 1
 0.999995
 1
 0.999994
 1
 0.999846
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999848
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999846
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999847
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999846
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999848
 1
 0.999994
 1
 0.999995
 1
 0.999994
 1
 0.999994
 1
 0.999846
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999844
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999845
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999845
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999846
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999845
 1
 0.999994
 1
 0.999995
 1
 0.999994
 1
 0.999994
 1
 0.999845
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999846
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999846
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999845
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999845
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999845
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999847
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999845
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999845
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999846
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999847
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999847
 1
 0.999994
 1
 0.999995
 1
 0.999994
 1
 0.999994
 1
 0.999845
 1
 0.999994
 1
 0.999994
 1
 0.999994
 0.999999
 0.999994
 1
 0.999846
 1
 0.999994
 1
 0.999994
 1
 0.951112
 1
 0.999995
 1
 0.999846
 1
 0.999994
 1
 0.999995
 1
 0.999994
 1
 0.999994
 1
 0.999847
 1
 0.999994
 1
 0.999994
 1
 0.999995
 1
 0.999994
 1
 0.999846
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999848
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999846
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999847
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999846
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999994
 1
 0.999848
 1
 0.999994
 1
 0.999995
 1
 0.999994
 1
 0.999994
 1
 0.999846
 1
 0.999994
 1
 0.999994
 0.1727
 0.148799
 0.134701
 0.126303
 0.169303
 0.144824
 0.531042
 0.25859
 0.601153
 0.405391
 0.422891
 1.51853
 0.135378
 0.28019
 0.221123
 0.402385
 0.095444
 1.13476
 0.131198
 0.163189
 0.130731
 0.337466
 0.127112
 0.175764
 0.0481973
 0.188827
 0.0728099
 0.298233
 0.035949
 0.120133

rickr · March 4, 2021, 2:43pm

None of those are zero. I should have given you a more complex command, to take the min of those absmax values to look for zero.

3dTstat -absmax -prefix - X.xmat.1D\' | 3dTstat -prefix - -min 1D:stdin\'

or even
[code3dTstat -absmax -prefix - X.xmat.1D' | sort -n | head -n 3



Anyway, there is probably something more complicated going on.  Would it be possible for you to mail me that Xmat file?  Clicking on my name should show my email address.

Thanks,

- rick

rickr · March 4, 2021, 7:43pm

Hi Ann,

Thank you for the X-matrix. The simple answer is that this seems fine.

In more detail, there are 2 points to ponder, one including our examples that I have not bothered to be picky about.

When including a high-pass filter (the 0.1 lower bound on your -regress_bandpass option), it is probably sufficient to include polorts up to 2. From 3 on up, they start to be well fit by the bandpass terms (causing higher condition numbers). So in your case, consider including “-regress_polort 2”. I will be more careful in examples of that, and may even have afni_proc.py warn about it. We’ll see…
The A/P motion terms for each run show a clear and consistent drift (possibly due to slowly sinking into a pillow). Those are very highly correlated with the linear polort drift terms.

Both of those contribute to the high condition numbers that 3dDeconvolve is whining about.
Neither is very concerning for the analysis.

Does that seem reasonable?

rick

annschoe · March 4, 2021, 8:29pm

Hello Rick,

Yes, indeed. Thank you very much for taking a look into the matter. I really appreciate it!

Sincerely,
Ann