Error with generating HTML QC reports

AFNI Message Board
1

Hello AFNI gurus!
I am running preprocessing on some data and was hoping to generate the the HTML quality control reports using:


apqc_make_tcsh.py -review_style pythonic -subj_dir . -uvar_json out.ss_review_uvars.json

as suggested here https://afni.nimh.nih.gov/pub/dist/doc/htmldoc/tutorials/apqc_html/overview.html.

But I’m getting the following error:
++ Found 39 files for QCing.
** ERROR: radcor number of ulays (0) doesn’t match number of olays (6)

Why might this be happening? I have BRIK/HEAD files for all runs in radcor.pb00.tcat/ and radcor.pb02.volreg/ that are called epi.ulay.r0?. Should these be used as underlays?

Here is the afni_proc.py command I’m using:


afni_proc.py -subj_id sub-001 -script proc.sub-001 -scr_overwrite           \
            -blocks tshift align tlrc volreg mask blur scale regress                   \
            -radial_correlate yes                                                                       \
            -radial_correlate_blocks tcat volreg                                                \
            -copy_anat /raid-09/LS/hannula/mrinmayi/ConcealMemory/derivatives/AFNI_01_Search_SSwarper/sub-001/anatSS.sub-001.nii                                             \
            -anat_has_skull no                                                                           \
            -anat_follower anat_w_skull anat /raid-09/LS/hannula/mrinmayi/ConcealMemory/derivatives/AFNI_01_Search_SSwarper/sub-001/anatU.sub-001.nii                     \
            -dsets /raid-09/LS/hannula/mrinmayi/ConcealMemory/rawdata/sub-001/func/sub-001_task-exp_run-01_bold.nii /raid-09/LS/hannula/mrinmayi/ConcealMemory/rawdata/sub-001/func/sub-001_task-exp_run-02_bold.nii /raid-09/LS/hannula/mrinmayi/ConcealMemory/rawdata/sub-001/func/sub-001_task-exp_run-03_bold.nii /raid-09/LS/hannula/mrinmayi/ConcealMemory/rawdata/sub-001/func/sub-001_task-exp_run-04_bold.nii /raid-09/LS/hannula/mrinmayi/ConcealMemory/rawdata/sub-001/func/sub-001_task-exp_run-05_bold.nii /raid-09/LS/hannula/mrinmayi/ConcealMemory/rawdata/sub-001/func/sub-001_task-exp_run-06_bold.nii                                               \
            -tcat_remove_first_trs 4                                                                   \
            -tcat_remove_last_trs 0                                                                   \
            -tshift_opts_ts -tpattern alt+z2                                                         \
            -align_opts_aea -big_move -cost lpc+ZZ -check_flip                      \
            -tlrc_base MNI152_2009_template_SSW.nii.gz                              \
            -tlrc_NL_warp                                                                                  \
            -tlrc_NL_warped_dsets /raid-09/LS/hannula/mrinmayi/ConcealMemory/derivatives/AFNI_01_Search_SSwarper/sub-001/anatQQ.sub-001.nii /raid-09/LS/hannula/mrinmayi/ConcealMemory/derivatives/AFNI_01_Search_SSwarper/sub-001/anatQQ.sub-001.aff12.1D /raid-09/LS/hannula/mrinmayi/ConcealMemory/derivatives/AFNI_01_Search_SSwarper/sub-001/anatQQ.sub-001_WARP.nii    \
            -volreg_align_to MIN_OUTLIER                                                      \
            -volreg_align_e2a                                                                            \
            -volreg_tlrc_warp                                                                             \
            -mask_epi_anat yes                                                                        \
            -blur_size 4                                                                                      \
            -blur_in_mask yes                                                                           \
            -regress_motion_per_run                                                                \
            -regress_censor_motion 0.3                                                            \
            -regress_censor_outliers 0.05                                                         \
            -regress_apply_mot_types demean deriv                                       \
            -html_review_style pythonic

And here is the output for afni -ver
Precompiled binary linux_openmp_64: Jul 3 2019 (Version AFNI_19.2.01 ‘Claudius’)

Just fyi, the QC scripts aren’t being run on their own in the afni_proc.py command because I’m running the preprocessing on a cluster, and we have to source a bunch of additional modules outside the job scheduler for it to be able to generate these plots.
Thank you,
Mrinmayi

2

Hi, Mrinmayi-

Just to check-- are you running a current apqc_make_tcsh.py on old afni_proc.py results (or, at least, afni_proc.py results from a version of AFNI from 2 years ago)? I don’t know that everything will mix there, unfortunately. In general, I think you that there might have been some changes in the intervening years that make that difficult. The typical assumption is that the apqc_make_tcsh.py and afni_proc.py version come from the same (or similar) AFNI versions.

Once you have a computer set up for running AFNI, I think the only “extra” dependency that the Pythonic APQC HTML has is python’s matplotlib (which, in python 2.7 might also require the “pillow” package to be able to make JPGs, oddly enough; in Python 3.*, this extra dependency doesn’t seem to occur). And anymore, I would just consider matplotlib part of the AFNI install. Does that not exist on the server that you are using?

And one potential solution might be trying to first re-run the gen_ss_review_scripts.py that creates a dictionary of information about files in the results directory (output in a JSON file), which apqc_make_tcsh.py uses. This is done just before apqc_make_tcsh.py is called in the proc.* script output.

Can you try copying your full results folder for your present subject to a new dir called, like “test.results” or something (just so we don’t mess up any files in your current output), and then put the following “redo_gen_and_apqc.tcsh” script into that same directory:


#!/bin/tcsh

# these 3 variables need to be filled in with subject-specific
# information
set subj      = sub-001 
set mot_limit = 0.3
set out_limit = 0.05

# -----------------------------------------------------------------------
# regenerate out*.json dictionary of values for APQC

gen_ss_review_scripts.py                                        \
    -mot_limit ${mot_limit} -out_limit ${out_limit}             \
    -exit0                                                      \
    -ss_review_dset out.ss_review.$subj.txt                     \
    -write_uvars_json out.ss_review_uvars.json

# not in AP proc script; this was added here to copy any existing QC
# directory to a backup "old" one
set pre_qc = `find . -maxdepth 1 -type d -name "QC*"`
if ( "${pre_qc}" != "" ) then
    set pre_qc_name = `echo ${pre_qc} | cut -b3-`
    set thedate = `date +%Y_%m_%d_%H_%M_%s`
    \mv ${pre_qc} old_${pre_qc_name}_${thedate}
endif

./@ss_review_basic |& tee out.ss_review.$subj.txt

# generate html ss review pages
# (akin to static images from running @ss_review_driver)
apqc_make_tcsh.py -review_style basic -subj_dir . \
    -uvar_json out.ss_review_uvars.json
tcsh @ss_review_html |& tee out.review_html
apqc_make_html.py -qc_dir QC_$subj

I think I have put the correct information for your subject here, based on your afni_proc.py command, but you would have to include the correct info for any subject in the future. Running this basically does the final steps of what afni_proc.py’s proc.* script would do, with the addition of moving/backing-up any existing QC directory. The idea is to have a JSON file of infomation that should match the expectations of version of AFNI you are using the other programs being run here. If this script is put into the “test.results” directory, you can type “tcsh redo_gen_and_apqc.tcsh” and see if that generates all the APQC.

Please let me know how that goes.

–pt

3

Hi Paul,
Thanks a lot for your response!

Actually both the preprocessing and the apqc_make_tcsh.py command are being run using the same version of AFNI from 2 years ago (it’s the one we have installed on our clusters). And the results folder does have a .json file in it. This is what is in that json file:


   "afni_package": "linux_openmp_64", 
   "afni_ver": "AFNI_19.2.01", 
   "align_anat": "sub-002_al_keep+orig.HEAD", 
   "censor_dset": "censor_sub-002_combined_2.1D", 
   "cormat_warn_dset": "out.cormat_warn.txt", 
   "df_info_dset": "out.df_info.txt", 
   "enorm_dset": "motion_sub-002_enorm.1D", 
   "errts_dset": "errts.sub-002.tproject+tlrc.HEAD", 
   "final_anat": "anat_final.sub-002+tlrc.HEAD", 
   "final_epi_dset": "final_epi_vr_base_min_outlier+tlrc.HEAD", 
   "final_view": "tlrc", 
   "flip_check_dset": "aea_checkflip_results.txt", 
   "flip_guess": "NO_FLIP", 
   "gcor_dset": "out.gcor.1D", 
   "have_radcor_dirs": "yes", 
   "mask_corr_dset": "out.mask_ae_dice.txt", 
   "mask_dset": "mask_epi_anat.sub-002+tlrc.HEAD", 
   "mot_limit": 0.3, 
   "motion_dset": "dfile_rall.1D", 
   "name": "user variables", 
   "nt_applied": 346, 
   "nt_orig": 348, 
   "num_stim": 0, 
   "out_limit": 0.05, 
   "outlier_dset": "outcount_rall.1D", 
   "pre_ss_warn_dset": "out.pre_ss_warn.txt", 
   "rm_trs": 4, 
   "ss_review_dset": "out.ss_review.sub-002.txt", 
   "stats_dset": "NO_STATS", 
   "subj": "sub-002", 
   "sum_ideal": "sum_baseline.1D", 
   "tcat_dset": "pb00.sub-002.r01.tcat+orig.HEAD", 
   "template": "MNI152_2009_template_SSW.nii.gz", 
   "tr": 2.0, 
   "tsnr_dset": "TSNR.sub-002+tlrc.HEAD", 
   "volreg_dset": "pb02.sub-002.r01.volreg+tlrc.HEAD", 
   "vr_base_dset": "vr_base_min_outlier+orig.HEAD", 
   "xmat_regress": "X.xmat.1D", 
   "xmat_uncensored": "X.nocensor.xmat.1D"

I tried running the steps from your post above and everything else seems to run successfully, but I’m still getting same error for the apqc_make_tcsh.py command (with the exception that before I ran those steps in your message, it said it had 39 files for QC-ing!):
++ Found 40 files for QCing.
** ERROR: radcor number of ulays (0) doesn’t match number of olays (6)

Side note: A friend of mine (Jenna from UWM) is doing the same thing on her data on the cluster using the same AFNI version, i.e., running afni_proc.py and then separately running apqc_make_tcsh.py commands- and she was able to generate the HTML files. She has one run of resting state data, and her afni_proc.py command is almost identical to Example 11 on the afni_proc help page. Haha! Not sure if that’s helpful for you to know, but that might suggest it has something to do with my data?
Thank you,
Mrinmayi

4

Hi, Mrinmayi-

Well, it’s good that the same versions of code are being used. I note that it would be better if they were modern versions—there have surely been a lot of changes (and improvements) in the APQC HTML in the past 2 years. It is also hard to troubleshoot code that is that old…

But I will share a Box directory with you, if you could upload that *.results directory, then I will take a look on this end.

–pt

5

Hi Mrinmayi,

I will have to see what happens with both -radial_correlate and -radial_correlate_blocks are applied. The former is an older version. I should probably change it to a fatal error to use both.

Anyway, how does it work if you remove “-radial_correlate yes”?

Thanks,

  • rick
6

Hi again,
Paul, the data is being uploaded now- it should be done in an hour or so!

Rick- thanks for your suggestion! I assumed the “-radial_correlate_blocks” was an additional option passed to radial correlate block that was turned on with the “-radial_correlate yes” flag. I’m running a new iteration without the latter flag. I’ll report back when it’s done!
Mrinmayi

7

Hi, Mrinmayi-

Thanks, I have started looking at this now. As with most things (and particularly with afni_proc.py), Rick’s suggestion will probably be the best way forward.

NB: If you could use a more uptodate AFNI version, that would reeeallly be recommendable.

Even if the official server one is managed by someone else, coudl you just put a copy of the binaries in your own home directory, and put that location into your $PATH ahead of the other version? That way, you know that the OS is set up to run AFNI already (because the server can run it), but you have your own, updatable version?

You appear to be using the OpenMP_64 version, and assuming that you do not already have a copy of AFNI binaries (or anything else) in a directory “~/abin”, you could do the following to go to your home directory, download the binaries there, unpack them, and move them into ~/abin:


cd
wget https://afni.nimh.nih.gov/pub/dist/tgz/linux_openmp_64.tgz
tar -xf linux_openmp_64.tgz
mv linux_openmp_64 abin

… and then edit your ~/.bashrc and ~/.cshrc files to have this “$HOME/abin” location first in your path.

If you then open a new terminal or source the ~/.*rc file of your current shell, you could then check that “which afni” shows that path location, and you can use this one.

That might be what I would do in order to keep up wtih a modern AFNI versioning, if that seems reasonable. You should even be able to stay uptodate by just running “@update.afni.binaries -d” as usual, because this new version will be your AFNI of choice.

–pt

8

Success!! Rick, I was able to generate the QC files by removing the “-radial_correlate yes” flag! Sorry- that was a stupid mistake. Thanks a lot!

Thanks for your suggestions, Paul. It does take a while for the admin folks to make cluster-wide changes to software versions, so installing these in my own path will make life much easier! I’ll definitely do that.

Thanks again, for all your help! :slight_smile:
Mrinmayi