apqc error: 'QC_SUBJ/media/qc_07_vstat_seed_rh-cort-vis.pbar.json'

AFNI version info (afni -ver): 24.1.02

apqc_make_tcsh.py -review_style pythonic -subj_dir . \
        -uvar_json out.ss_review_uvars.json
    apqc_make_html.py -qc_dir QC_$subj

Dear afni experts,

I got an error as "No such file or directory: 'QC_SUBJ/media/qc_07_vstat_seed_rh-cort-vis.pbar.json.'"
I also found there weren't full outputs of preprocessing as it ended with run_qc_06.

Thank you very much for your time!

Best. regards
Yan

Hi, Yan-

Hmmm, it would help to see a bit more output. Could you please copy+paste the output of

afni_system_check.py -check_all

... and perhaps your afni_proc.py command as well?

Also, could you also copy+paste the terminal outputs of running one of these commands (depending on your shell, which you can tell by the letter-part of what is output by running echo $0):

# if shell is tcsh or csh
apqc_make_tcsh.py -review_style pythonic -subj_dir . \
        -uvar_json out.ss_review_uvars.json -do_log |& tee log_apqc.tcsh

# if shell is bash or zsh
apqc_make_tcsh.py -review_style pythonic -subj_dir . \
        -uvar_json out.ss_review_uvars.json -do_log 2>&1 | tee log_apqc.tcsh

?

thanks,
pt

Hi pt,

Thank you so much for your response!

afni_system_check.py -check_all

Output:

-------------------------------- general ---------------------------------
architecture:         64bit ELF
cpu type:             x86_64
system:               Linux
release:              4.18.0-513.24.1.el8_9.x86_64
version:              #1 SMP Mon Apr 8 11:23:13 EDT 2024
distribution:         debian bookworm/sid 
number of CPUs:       2
apparent login shell: tcsh  (current shell is bash)
shell RC file:        .tcshrc (exists)

--------------------- AFNI and related program tests ---------------------
which afni           : /opt/afni/afni
afni version         : Precompiled binary linux_ubuntu_16_64: Apr  8 2024 
                     : AFNI_24.1.02 'Publius Septimius Geta'
AFNI_version.txt     : AFNI_24.1.02, linux_ubuntu_16_64, Apr 08 2024, official
which python         : /opt/conda/bin/python
python version       : 2.7.18
which R              : /usr/bin/R
R version            : R version 4.1.2 (x86_64-pc-linux-gnu)

instances of various programs found in PATH:
    afni    : 1   (/opt/afni/afni)
    R       : 1   (/usr/bin/R)
    python  : 1   (/opt/conda/bin/python2.7)
    python2 : 1   (/opt/conda/bin/python2.7)
    python3 : 1   (/usr/bin/python3.10)


testing ability to start various programs...
    afni                 : success
    suma                 : success
    3dSkullStrip         : success
    3dAllineate          : success
    3dRSFC               : success
    SurfMesh             : success
    3dClustSim           : success
    uber_subject.py      : success
    3dMVM                : success
()
------------------------ dependent program tests -------------------------
checking for dependent programs...

which tcsh           : /usr/bin/tcsh
which Xvfb           : /usr/bin/Xvfb

checking for R packages...
    rPkgsInstall -pkgs ALL -check : success

R RHOME : /usr/lib/R

------------------------------ python libs -------------------------------

++ module loaded: matplotlib.pyplot
   module file : /opt/conda/lib/python2.7/site-packages/matplotlib/pyplot.pyc
   matplotlib version : 2.2.3

** failed to load module flask
-- flask is not required, but is desirable

** failed to load module flask_cors
-- flask_cors is not required, but is desirable

-------------------------------- env vars --------------------------------
PATH                       = /opt/afni:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

PYTHONPATH                 = 
R_LIBS                     = /opt/R
LD_LIBRARY_PATH            = /.singularity.d/libs
DYLD_LIBRARY_PATH          = 
DYLD_FALLBACK_LIBRARY_PATH = 
CONDA_SHLVL                = 0
CONDA_DEFAULT_ENV          = 

----------------------------- eval dot files -----------------------------
()
----------- AFNI $HOME files -----------
()
    .afnirc                   : missing
    .sumarc                   : missing
    .afni/help/all_progs.COMP : missing

--------- shell startup files ----------

   ** warning: .tcshrc does NOT seem to contain 'source .cshrc'
   -- (csh and tcsh will use different files)
   -- considered operations: path, apsearch
   
   -- note: followers should not need edits, so edit flags should be 0
      (have 0 follower(s), which can be ignored)
   
   dot file test : want 5 modifications across 3 files:
   
      file             path  flatdir  apsearch        follower
      ---------------  ----  -------  --------        --------
      .tcshrc          1     0        1               0     
      .cshrc           1     0        1               0     
      .bashrc          1     0        0               0     
   
------------------------------ data checks -------------------------------
data dir : missing AFNI_data6
data dir : missing AFNI_demos
data dir : missing suma_demo
data dir : missing afni_handouts
atlas    : found TT_N27+tlrc  under /opt/afni

------------------------------ OS specific -------------------------------
which apt-get        : /usr/bin/apt-get
apt-get version      : apt 2.4.11 (amd64)


=========================  summary, please fix:  =========================
*  shell bash: MISSING login setup file, e.g. .bash_profile
*  shell bash: MISSING non-login setup file .bashrc
*  shell bash: consider sourcing (non-login) .bashrc from (login) .bash_profile
*  please run: cp /opt/afni/AFNI.afnirc ~/.afnirc
*  please run: "suma -update_env" for .sumarc
*  please run: apsearch -update_all_afni_help
*  dot file test : want 5 modifications across 3 files:
*  insufficient data for AFNI bootcamp
   (see "Prepare for Bootcamp" on install pages)
apqc_make_tcsh.py -review_style pythonic -subj_dir . \
        -uvar_json out.ss_review_uvars.json -do_log |& tee log_apqc.tcsh

Output:

++ Done making (executable) IC errts script: 
      run_instacorr_errts.tcsh
++ Done making (executable) GV errts script: 
      run_graphview_errts.tcsh
++ Done making (executable) IC pbrun script: 
      run_instacorr_pbrun.tcsh
++ Done making (executable) GV pbrun script: 
      run_graphview_pbrun.tcsh
+* WARN: output QC dir exists already: QC_SUBJ
   -> backing it up to: QC_SUBJ_2024-04-25-12-58-52
++ APQC create: qc_00_vorig_EPI
++ APQC create: qc_01_vorig_anat
++ APQC create: qc_02_vorig_olap
++ APQC create: qc_03_ve2a_epi2anat
++ APQC create: qc_04_va2t_anat2temp
++ APQC create: qc_05_va2t_mask2final
('+* Removing and remaking vstat-QC dir:', 'vstat_seedcorr')
++ APQC create: qc_06_vstat_seed_lh-PCC
++ APQC create: qc_07_vstat_seed_rh-cort-vis
Traceback (most recent call last):
  File "/opt/afni/apqc_make_tcsh.py", line 678, in <module>
    seed )
  File "/opt/afni/afnipy/lib_apqc_tcsh.py", line 3308, in apqc_vstat_seedcorr
    with open(pbar_json, 'r') as fff:
IOError: [Errno 2] No such file or directory: 'QC_SUBJ/media/qc_07_vstat_seed_rh-cort-vis.pbar.json'

Many thanks
Yan

Hi, Yan-

Thanks for that info. It is a very odd place for the code to break, because essentially the same thing that just happened for the "seed_lh-PCC" vstat map---completed without complaint---is just being done again for "seed_rh-cort-vis", but it breaks in the latter case.

I just ran this on my computer with essentially the same AFNI version number, and also using Python 2.7 (which might not be so recommended anymore, given that Python 2.7 has been officially deprecated for a long time---most AFNI programs are still fine with it, but it will be catered for less-and-less in the future, and it would probably make sense to move toward using Python 3.*). I had no errors. Hmmm.

Could you please copy+paste your afni_proc.py command, and I will try a more exact copy of what you are running?

thanks,
pt

Hi pt,

Thank you so much for your kind reply. I'm sorry for the delay for I missed your message.

Pleas find my command below. And also this only happened with one of my rs-fMRI session. So could it be due to the quality of the image?

afni_proc.py -blocks despike tshift align tlrc volreg mask combine blur scale regress \
		 -copy_anat t1w_TRADA.nii \
		 -anat_has_skull yes \
		 -dsets_me_echo f_e1.nii \
		 -dsets_me_echo f_e2.nii \
		 -dsets_me_echo f_e3.nii \
		 -echo_times 12.7 31.0 48.0 \
		 -reg_echo 2\
		 -tcat_remove_first_trs 10 \
		 -align_opts_aea -cost nmi -check_flip -skullstrip_opts -push_to_edge \
		 -tlrc_base MNI152_2009_template_SSW.nii.gz -tlrc_NL_warp \
		 -volreg_align_to MIN_OUTLIER -volreg_align_e2a -volreg_tlrc_warp \
		 -mask_epi_anat yes -mask_segment_anat yes -mask_segment_erode yes \
		 -combine_method tedana -combine_tedana_path /opt/afni/tedana_wrapper.py \
		 -blur_size 6 -blur_in_mask yes -regress_bandpass 0.01 0.1 \
		 -regress_ROI WMe CSF -regress_motion_per_run -regress_censor_motion 3.0 \
		 -regress_censor_outliers 0.05 -regress_apply_mot_types demean deriv \
		 -regress_est_blur_epits -html_review_style pythonic

Many thanks
Yan

We are using Python 2.7 because the tedana/0.0.12 didn't work with Python 3. Tedana.py couldn't be found.

Best
Yan

Hi, Yan-

Thanks for posting this AP command, which I just spaced out a bit with vertical alignment:

afni_proc.py                                               \
    -blocks                   despike tshift align tlrc    \
                              volreg mask combine blur     \
                              scale regress                \
    -copy_anat                t1w_TRADA.nii                \
    -anat_has_skull           yes                          \
    -dsets_me_echo            f_e1.nii                     \
    -dsets_me_echo            f_e2.nii                     \
    -dsets_me_echo            f_e3.nii                     \
    -echo_times               12.7 31.0 48.0               \
    -reg_echo                 2                            \
    -tcat_remove_first_trs    10                           \
    -align_opts_aea           -cost nmi                    \
                              -check_flip                  \
                              -skullstrip_opts             \
                              -push_to_edge                \
    -tlrc_base                MNI152_2009_template_SSW.nii.gz \
    -tlrc_NL_warp                                          \
    -volreg_align_to          MIN_OUTLIER                  \
    -volreg_align_e2a                                      \
    -volreg_tlrc_warp                                      \
    -mask_epi_anat            yes                          \
    -mask_segment_anat        yes                          \
    -mask_segment_erode       yes                          \
    -combine_method           tedana                       \
    -combine_tedana_path      /opt/afni/tedana_wrapper.py  \
    -blur_size                6                            \
    -blur_in_mask             yes                          \
    -regress_bandpass         0.01 0.1                     \
    -regress_ROI              WMe CSF                      \
    -regress_motion_per_run                                \
    -regress_censor_motion    3.0                          \
    -regress_censor_outliers  0.05                         \
    -regress_apply_mot_types  demean deriv                 \
    -regress_est_blur_epits                                \
    -html_review_style        pythonic

I have a few comments on different features. I will start with the ME-FMRI specific one:

  • Re. combination method: -combine_method tedana runs a very old version of tedana, one which indeed is in Python 2.7 (from Kundu et al., 2015). There is a much newer, more modernly maintained and developed version from Du Pre et al., 2021 that you would use with method name mtedana instead of tedana, described here. We typically recommend people use the latter, unless they are developers or comparing methods, given the relative age and current states of active development. Would you perhaps want to try the latter?

  • The regress censor motion is really a big value (-regress_censor_motion 3.0). The unit of it is approx. millimeters. I would typically expect that to be more of order 0.3 or so, possibly higher for children or motion prone subjects (like up to 1.0, perhaps).

  • Re. -regress_ROI WMe CSF, we would typically only include tissue-based regressors when those maps have been made with great detail, e.g., with FreeSurfer (see this AP example and the anat_follower .. options and comments above the command). Using the simple 3dSeg segmentation (which is what the above would use) is perhaps not exact enough, and errors in boundaries could lead to really bad processing results, grabbing GM accidentally. For that reason, too, we would typically erode the WM maps.

  • Re. bandpassing (-regress_bandpass 0.01 0.1): we know it is commonly done in the field, but there are strong reasons not to do so, and we would likely not recommend doing so in most cases, because the penalty of degrees of freedom lost is high (and there is useful info in those shorter times scales/higher frequencies). See this AP help note for more discussion and references.

  • Re. blurring (-blur_size 6): we typically recommend blurring about 1.5 times the min or ave EPI voxel dim for single echo EPI, but for multi-echo FMRI, which you have here, you probably want much more minimal EPI. I don't know your voxel size, but likely 6mm is pretty big compared to it. ME-FMRI has notably higher voxelwise TSNR than single echo EPI, and so less blurring is needed and this 6 mm blurring might spread things out a lot.

  • For nonlinear alignment and skullstripping the anatomical volume, we would typically recommend running sswarper2 (the updated version of previous @SSwarper) before running afni_proc.py and then handing its results over. That is more computationally efficient and generally that program does a better job at nonlinear warp estimation and skullstripping job; see the sswarper2 help for running it, and the recipe for passing its useful outputs to afni_proc.py.

  • To help EPI-anatomical alignment in human FMRI data, we would typically include -align_unifize_epi local, which reduces the influence of potential EPI brightness inhomogeneity.

  • Also, for EPI-anatomical alignment we typically prefer having -cost lpc+ZZ rather than -cost nmi; the latter can be OK, but the former is usually quite solid. Has that not worked?

thanks, and sorry for the deluge of points.
pt

Hi Paul,

Thank you so much for your kind suggestions. They are really helpful! We will modify our codes and have a try.

Many thanks
Yan

Hi Paul,

I have modified our afni_proc.py function, and it's as following:

afni_proc.py -blocks despike tshift align tlrc volreg mask combine blur     \
      scale regress -copy_anat sswarp/anatSS.SUBJ.nii -anat_has_skull no      \
      -anat_follower anat_w_skull anat sswarp/anatU.SUBJ.nii                  \
      -anat_follower_ROI aaseg anat aparc.a2009s+aseg_REN_all.nii.gz          \
      -anat_follower_ROI aeseg epi aparc.a2009s+aseg_REN_all.nii.gz           \
      -anat_follower_ROI Fsvent epi fs_ap_latvent.nii.gz -anat_follower_ROI   \
      FSWe epi fs_aq_wm.nii.gz -anat_follower_erode Fsvent FSWe               \
      -dsets_me_echo f_e1.nii -dsets_me_echo f_e2.nii -dsets_me_echo f_e3.nii \
      -echo_times 12.7 31.0 48.0 -reg_echo 2 -tcat_remove_first_trs 10        \
      -align_opts_aea -giant_move -cost lpc+ZZ -check_flip -align_unifize_epi \
      local -tlrc_base MNI152_2009_template_SSW.nii.gz -tlrc_NL_warp          \
      -tlrc_NL_warped_dsets sswarp/anatQQ.SUBJ.nii                            \
      sswarp/anatQQ.SUBJ.aff12.1D sswarp/anatQQ.SUBJ_WARP.nii                 \
      -volreg_align_to MIN_OUTLIER -volreg_align_e2a -volreg_tlrc_warp        \
      -mask_epi_anat yes -mask_segment_anat yes -mask_segment_erode yes       \
      -combine_method m_tedana -combine_tedana_path /usr/local/bin/tedana     \
      -regress_motion_per_run -regress_ROI_PC FSvent 3                        \
      -regress_ROI_PC_per_run FSvent -regress_make_corr_vols aeseg FSvent     \
      -regress_anaticor_fast -regress_anaticor_label FSWe                     \
      -regress_est_blur_errts -regress_censor_motion 0.3                      \
      -regress_censor_outliers 0.05 -regress_apply_mot_types demean deriv     \
      -regress_est_blur_epits -html_review_style pythonic

I got some warnings about the FS atlases:
"** WARNING: anat follower does not seem to exist: aparc.a2009s+aseg_REN_all.nii.gz
originally from aparc.a2009s+aseg_REN_all.nii.gz
** WARNING: anat follower does not seem to exist: aparc.a2009s+aseg_REN_all.nii.gz
originally from aparc.a2009s+aseg_REN_all.nii.gz
** WARNING: anat follower does not seem to exist: fs_ap_latvent.nii.gz
originally from fs_ap_latvent.nii.gz
** WARNING: anat follower does not seem to exist: fs_aq_wm.nii.gz
originally from fs_aq_wm.nii.gz
-- template = 'MNI152_2009_template_SSW.nii.gz', exists = 1
-- have APQC atlas APQC_atlas_MNI.nii.gz
-- will use min outlier volume as motion base
** ERROR: no anat follower or ROI dict label 'FSvent'
** invalid block : regress"

Could you kindly help with that?

Many thanks
Yan

Hi, Yan-

Just to note, here is your same AP command just spaced out vertically:

afni_proc.py                                                                 \
    -blocks                   despike tshift align tlrc volreg mask combine  \
                              blur scale regress                             \
    -copy_anat                sswarp/anatSS.SUBJ.nii                         \
    -anat_has_skull           no                                             \
    -anat_follower            anat_w_skull anat sswarp/anatU.SUBJ.nii        \
    -anat_follower_ROI        aaseg anat aparc.a2009s+aseg_REN_all.nii.gz    \
    -anat_follower_ROI        aeseg epi aparc.a2009s+aseg_REN_all.nii.gz     \
    -anat_follower_ROI        Fsvent epi fs_ap_latvent.nii.gz                \
    -anat_follower_ROI        FSWe epi fs_aq_wm.nii.gz                       \
    -anat_follower_erode      Fsvent FSWe                                    \
    -dsets_me_echo            f_e1.nii                                       \
    -dsets_me_echo            f_e2.nii                                       \
    -dsets_me_echo            f_e3.nii                                       \
    -echo_times               12.7 31.0 48.0                                 \
    -reg_echo                 2                                              \
    -tcat_remove_first_trs    10                                             \
    -align_opts_aea           -giant_move -cost lpc+ZZ                       \
                              -check_flip                                    \
    -align_unifize_epi        local                                          \
    -tlrc_base                MNI152_2009_template_SSW.nii.gz                \
    -tlrc_NL_warp                                                            \
    -tlrc_NL_warped_dsets     sswarp/anatQQ.SUBJ.nii                         \
                              sswarp/anatQQ.SUBJ.aff12.1D                    \
                              sswarp/anatQQ.SUBJ_WARP.nii                    \
    -volreg_align_to          MIN_OUTLIER                                    \
    -volreg_align_e2a                                                        \
    -volreg_tlrc_warp                                                        \
    -mask_epi_anat            yes                                            \
    -mask_segment_anat        yes                                            \
    -mask_segment_erode       yes                                            \
    -combine_method           m_tedana                                       \
    -combine_tedana_path      /usr/local/bin/tedana                          \
    -regress_motion_per_run                                                  \
    -regress_ROI_PC           FSvent 3                                       \
    -regress_ROI_PC_per_run   FSvent                                         \
    -regress_make_corr_vols   aeseg FSvent                                   \
    -regress_anaticor_fast                                                   \
    -regress_anaticor_label   FSWe                                           \
    -regress_est_blur_errts                                                  \
    -regress_censor_motion    0.3                                            \
    -regress_censor_outliers  0.05                                           \
    -regress_apply_mot_types  demean deriv                                   \
    -regress_est_blur_epits                                                  \
    -html_review_style        pythonic

  • I would add a -subj_id .. option, so the outputs know the correct name of the subject.

  • To your question about this warning that turns fatal (and similar):

    WARNING: anat follower does not seem to exist:    aparc.a2009s+aseg_REN_all.nii.gz
    

    ... you need to provide a path to those files. Likely somethat that includes the SUMA/ directory, created after running both FreeSurfer's recon-all and AFNI's @SUMA_Make_Spec_FS. It is analogous to needing a path for your -tlrc_NL_warped_dsets .. for loading in the SSwarper results.

Perhaps a relevant question is why do you need a path for the FreeSurfer- and SSwarper-generated files, but not for MNI152_2009_template_SSW.nii.gz? Well, AFNI distributes some standard reference datasets, including that MNI*nii.gz one. AFNI will search a few locations (including the abin directory where the binaries live) for reference files, so you don't have to copy them around everywhere. The MNI*nii.gz falls into that category.

Slight side note: You can also also set up other directories to be in this list, and those might have other properties like loading those datasets into the AFNI GUI to always be selectable as underlay/overlay, even without copying them into your local directory. That functionality in particular is called setting up the AFNI_GLOBAL_SESSION environment variable, which Daniel discusses in the AFNI Academy lectures here.

OK, so that is why MNI*nii.gz typically won't need a path to it, but the other files you have do because they are not special/broadly applicable reference files that live in a special directory. They are special in their own way, sure, but they are specific to a particular subject only, and you need to tell afni_proc.py where to find them.

--pt

Another side note, I would add a couple more options for getting a bit more QC:

-radial_correlate_blocks  tcat volreg regress

-volreg_compute_tsnr      yes   

--pt

Hi Paul,

Thank you so much for your kind reply! I was wondering if that means I should run -recon-all and @SUMA_Make_Spec_FS for each subject first before running afni_pro.py.

Best wishes
Yan

Hi, Yan-

Yes, that is correct. FreeSurfer creates a detailed surface mesh and anatomical parcellation for each subject, output in their own anatomical space. It is a person-by-person analysis to run. @SUMA_Make_Spec_FS copies the outputs to more widely used NIFTI and GIFTI formats, as well as standardizing the mesh (at the std.141 and std.60 resolutions), and makes other things like the *REN* files of subsets of tissue classes, a parcellation-based brain mask fs_mask*, the *spec files of surface families, and more.

For a bit more detail/background, see:

--pt