AFNI version info (afni -ver): 24.1.02
apqc_make_tcsh.py -review_style pythonic -subj_dir . \
-uvar_json out.ss_review_uvars.json
apqc_make_html.py -qc_dir QC_$subj
Dear afni experts,
I got an error as "No such file or directory: 'QC_SUBJ/media/qc_07_vstat_seed_rh-cort-vis.pbar.json.'"
I also found there weren't full outputs of preprocessing as it ended with run_qc_06.
Thank you very much for your time!
Best. regards
Yan
Hi, Yan-
Hmmm, it would help to see a bit more output. Could you please copy+paste the output of
afni_system_check.py -check_all
... and perhaps your afni_proc.py command as well?
Also, could you also copy+paste the terminal outputs of running one of these commands (depending on your shell, which you can tell by the letter-part of what is output by running echo $0):
# if shell is tcsh or csh
apqc_make_tcsh.py -review_style pythonic -subj_dir . \
-uvar_json out.ss_review_uvars.json -do_log |& tee log_apqc.tcsh
# if shell is bash or zsh
apqc_make_tcsh.py -review_style pythonic -subj_dir . \
-uvar_json out.ss_review_uvars.json -do_log 2>&1 | tee log_apqc.tcsh
?
thanks,
pt
Hi pt,
Thank you so much for your response!
afni_system_check.py -check_all
Output:
-------------------------------- general ---------------------------------
architecture: 64bit ELF
cpu type: x86_64
system: Linux
release: 4.18.0-513.24.1.el8_9.x86_64
version: #1 SMP Mon Apr 8 11:23:13 EDT 2024
distribution: debian bookworm/sid
number of CPUs: 2
apparent login shell: tcsh (current shell is bash)
shell RC file: .tcshrc (exists)
--------------------- AFNI and related program tests ---------------------
which afni : /opt/afni/afni
afni version : Precompiled binary linux_ubuntu_16_64: Apr 8 2024
: AFNI_24.1.02 'Publius Septimius Geta'
AFNI_version.txt : AFNI_24.1.02, linux_ubuntu_16_64, Apr 08 2024, official
which python : /opt/conda/bin/python
python version : 2.7.18
which R : /usr/bin/R
R version : R version 4.1.2 (x86_64-pc-linux-gnu)
instances of various programs found in PATH:
afni : 1 (/opt/afni/afni)
R : 1 (/usr/bin/R)
python : 1 (/opt/conda/bin/python2.7)
python2 : 1 (/opt/conda/bin/python2.7)
python3 : 1 (/usr/bin/python3.10)
testing ability to start various programs...
afni : success
suma : success
3dSkullStrip : success
3dAllineate : success
3dRSFC : success
SurfMesh : success
3dClustSim : success
uber_subject.py : success
3dMVM : success
()
------------------------ dependent program tests -------------------------
checking for dependent programs...
which tcsh : /usr/bin/tcsh
which Xvfb : /usr/bin/Xvfb
checking for R packages...
rPkgsInstall -pkgs ALL -check : success
R RHOME : /usr/lib/R
------------------------------ python libs -------------------------------
++ module loaded: matplotlib.pyplot
module file : /opt/conda/lib/python2.7/site-packages/matplotlib/pyplot.pyc
matplotlib version : 2.2.3
** failed to load module flask
-- flask is not required, but is desirable
** failed to load module flask_cors
-- flask_cors is not required, but is desirable
-------------------------------- env vars --------------------------------
PATH = /opt/afni:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PYTHONPATH =
R_LIBS = /opt/R
LD_LIBRARY_PATH = /.singularity.d/libs
DYLD_LIBRARY_PATH =
DYLD_FALLBACK_LIBRARY_PATH =
CONDA_SHLVL = 0
CONDA_DEFAULT_ENV =
----------------------------- eval dot files -----------------------------
()
----------- AFNI $HOME files -----------
()
.afnirc : missing
.sumarc : missing
.afni/help/all_progs.COMP : missing
--------- shell startup files ----------
** warning: .tcshrc does NOT seem to contain 'source .cshrc'
-- (csh and tcsh will use different files)
-- considered operations: path, apsearch
-- note: followers should not need edits, so edit flags should be 0
(have 0 follower(s), which can be ignored)
dot file test : want 5 modifications across 3 files:
file path flatdir apsearch follower
--------------- ---- ------- -------- --------
.tcshrc 1 0 1 0
.cshrc 1 0 1 0
.bashrc 1 0 0 0
------------------------------ data checks -------------------------------
data dir : missing AFNI_data6
data dir : missing AFNI_demos
data dir : missing suma_demo
data dir : missing afni_handouts
atlas : found TT_N27+tlrc under /opt/afni
------------------------------ OS specific -------------------------------
which apt-get : /usr/bin/apt-get
apt-get version : apt 2.4.11 (amd64)
========================= summary, please fix: =========================
* shell bash: MISSING login setup file, e.g. .bash_profile
* shell bash: MISSING non-login setup file .bashrc
* shell bash: consider sourcing (non-login) .bashrc from (login) .bash_profile
* please run: cp /opt/afni/AFNI.afnirc ~/.afnirc
* please run: "suma -update_env" for .sumarc
* please run: apsearch -update_all_afni_help
* dot file test : want 5 modifications across 3 files:
* insufficient data for AFNI bootcamp
(see "Prepare for Bootcamp" on install pages)
apqc_make_tcsh.py -review_style pythonic -subj_dir . \
-uvar_json out.ss_review_uvars.json -do_log |& tee log_apqc.tcsh
Output:
++ Done making (executable) IC errts script:
run_instacorr_errts.tcsh
++ Done making (executable) GV errts script:
run_graphview_errts.tcsh
++ Done making (executable) IC pbrun script:
run_instacorr_pbrun.tcsh
++ Done making (executable) GV pbrun script:
run_graphview_pbrun.tcsh
+* WARN: output QC dir exists already: QC_SUBJ
-> backing it up to: QC_SUBJ_2024-04-25-12-58-52
++ APQC create: qc_00_vorig_EPI
++ APQC create: qc_01_vorig_anat
++ APQC create: qc_02_vorig_olap
++ APQC create: qc_03_ve2a_epi2anat
++ APQC create: qc_04_va2t_anat2temp
++ APQC create: qc_05_va2t_mask2final
('+* Removing and remaking vstat-QC dir:', 'vstat_seedcorr')
++ APQC create: qc_06_vstat_seed_lh-PCC
++ APQC create: qc_07_vstat_seed_rh-cort-vis
Traceback (most recent call last):
File "/opt/afni/apqc_make_tcsh.py", line 678, in <module>
seed )
File "/opt/afni/afnipy/lib_apqc_tcsh.py", line 3308, in apqc_vstat_seedcorr
with open(pbar_json, 'r') as fff:
IOError: [Errno 2] No such file or directory: 'QC_SUBJ/media/qc_07_vstat_seed_rh-cort-vis.pbar.json'
Many thanks
Yan
Hi, Yan-
Thanks for that info. It is a very odd place for the code to break, because essentially the same thing that just happened for the "seed_lh-PCC" vstat map---completed without complaint---is just being done again for "seed_rh-cort-vis", but it breaks in the latter case.
I just ran this on my computer with essentially the same AFNI version number, and also using Python 2.7 (which might not be so recommended anymore, given that Python 2.7 has been officially deprecated for a long time---most AFNI programs are still fine with it, but it will be catered for less-and-less in the future, and it would probably make sense to move toward using Python 3.*). I had no errors. Hmmm.
Could you please copy+paste your afni_proc.py command, and I will try a more exact copy of what you are running?
thanks,
pt
Hi pt,
Thank you so much for your kind reply. I'm sorry for the delay for I missed your message.
Pleas find my command below. And also this only happened with one of my rs-fMRI session. So could it be due to the quality of the image?
afni_proc.py -blocks despike tshift align tlrc volreg mask combine blur scale regress \
-copy_anat t1w_TRADA.nii \
-anat_has_skull yes \
-dsets_me_echo f_e1.nii \
-dsets_me_echo f_e2.nii \
-dsets_me_echo f_e3.nii \
-echo_times 12.7 31.0 48.0 \
-reg_echo 2\
-tcat_remove_first_trs 10 \
-align_opts_aea -cost nmi -check_flip -skullstrip_opts -push_to_edge \
-tlrc_base MNI152_2009_template_SSW.nii.gz -tlrc_NL_warp \
-volreg_align_to MIN_OUTLIER -volreg_align_e2a -volreg_tlrc_warp \
-mask_epi_anat yes -mask_segment_anat yes -mask_segment_erode yes \
-combine_method tedana -combine_tedana_path /opt/afni/tedana_wrapper.py \
-blur_size 6 -blur_in_mask yes -regress_bandpass 0.01 0.1 \
-regress_ROI WMe CSF -regress_motion_per_run -regress_censor_motion 3.0 \
-regress_censor_outliers 0.05 -regress_apply_mot_types demean deriv \
-regress_est_blur_epits -html_review_style pythonic
Many thanks
Yan
We are using Python 2.7 because the tedana/0.0.12 didn't work with Python 3. Tedana.py couldn't be found.
Best
Yan
Hi, Yan-
Thanks for posting this AP command, which I just spaced out a bit with vertical alignment:
afni_proc.py \
-blocks despike tshift align tlrc \
volreg mask combine blur \
scale regress \
-copy_anat t1w_TRADA.nii \
-anat_has_skull yes \
-dsets_me_echo f_e1.nii \
-dsets_me_echo f_e2.nii \
-dsets_me_echo f_e3.nii \
-echo_times 12.7 31.0 48.0 \
-reg_echo 2 \
-tcat_remove_first_trs 10 \
-align_opts_aea -cost nmi \
-check_flip \
-skullstrip_opts \
-push_to_edge \
-tlrc_base MNI152_2009_template_SSW.nii.gz \
-tlrc_NL_warp \
-volreg_align_to MIN_OUTLIER \
-volreg_align_e2a \
-volreg_tlrc_warp \
-mask_epi_anat yes \
-mask_segment_anat yes \
-mask_segment_erode yes \
-combine_method tedana \
-combine_tedana_path /opt/afni/tedana_wrapper.py \
-blur_size 6 \
-blur_in_mask yes \
-regress_bandpass 0.01 0.1 \
-regress_ROI WMe CSF \
-regress_motion_per_run \
-regress_censor_motion 3.0 \
-regress_censor_outliers 0.05 \
-regress_apply_mot_types demean deriv \
-regress_est_blur_epits \
-html_review_style pythonic
I have a few comments on different features. I will start with the ME-FMRI specific one:
Re. combination method: -combine_method tedana runs a very old version of tedana, one which indeed is in Python 2.7 (from Kundu et al., 2015). There is a much newer, more modernly maintained and developed version from Du Pre et al., 2021 that you would use with method name mtedana instead of tedana, described here. We typically recommend people use the latter, unless they are developers or comparing methods, given the relative age and current states of active development. Would you perhaps want to try the latter?
The regress censor motion is really a big value (-regress_censor_motion 3.0). The unit of it is approx. millimeters. I would typically expect that to be more of order 0.3 or so, possibly higher for children or motion prone subjects (like up to 1.0, perhaps).
Re. -regress_ROI WMe CSF, we would typically only include tissue-based regressors when those maps have been made with great detail, e.g., with FreeSurfer (see this AP example and the anat_follower .. options and comments above the command). Using the simple 3dSeg segmentation (which is what the above would use) is perhaps not exact enough, and errors in boundaries could lead to really bad processing results, grabbing GM accidentally. For that reason, too, we would typically erode the WM maps.
Re. bandpassing (-regress_bandpass 0.01 0.1): we know it is commonly done in the field, but there are strong reasons not to do so, and we would likely not recommend doing so in most cases, because the penalty of degrees of freedom lost is high (and there is useful info in those shorter times scales/higher frequencies). See this AP help note for more discussion and references.
Re. blurring (-blur_size 6): we typically recommend blurring about 1.5 times the min or ave EPI voxel dim for single echo EPI, but for multi-echo FMRI, which you have here, you probably want much more minimal EPI. I don't know your voxel size, but likely 6mm is pretty big compared to it. ME-FMRI has notably higher voxelwise TSNR than single echo EPI, and so less blurring is needed and this 6 mm blurring might spread things out a lot.
For nonlinear alignment and skullstripping the anatomical volume, we would typically recommend running sswarper2 (the updated version of previous @SSwarper) before running afni_proc.py and then handing its results over. That is more computationally efficient and generally that program does a better job at nonlinear warp estimation and skullstripping job; see the sswarper2 help for running it, and the recipe for passing its useful outputs to afni_proc.py.
To help EPI-anatomical alignment in human FMRI data, we would typically include -align_unifize_epi local, which reduces the influence of potential EPI brightness inhomogeneity.
Also, for EPI-anatomical alignment we typically prefer having -cost lpc+ZZ rather than -cost nmi; the latter can be OK, but the former is usually quite solid. Has that not worked?
thanks, and sorry for the deluge of points.
pt
Hi Paul,
Thank you so much for your kind suggestions. They are really helpful! We will modify our codes and have a try.
Many thanks
Yan
Hi Paul,
I have modified our afni_proc.py function, and it's as following:
afni_proc.py -blocks despike tshift align tlrc volreg mask combine blur \
scale regress -copy_anat sswarp/anatSS.SUBJ.nii -anat_has_skull no \
-anat_follower anat_w_skull anat sswarp/anatU.SUBJ.nii \
-anat_follower_ROI aaseg anat aparc.a2009s+aseg_REN_all.nii.gz \
-anat_follower_ROI aeseg epi aparc.a2009s+aseg_REN_all.nii.gz \
-anat_follower_ROI Fsvent epi fs_ap_latvent.nii.gz -anat_follower_ROI \
FSWe epi fs_aq_wm.nii.gz -anat_follower_erode Fsvent FSWe \
-dsets_me_echo f_e1.nii -dsets_me_echo f_e2.nii -dsets_me_echo f_e3.nii \
-echo_times 12.7 31.0 48.0 -reg_echo 2 -tcat_remove_first_trs 10 \
-align_opts_aea -giant_move -cost lpc+ZZ -check_flip -align_unifize_epi \
local -tlrc_base MNI152_2009_template_SSW.nii.gz -tlrc_NL_warp \
-tlrc_NL_warped_dsets sswarp/anatQQ.SUBJ.nii \
sswarp/anatQQ.SUBJ.aff12.1D sswarp/anatQQ.SUBJ_WARP.nii \
-volreg_align_to MIN_OUTLIER -volreg_align_e2a -volreg_tlrc_warp \
-mask_epi_anat yes -mask_segment_anat yes -mask_segment_erode yes \
-combine_method m_tedana -combine_tedana_path /usr/local/bin/tedana \
-regress_motion_per_run -regress_ROI_PC FSvent 3 \
-regress_ROI_PC_per_run FSvent -regress_make_corr_vols aeseg FSvent \
-regress_anaticor_fast -regress_anaticor_label FSWe \
-regress_est_blur_errts -regress_censor_motion 0.3 \
-regress_censor_outliers 0.05 -regress_apply_mot_types demean deriv \
-regress_est_blur_epits -html_review_style pythonic
I got some warnings about the FS atlases:
"** WARNING: anat follower does not seem to exist: aparc.a2009s+aseg_REN_all.nii.gz
originally from aparc.a2009s+aseg_REN_all.nii.gz
** WARNING: anat follower does not seem to exist: aparc.a2009s+aseg_REN_all.nii.gz
originally from aparc.a2009s+aseg_REN_all.nii.gz
** WARNING: anat follower does not seem to exist: fs_ap_latvent.nii.gz
originally from fs_ap_latvent.nii.gz
** WARNING: anat follower does not seem to exist: fs_aq_wm.nii.gz
originally from fs_aq_wm.nii.gz
-- template = 'MNI152_2009_template_SSW.nii.gz', exists = 1
-- have APQC atlas APQC_atlas_MNI.nii.gz
-- will use min outlier volume as motion base
** ERROR: no anat follower or ROI dict label 'FSvent'
** invalid block : regress"
Could you kindly help with that?
Many thanks
Yan
Hi, Yan-
Just to note, here is your same AP command just spaced out vertically:
afni_proc.py \
-blocks despike tshift align tlrc volreg mask combine \
blur scale regress \
-copy_anat sswarp/anatSS.SUBJ.nii \
-anat_has_skull no \
-anat_follower anat_w_skull anat sswarp/anatU.SUBJ.nii \
-anat_follower_ROI aaseg anat aparc.a2009s+aseg_REN_all.nii.gz \
-anat_follower_ROI aeseg epi aparc.a2009s+aseg_REN_all.nii.gz \
-anat_follower_ROI Fsvent epi fs_ap_latvent.nii.gz \
-anat_follower_ROI FSWe epi fs_aq_wm.nii.gz \
-anat_follower_erode Fsvent FSWe \
-dsets_me_echo f_e1.nii \
-dsets_me_echo f_e2.nii \
-dsets_me_echo f_e3.nii \
-echo_times 12.7 31.0 48.0 \
-reg_echo 2 \
-tcat_remove_first_trs 10 \
-align_opts_aea -giant_move -cost lpc+ZZ \
-check_flip \
-align_unifize_epi local \
-tlrc_base MNI152_2009_template_SSW.nii.gz \
-tlrc_NL_warp \
-tlrc_NL_warped_dsets sswarp/anatQQ.SUBJ.nii \
sswarp/anatQQ.SUBJ.aff12.1D \
sswarp/anatQQ.SUBJ_WARP.nii \
-volreg_align_to MIN_OUTLIER \
-volreg_align_e2a \
-volreg_tlrc_warp \
-mask_epi_anat yes \
-mask_segment_anat yes \
-mask_segment_erode yes \
-combine_method m_tedana \
-combine_tedana_path /usr/local/bin/tedana \
-regress_motion_per_run \
-regress_ROI_PC FSvent 3 \
-regress_ROI_PC_per_run FSvent \
-regress_make_corr_vols aeseg FSvent \
-regress_anaticor_fast \
-regress_anaticor_label FSWe \
-regress_est_blur_errts \
-regress_censor_motion 0.3 \
-regress_censor_outliers 0.05 \
-regress_apply_mot_types demean deriv \
-regress_est_blur_epits \
-html_review_style pythonic
I would add a -subj_id .. option, so the outputs know the correct name of the subject.
To your question about this warning that turns fatal (and similar):
WARNING: anat follower does not seem to exist: aparc.a2009s+aseg_REN_all.nii.gz
... you need to provide a path to those files. Likely somethat that includes the SUMA/ directory, created after running both FreeSurfer's recon-all and AFNI's @SUMA_Make_Spec_FS. It is analogous to needing a path for your -tlrc_NL_warped_dsets .. for loading in the SSwarper results.
Perhaps a relevant question is why do you need a path for the FreeSurfer- and SSwarper-generated files, but not for MNI152_2009_template_SSW.nii.gz? Well, AFNI distributes some standard reference datasets, including that MNI*nii.gz one. AFNI will search a few locations (including the abin directory where the binaries live) for reference files, so you don't have to copy them around everywhere. The MNI*nii.gz falls into that category.
Slight side note: You can also also set up other directories to be in this list, and those might have other properties like loading those datasets into the AFNI GUI to always be selectable as underlay/overlay, even without copying them into your local directory. That functionality in particular is called setting up the AFNI_GLOBAL_SESSION environment variable, which Daniel discusses in the AFNI Academy lectures here.
OK, so that is why MNI*nii.gz typically won't need a path to it, but the other files you have do because they are not special/broadly applicable reference files that live in a special directory. They are special in their own way, sure, but they are specific to a particular subject only, and you need to tell afni_proc.py where to find them.
--pt
Another side note, I would add a couple more options for getting a bit more QC:
-radial_correlate_blocks tcat volreg regress
-volreg_compute_tsnr yes
--pt
Hi Paul,
Thank you so much for your kind reply! I was wondering if that means I should run -recon-all and @SUMA_Make_Spec_FS for each subject first before running afni_pro.py.
Best wishes
Yan
Hi, Yan-
Yes, that is correct. FreeSurfer creates a detailed surface mesh and anatomical parcellation for each subject, output in their own anatomical space. It is a person-by-person analysis to run. @SUMA_Make_Spec_FS copies the outputs to more widely used NIFTI and GIFTI formats, as well as standardizing the mesh (at the std.141 and std.60 resolutions), and makes other things like the *REN* files of subsets of tissue classes, a parcellation-based brain mask fs_mask*, the *spec files of surface families, and more.
For a bit more detail/background, see:
--pt
Hi Paul,
Thank you so much for all your help! Our new afni_proc.py command has been running well so far. Can I ask a very basic question? Does 3dTproject detrend the fmri data?
Also, we have another single-echo dataset with resting state fMRI, t1w image, and an additional high resolution EPI. I was wondering how we could include the high resolution EPI to improve our preprocessing. I guessed we should first align the high-resolution EPI image to the T1W anatomical image and then align the rs-fMRI to high-resolution EPI. Can that be integrated into the afni_proc.py command?
Many thanks
Yan
The National Institute of Mental Health (NIMH) is part of the National Institutes of Health (NIH), a component of the U.S. Department of Health and Human Services.
Freedom of Information Act |
HHS Vulnerability Disclosure
National Institute of Mental
Health | National Institutes of Health | U.S.
Department of Health and Human Services |