Possible to start afniproc.py routine at regress step?

mamlung-KU · November 16, 2022, 8:18pm

We are moving away from using our in-house analysis scripts to use afniproc.py. I was able to get the pipeline running successfully and am pleased with the results I am getting. We are considering two different analyses at the regress step. The difference is how trials are binned (either based on the task parameters or based on participant’s behavioral responses, not really important to know more than that). Typically, what we would do in the past is run all preprocessing up to the 3dDeconvolve step and then branch and run two separate 3dDeconvolve analyses for each participant.

My question is, can we start the afniproc.py routine at the regress block and not have to re-run all the preprocessing that happens before that stage since it’s already been done for a given participant. See my command below. Essentially, is it possible to create a second proc script that would only include the regress block, but utilize the data files created when this full command was run?

One idea I had was making a copy of the proc script and manually editing it to only include 3dDec and after, but haven’t gone down that rabbit hole just yet.

Thanks for any guidance you can offer

Here’s the command we are using:
afni_proc.py
-subj_id $name
-copy_anat $name.T1+orig
-anat_has_skull yes
-dsets $name.apt1+orig $name.apt2+orig $name.apt3+orig $name.apt4+orig
#-blocks tshift align tlrc volreg blur mask scale regress
-blocks regress
-align_opts_aea -cost lpc+ZZ -check_flip
-tlrc_base TT_N27+tlrc
-tlrc_NL_warp
-volreg_align_to MIN_OUTLIER
-volreg_align_e2a
-volreg_tlrc_warp
-mask_epi_anat yes
-blur_size 3.5
-blur_in_automask
-regress_stim_times
stim_times/$name.stim.NeutApp_decide.1D
stim_times/$name.stim.NeutApp_input.1D
stim_times/$name.stim.NeutAmb_decide.1D
stim_times/$name.stim.NeutAmb_input.1D
stim_times/$name.stim.NeutAvo_decide.1D
stim_times/$name.stim.NeutAvo_input.1D
stim_times/$name.stim.NeutMissing_decide.1D
stim_times/$name.stim.NeutMissing_input.1D
stim_times/$name.stim.AlcApp_decide.1D
stim_times/$name.stim.AlcApp_input.1D
stim_times/$name.stim.AlcAmb_decide.1D
stim_times/$name.stim.AlcAmb_input.1D
stim_times/$name.stim.AlcAvo_decide.1D
stim_times/$name.stim.AlcAvo_input.1D
stim_times/$name.stim.AlcMissing_decide.1D
stim_times/$name.stim.AlcMissing_input.1D
-regress_stim_labels
NeutApp_decide
NeutApp_input
NeutAmb_decide
NeutAmb_input
NeutAvo_decide
NeutAvo_input
NeutMissing_decide
NeutMissing_input
AlcApp_decide
AlcApp_input
AlcAmb_decide
AlcAmb_input
AlcAvo_decide
AlcAvo_input
AlcMissing_decide
AlcMissing_input
-regress_stim_types AM1
-regress_basis ‘dmBLOCK(1)’
-regress_opts_3dD
-allzero_OK
-GOFORIT 11
-regress_global_times
-regress_motion_per_run
-regress_censor_motion 0.3
-regress_censor_outliers 0.05
-regress_compute_fitts
-regress_est_blur_epits
-regress_est_blur_errts
-html_review_style basic
-execute

rickr · November 16, 2022, 11:28pm

Sure, indirectly.

Add an option to your command like

-write_3dD_script RUN_3dD.tcsh

(and possibly include -write_3dD_prefix, though that is easy to edit).

That will generate a 3dDeconvolve command script that assumes all of the rest of the processing has happened, with a variable for a new output prefix.
You could possibly modify the regression options first, as it might be easier to modify those options in the afni_proc.py command rather than in the 3dDeconvolve command.

Note that it will generate a script in your original directory (e.g. where the proc.* script would be written to), but it is designed to be run from within an existing ‘results’ directory.

Anyway, give that a try and let me know how it goes.

rick

ptaylor · November 17, 2022, 3:08am

Hi-

If I could make a few comments—

I notice you explicitly ask for the “basic” form of QC HTML:


-html_review_style basic \

The “pythonic” one is more complete and nicer-looking, and only adds the requirement of having Python’s Matplotlib module (ver >=2.2.3) installed. That might provide a nicer QC experience?

You have nonlinear alignment to a template, which is great. It is certainly fine to implement that as you have here, but often we recommend running @SSwarper on the T1w volume prior to afni_proc.py to do this, and then to provide the calculated results to afni_proc.py. One benefit of this is that if you decide to rerun afni_proc.py later, changing and option or something, then you don’t have to go through the nonlinear alignment process again—it is computationally expensive and so can take a while. Additionally, @SSwarper will perform skullstripping (that’s the “SS” in the name), and often do a better job than 3dSkullStrip alone (which is the default in the current afni_proc.py command). You then don’t have to skullstrip in afni_proc.py, either.
Your current “align” block options are good, particularly using lpc+ZZ for EPI-anatomical alignment, but we have recently developed another option to further help, esp. when the EPI has brightness inhomogeneity. This would be used by simply adding the following option+value to your existing command (while still also using your existing align options):


-align_unifize_epi        local                                          \

Even if your EPI is not inhomogeneous, then this doesn’t seem to hurt, so it seems generally useful to include (in human MRI analysis).

For some further QC output, we often include this option:


-radial_correlate_blocks  tcat volreg                                    \

Happy to discuss any questions about these additional considerations, if any sound of interest.

–pt

mamlung-KU · November 17, 2022, 9:54pm

Thank you for these very helpful suggestions!

While I’m talking with you, I was wondering if there is a way to generate activation montage images from the command line. We typically open our output in AFNI, open a montage view, set the overlay sub-brick, set the pval, and save a jpeg for each participant. I was wondering if theres a handy AFNI command or tool that would allow us to automate this on the command line. May be a total shot in the dark.

ptaylor · November 17, 2022, 11:26pm

That is a topic near-and-dear to our collective hearts.

You can use @chauffeur_afni to “drive” automatic image/montage creation. Examples are provided here:
https://afni.nimh.nih.gov/pub/dist/doc/htmldoc/tutorials/auto_image/auto_%40chauffeur_afni.html
This can be combined with 2dcat, too:
https://afni.nimh.nih.gov/pub/dist/doc/htmldoc/tutorials/auto_image/auto_2dcat.html

Importantly, these tools power the automatic QC HTML generation in afni_proc.py:
https://afni.nimh.nih.gov/pub/dist/doc/htmldoc/tutorials/apqc_html/main_toc.html
I suspect you will find that useful for quality control (esp. in the “pythonic” form, which I cited above).

If you have any questions about making images, just let me know. There is a lot of GUI functionality encapsulated there. Most of what you can do in the GUI can be driven in a couple ways; @chauffeur_afni is one of those ways, trying to wrap around various options, environment settings, key presses and more.

You can also make driver scripts to open up AFNI and leave it running, with lots of choices made automatically. The @DriveAfni demo goes through some of that, but again, I/we are happy to discuss this more to facilitate it.

Being able to have these visualizations be systematic is very, very powerful and convenient for understanding data

–pt

rickr · November 18, 2022, 1:33am

And notably, as Paul referenced, please look at the images that have already been made as part of the APQC output.

From the subject *.results directory:

afni_open -b QC*/index.html

rick

mamlung-KU · November 21, 2022, 5:13pm

The QC output is outstanding and very helpful.

I’m trying to run Example 0 on the website you provided, but nothing is saving to my QC folder.

Here’s my command:

set opref = QC/anat

@chauffeur_afni -ulay 2072_test.T1+orig -prefix ${opref} -set_xhairs MULTI -montx 3 -monty 3 -label_mode 1 -label_mode 4

The command seems to run fine (see terminal output below). But when I go into QC folder, there’s nothing there.

++ My command:
@chauffeur_afni -ulay 2072_test.T1+orig -prefix QC/anat -set_xhairs MULTI -montx 3 -monty 3 -label_mode 1 -label_mode 4

++ Using AFNI ver : AFNI_22.3.05
++ chauffeur ver : 6.51
\n------------------ start of optionizing ------------------\n
++ Found input file: 2072_test.T1+orig
+* Noting: you did NOT provide an overlay, and yet
you did NOT turn off the overlay with ‘-olay_off’
→ so I assume you just want to view the underlay.
++ Using blowup factor: 2
++ Using opacity: 6
++ Making temporary work directory to copy vis files: QC/__tmp_chauf_RzN4p7uicct
++ Converted 0 to labels
++ Final subbrick indices: -1 -1 -1
++ Copy ulay to visualize (volumetric) within user’s range:
++ Ulay to be visualized within user range:
[0%, 98%] → [0.000000, 2408.000000]
++ 3dcalc: AFNI version=AFNI_22.3.05 (Nov 9 2022) [64-bit]
++ Authored by: A cast of thousands
++ Output dataset ./QC/__tmp_chauf_RzN4p7uicct/tmp_ulay.nii
++ No overlay viewing
++ Dimensions (xyzt): 256 256 188 1
++ (initial) Slice spacing ordered (x,y,z) is: 28 28 20
++ 3dmaskdump: AFNI version=AFNI_22.3.05 (Nov 9 2022) [64-bit]
++ Executing 3dcalc()
argv[0]=3dcalc argv[1]=-a argv[2]=2072_test.T1+orig[0] argv[3]=-expr argv[4]=a argv[5]=-byte argv[6]=-session argv[7]=/var/folders/y5/gzw6hjys3_90fqb0mdfm0q2m0000gn/T/ argv[8]=-prefix argv[9]=3dcalc_AFN_qs-ZU95XqkCGUUmM0dkiyg argv[10]=-verbose
++ 3dcalc: AFNI version=AFNI_22.3.05 (Nov 9 2022) [64-bit]
++ Authored by: A cast of thousands
++ Computing sub-brick 0
++ Scaling output to type byte brick(s)
++ Sub-brick 0 scale factor = 0.021250
++ Computing output statistics
++ Output dataset /var/folders/y5/gzw6hjys3_90fqb0mdfm0q2m0000gn/T/3dcalc_AFN_qs-ZU95XqkCGUUmM0dkiyg+orig.BRIK
++ 12320768 voxels in the entire dataset (no mask)
++ 1 voxels in the boxes and/or balls
++ Using only the boxes+balls mask
++ Will have: SET_IJK 127 127 94
\n------------------- end of optionizing -------------------\n
– trying to start Xvfb :284
[1] 74485

+++ Command Echo:
afni -q -no1D -noplugins -no_detach -com SWITCH_UNDERLAY tmp_ulay.nii -com SWITCH_OVERLAY tmp_ulay.nii -com SEE_OVERLAY - -com OPEN_WINDOW sagittalimage opacity=6 mont=3x3:28:0:black crop=0:0,0:0 -com OPEN_WINDOW coronalimage opacity=6 mont=3x3:28:0:black crop=0:0,0:0 -com OPEN_WINDOW axialimage opacity=6 mont=3x3:20:0:black crop=0:0,0:0 -com SET_PBAR_ALL -99 1 Plasma -com DO_NOTHING -com SET_SUBBRICKS -1 -1 -1 -com SET_ULAY_RANGE A.all 0.000000 2408.000000 -com SET_FUNC_RANGE 0 -com SET_THRESHNEW 0 * -com SET_FUNC_ALPHA No -com SET_FUNC_BOXED No -com SET_FUNC_RESAM NN.NN -com SET_XHAIRS MULTI -com SET_XHAIR_GAP -1 -com SET_IJK 127 127 94 -com SAVE_PNG axialimage QC/anat.axi blowup=2 -com SAVE_PNG sagittalimage QC/anat.sag blowup=2 -com SAVE_PNG coronalimage QC/anat.cor blowup=2 -com QUITT QC/__tmp_chauf_RzN4p7uicct

+/Users/amlung/neuro/rescue/2072_test/QC/__tmp_chauf_RzN4p7uicct
AFNI QUITTs!
\n+* Removing temporary image directory ‘QC/__tmp_chauf_RzN4p7uicct’.\n
[1] Done Xvfb :284 -screen 0 1024x768x24

++ DONE (good exit)
see: QC/anat*

mamlung-KU · November 21, 2022, 8:47pm

I’m reviewing the QC for one of our participants and the alignment is not very good at all. I’m not sure why, so I was hoping you could give me some ideas about what might be happening here.

Our old piecemeal pipeline (did not have an alignment step, just VR and TLRC) yielded much better correspondence between the anatomical and output of 3dDeconvolve. The first attachment shows the va2t part of QT. The second attachment shows the stats output dataset with threshold set to 1.0 to show the poor alignment.

rickr · November 21, 2022, 10:18pm

I expect the ve2a block of the APQC HTML report shows a failed alignment, too. Is that the case?

Do you have -giant_move included with -align_opts_aea?
Did you use Paul’s suggested ‘-align_unifize_epi local’?

Please feel free to post the current afni_proc.py command.

rick

ptaylor · November 21, 2022, 11:31pm

Hi-

That command produced an image for me (using a different anatomical dset that I had).

Can you please provide the output of


afni_system_check.py -check_all

?

thanks,
pt

mamlung-KU · November 22, 2022, 1:13am

Last login: Mon Nov 21 19:02:29 on ttys000
-------- freesurfer-Darwin-lion-stable-pub-v5.3.0 --------
Setting up environment for FreeSurfer/FS-FAST (and FSL)
FREESURFER_HOME /Applications/freesurfer
FSFAST_HOME /Applications/freesurfer/fsfast
FSF_OUTPUT_FORMAT nii.gz
SUBJECTS_DIR /Applications/freesurfer/subjects
MNI_DIR /Applications/freesurfer/mni
FSL_DIR /usr/local/fsl
[Michaels-MacBook-Pro-2:~] amlung% afni_system_check.py -check_all
-------------------------------- general ---------------------------------
architecture: 64bit
cpu type: i386
system: Darwin
release: 18.7.0
version: Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64
distribution: 10.14.6
number of CPUs: 4
apparent login shell: tcsh
shell RC file: .tcshrc (exists)

--------------------- AFNI and related program tests ---------------------
which afni : /Users/amlung/abin/afni
afni version : Precompiled binary macos_10.12_local: Nov 9 2022
: AFNI_22.3.05 ‘Lucius Verus’
AFNI_version.txt : AFNI_22.3.05, macos_10.12_local, Nov 09 2022
which python : /usr/local/bin/python
python version : 2.7.16
which R : /usr/local/bin/R
R version : R version 4.0.2 (2020-06-22) – “Taking Off Again”
which tcsh : /bin/tcsh

instances of various programs found in PATH:
afni : 1 (/Users/amlung/abin/afni)
R : 1 (/Library/Frameworks/R.framework/Versions/4.0/Resources/bin/R)
python : 2
/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/bin/python2.7
/usr/bin/python
python2 : 1 (/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/bin/python2.7)
python3 : 1 (/Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11)

testing ability to start various programs…
afni : success
suma : success
3dSkullStrip : success
uber_subject.py : success
3dAllineate : success
3dRSFC : success
SurfMesh : success
3dClustSim : success
3dMVM : FAILURE
Error in dyn.load(ll) :
unable to load shared object ‘/Users/amlung/abin/R_io.so’:
dlopen(/Users/amlung/abin/R_io.so, 6): Library not loaded: /sw/Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libR.dylib
Referenced from: /Users/amlung/abin/R_io.so
Reason: image not found
Calls: source … withVisible → eval → eval → set_R_io → dyn.load
Execution halted

checking for R packages…
rPkgsInstall -pkgs ALL -check : success

R RHOME : /Library/Frameworks/R.framework/Resources

checking for $HOME files…
.afnirc : found
.sumarc : found
.afni/help/all_progs.COMP : found

------------------------------ python libs -------------------------------
++ module loaded: PyQt4
module file : /usr/local/lib/python2.7/site-packages/PyQt4/init.pyc

** have PyQt4, but cannot load QtCore, QtGui; error is:

Traceback (most recent call last):
File “”, line 1, in
ImportError: No module named sip

** failed to load module matplotlib.pyplot
– matplotlib.pyplot is required

– python binaries under /usr/local/bin:
/usr/local/bin/python (sym link to /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/bin/python2.7)
/usr/local/bin/python3 (sym link to /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11)
/usr/local/bin/python2 (sym link to /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/bin/python2.7)
/usr/local/bin/python3.7 (sym link to /usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/bin/python3.7)
/usr/local/bin/python2.7 (sym link to /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/bin/python2.7)

-------------------------------- env vars --------------------------------
PATH = /Applications/freesurfer/bin:/Applications/freesurfer/fsfast/bin:/Applications/freesurfer/tktools:/usr/local/fsl/bin:/Applications/freesurfer/mni/bin:/Library/Frameworks/Python.framework/Versions/3.11/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Users/amlung/abin:/Applications/workbench/bin_macosx64/

PYTHONPATH = /usr/local/lib/python2.7/site-packages

R_LIBS =
LD_LIBRARY_PATH =
DYLD_LIBRARY_PATH (sub-shell) = /opt/X11/lib/flat_namespace
DYLD_FALLBACK_LIBRARY_PATH (sub-shell) = /Users/amlung/abin

------------------------------ data checks -------------------------------
data dir : missing AFNI_data6
data dir : missing AFNI_demos
data dir : missing suma_demo
data dir : missing afni_handouts
atlas : found TT_N27+tlrc under /Users/amlung/abin

------------------------------ OS specific -------------------------------
XQuartz version : 2.8.2

which brew : /usr/local/bin/brew
brew version : Homebrew >=1.7.1 (shallow or no git repository)

++ found PyQt4 under /usr/local/lib/python2.7/site-packages
++ found 1 dylib files under ‘/opt/X11/lib/flat_namespace’
– found ‘libXt’ dylib files:
/opt/X11/lib/flat_namespace/libXt.6.dylib
– recent OS X, cheating to check DYLD_LIBRARY_PATH in cur shell ‘tcsh’…
++ found evar DYLD_LIBRARY_PATH = /opt/X11/lib/flat_namespace

========================= summary, please fix: =========================

AFNI programs show FAILURE
check for partial install of PyQt4
python library matplotlib is required
(see AFNI install docs for details)
insufficient data for AFNI bootcamp
(see “Prepare for Bootcamp” on install pages)
consider installing gcc under homebrew
consider installing glib under homebrew

[Michaels-MacBook-Pro-2:~] amlung%
[Michaels-MacBook-Pro-2:~] amlung%
[Michaels-MacBook-Pro-2:~] amlung%
[Michaels-MacBook-Pro-2:~] amlung%
[Michaels-MacBook-Pro-2:~] amlung% python -v

installing zipimport hook

import zipimport # builtin

installed zipimport hook

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.py

import site # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.pyc

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.py

import os # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.pyc
import errno # builtin
import posix # builtin

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py

import posixpath # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.pyc

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/stat.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/stat.py

import stat # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/stat.pyc

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/genericpath.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/genericpath.py

import genericpath # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/genericpath.pyc

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/warnings.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/warnings.py

import warnings # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/warnings.pyc

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/linecache.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/linecache.py

import linecache # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/linecache.pyc

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/types.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/types.py

import types # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/types.pyc

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/UserDict.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/UserDict.py

import UserDict # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/UserDict.pyc

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_abcoll.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_abcoll.py

import _abcoll # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_abcoll.pyc

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/abc.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/abc.py

import abc # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/abc.pyc

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_weakrefset.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_weakrefset.py

import _weakrefset # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_weakrefset.pyc
import _weakref # builtin

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.py

import copy_reg # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.pyc

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/traceback.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/traceback.py

import traceback # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/traceback.pyc

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sysconfig.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sysconfig.py

import sysconfig # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sysconfig.pyc

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/re.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/re.py

import re # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/re.pyc

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_compile.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_compile.py

import sre_compile # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_compile.pyc
import _sre # builtin

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_parse.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_parse.py

import sre_parse # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_parse.pyc

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_constants.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_constants.py

import sre_constants # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_constants.pyc
dlopen(“/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_locale.so”, 2);
import _locale # dynamically loaded from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_locale.so

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_sysconfigdata.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_sysconfigdata.py

import _sysconfigdata # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_sysconfigdata.pyc

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_osx_support.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_osx_support.py

import _osx_support # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_osx_support.pyc

/usr/local/lib/python2.7/site-packages/sitecustomize.pyc matches /usr/local/lib/python2.7/site-packages/sitecustomize.py

import sitecustomize # precompiled from /usr/local/lib/python2.7/site-packages/sitecustomize.pyc
import encodings # directory /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/init.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/init.py

import encodings # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/init.pyc

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/codecs.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/codecs.py

import codecs # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/codecs.pyc
import _codecs # builtin

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/aliases.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/aliases.py

import encodings.aliases # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/aliases.pyc

/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.pyc matches /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.py

import encodings.utf_8 # precompiled from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.pyc
Python 2.7.16 (default, Sep 2 2019, 12:01:16)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type “help”, “copyright”, “credits” or “license” for more information.
dlopen(“/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/readline.so”, 2);
import readline # dynamically loaded from /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/readline.so

KeyboardInterrupt

exit
Use exit() or Ctrl-D (i.e. EOF) to exit
exit()

clear builtin._

clear sys.path

clear sys.argv

clear sys.ps1

clear sys.ps2

clear sys.exitfunc

clear sys.exc_type

clear sys.exc_value

clear sys.exc_traceback

clear sys.last_type

clear sys.last_value

clear sys.last_traceback

clear sys.path_hooks

clear sys.path_importer_cache

clear sys.meta_path

clear sys.flags

clear sys.float_info

restore sys.stdin

restore sys.stdout

restore sys.stderr

cleanup main

cleanup[1] encodings

cleanup[1] site

cleanup[1] sysconfig

cleanup[1] abc

cleanup[1] _weakrefset

cleanup[1] sre_constants

cleanup[1] _codecs

cleanup[1] _warnings

cleanup[1] zipimport

cleanup[1] _sysconfigdata

cleanup[1] mpl_toolkits

cleanup[1] encodings.utf_8

cleanup[1] _osx_support

cleanup[1] codecs

cleanup[1] readline

cleanup[1] sitecustomize

cleanup[1] signal

cleanup[1] traceback

cleanup[1] posix

cleanup[1] encodings.aliases

cleanup[1] exceptions

cleanup[1] _weakref

cleanup[1] re

cleanup[1] _locale

cleanup[1] sre_compile

cleanup[1] _sre

cleanup[1] sre_parse

cleanup[2] copy_reg

cleanup[2] posixpath

cleanup[2] errno

cleanup[2] _abcoll

cleanup[2] types

cleanup[2] genericpath

cleanup[2] stat

cleanup[2] warnings

cleanup[2] UserDict

cleanup[2] os.path

cleanup[2] linecache

cleanup[2] os

cleanup sys

cleanup builtin

cleanup ints: 20 unfreed ints

cleanup floats

[Michaels-MacBook-Pro-2:~] amlung% python -V
Python 2.7.16
[Michaels-MacBook-Pro-2:~] amlung% python3 -version
Unknown option: -e
usage: /Library/Frameworks/Python.framework/Versions/3.11/Resources/Python.app/Contents/MacOS/Python [option] … [-c cmd | -m mod | file | -] [arg] …
Try `python -h’ for more information.
[Michaels-MacBook-Pro-2:~] amlung% python3 --version
Python 3.11.0
[Michaels-MacBook-Pro-2:~] amlung% pip install matplotlib
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Requirement already satisfied: matplotlib in /usr/local/lib/python2.7/site-packages (2.2.5)
Requirement already satisfied: subprocess32 in /usr/local/lib/python2.7/site-packages (from matplotlib) (3.5.4)
Requirement already satisfied: pytz in /usr/local/lib/python2.7/site-packages (from matplotlib) (2022.6)
Requirement already satisfied: backports.functools-lru-cache in /usr/local/lib/python2.7/site-packages (from matplotlib) (1.6.4)
Requirement already satisfied: six>=1.10 in /usr/local/lib/python2.7/site-packages (from matplotlib) (1.16.0)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python2.7/site-packages (from matplotlib) (2.8.2)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python2.7/site-packages (from matplotlib) (1.1.0)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python2.7/site-packages (from matplotlib) (0.10.0)
Requirement already satisfied: numpy>=1.7.1 in /usr/local/lib/python2.7/site-packages (from matplotlib) (1.16.6)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python2.7/site-packages (from matplotlib) (2.4.7)
Requirement already satisfied: setuptools in /usr/local/lib/python2.7/site-packages (from kiwisolver>=1.0.1->matplotlib) (41.0.1)
[Michaels-MacBook-Pro-2:~] amlung%
[Michaels-MacBook-Pro-2:~] amlung%
[Michaels-MacBook-Pro-2:~] amlung%
[Michaels-MacBook-Pro-2:~] amlung%
[Michaels-MacBook-Pro-2:~] amlung%
[Michaels-MacBook-Pro-2:~] amlung%
[Michaels-MacBook-Pro-2:~] amlung%
[Michaels-MacBook-Pro-2:~] amlung% afni_system_check.py -check_all
-------------------------------- general ---------------------------------
architecture: 64bit
cpu type: i386
system: Darwin
release: 18.7.0
version: Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64
distribution: 10.14.6
number of CPUs: 4
apparent login shell: tcsh
shell RC file: .tcshrc (exists)

--------------------- AFNI and related program tests ---------------------
which afni : /Users/amlung/abin/afni
afni version : Precompiled binary macos_10.12_local: Nov 9 2022
: AFNI_22.3.05 ‘Lucius Verus’
AFNI_version.txt : AFNI_22.3.05, macos_10.12_local, Nov 09 2022
which python : /usr/local/bin/python
python version : 2.7.16
which R : /usr/local/bin/R
R version : R version 4.0.2 (2020-06-22) – “Taking Off Again”
which tcsh : /bin/tcsh

instances of various programs found in PATH:
afni : 1 (/Users/amlung/abin/afni)
R : 1 (/Library/Frameworks/R.framework/Versions/4.0/Resources/bin/R)
python : 2
/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/bin/python2.7
/usr/bin/python
python2 : 1 (/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/bin/python2.7)
python3 : 1 (/Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11)

testing ability to start various programs…
afni : success
suma : success
3dSkullStrip : success
uber_subject.py : success
3dAllineate : success
3dRSFC : success
SurfMesh : success
3dClustSim : success
3dMVM : FAILURE
Error in dyn.load(ll) :
unable to load shared object ‘/Users/amlung/abin/R_io.so’:
dlopen(/Users/amlung/abin/R_io.so, 6): Library not loaded: /sw/Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libR.dylib
Referenced from: /Users/amlung/abin/R_io.so
Reason: image not found
Calls: source … withVisible → eval → eval → set_R_io → dyn.load
Execution halted

checking for R packages…
rPkgsInstall -pkgs ALL -check : success

R RHOME : /Library/Frameworks/R.framework/Resources

checking for $HOME files…
.afnirc : found
.sumarc : found
.afni/help/all_progs.COMP : found

------------------------------ python libs -------------------------------
++ module loaded: PyQt4
module file : /usr/local/lib/python2.7/site-packages/PyQt4/init.pyc

** have PyQt4, but cannot load QtCore, QtGui; error is:

Traceback (most recent call last):
File “”, line 1, in
ImportError: No module named sip

** failed to load module matplotlib.pyplot
– matplotlib.pyplot is required

– python binaries under /usr/local/bin:
/usr/local/bin/python (sym link to /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/bin/python2.7)
/usr/local/bin/python3 (sym link to /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11)
/usr/local/bin/python2 (sym link to /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/bin/python2.7)
/usr/local/bin/python3.7 (sym link to /usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/bin/python3.7)
/usr/local/bin/python2.7 (sym link to /usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/bin/python2.7)

-------------------------------- env vars --------------------------------
PATH = /Applications/freesurfer/bin:/Applications/freesurfer/fsfast/bin:/Applications/freesurfer/tktools:/usr/local/fsl/bin:/Applications/freesurfer/mni/bin:/Library/Frameworks/Python.framework/Versions/3.11/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Users/amlung/abin:/Applications/workbench/bin_macosx64/

PYTHONPATH = /usr/local/lib/python2.7/site-packages

R_LIBS =
LD_LIBRARY_PATH =
DYLD_LIBRARY_PATH (sub-shell) = /opt/X11/lib/flat_namespace
DYLD_FALLBACK_LIBRARY_PATH (sub-shell) = /Users/amlung/abin

------------------------------ data checks -------------------------------
data dir : missing AFNI_data6
data dir : missing AFNI_demos
data dir : missing suma_demo
data dir : missing afni_handouts
atlas : found TT_N27+tlrc under /Users/amlung/abin

------------------------------ OS specific -------------------------------
XQuartz version : 2.8.2

which brew : /usr/local/bin/brew
brew version : Homebrew >=1.7.1 (shallow or no git repository)

++ found PyQt4 under /usr/local/lib/python2.7/site-packages
++ found 1 dylib files under ‘/opt/X11/lib/flat_namespace’
– found ‘libXt’ dylib files:
/opt/X11/lib/flat_namespace/libXt.6.dylib
– recent OS X, cheating to check DYLD_LIBRARY_PATH in cur shell ‘tcsh’…
++ found evar DYLD_LIBRARY_PATH = /opt/X11/lib/flat_namespace

========================= summary, please fix: =========================

AFNI programs show FAILURE
check for partial install of PyQt4
python library matplotlib is required
(see AFNI install docs for details)
insufficient data for AFNI bootcamp
(see “Prepare for Bootcamp” on install pages)
consider installing gcc under homebrew
consider installing glib under homebrew

mamlung-KU · November 22, 2022, 1:20am

Hi Rick,
Attached image of the ve2a block. Looks like alignment failed there, too.

Did not include -giant_move, but align_unifize_epi local is included

Here’s the command:
afni_proc.py
-subj_id $name
-copy_anat $name.T1+orig
-anat_has_skull yes
-dsets $name.apt1+orig $name.apt2+orig $name.apt3+orig $name.apt4+orig
-blocks tshift align tlrc volreg blur mask scale regress
-align_opts_aea -cost lpc+ZZ -check_flip -align_unifize_epi local
-tlrc_base TT_N27+tlrc
-tlrc_NL_warp
-volreg_align_to MIN_OUTLIER
-volreg_align_e2a
-volreg_tlrc_warp
-mask_epi_anat yes
-blur_size 3.5
-blur_in_automask
-regress_stim_times
stim_times/$name.NeutAppDec_local.txt
stim_times/$name.NeutAppInp_local.txt
stim_times/$name.NeutAmbDec_local.txt
stim_times/$name.NeutAmbInp_local.txt
stim_times/$name.NeutAvoDec_local.txt
stim_times/$name.NeutAvoInp_local.txt
stim_times/$name.NeutMisDec_local.txt
stim_times/$name.NeutMisInp_local.txt
stim_times/$name.AlcAppDec_local.txt
stim_times/$name.AlcAppInp_local.txt
stim_times/$name.AlcAmbDec_local.txt
stim_times/$name.AlcAmbInp_local.txt
stim_times/$name.AlcAvoDec_local.txt
stim_times/$name.AlcAvoInp_local.txt
stim_times/$name.AlcMisDec_local.txt
stim_times/$name.AlcMisInp_local.txt
-regress_stim_labels
NeutAppDec
NeutAppInp
NeutAmbDec
NeutAmbInp
NeutAvoDec
NeutAvoInp
NeutMisDec
NeutMisInp
AlcAppDec
AlcAppInp
AlcAmbDec
AlcAmbInp
AlcAvoDec
AlcAvoInp
AlcMisDec
AlcMisInp
-regress_stim_types AM1
-regress_basis ‘dmBLOCK(1)’
-regress_opts_3dD
-allzero_OK
-GOFORIT 11
-regress_local_times
-regress_motion_per_run
-regress_censor_extern …/rescue_apt_dummyCensor.1D
-regress_censor_motion 0.3
-regress_censor_outliers 0.05
-regress_compute_fitts
-regress_est_blur_epits
-regress_est_blur_errts
-radial_correlate_blocks tcat volreg
-html_review_style pythonic \

mamlung-KU · November 22, 2022, 4:28am

I re-ran with -giant_move in the align_opts and it’s worse. see attached.

ptaylor · November 22, 2022, 11:31am

OK, thanks for sharing that.

Some of the errors in the “please fix” can be ignored at the moment (e.g., the stats program ones, because on a Mac at the moment users are required to use R v3.6.*).

Installing matplotlib would greatly improve the APQC HTML, because all the line plots will look a lot nicer (and a few other things get calculated).

From these two messages:


* consider installing gcc under homebrew
* consider installing glib under homebrew

I wonder if this Xcode/Xquartz installation was successful:
… https://afni.nimh.nih.gov/pub/dist/doc/htmldoc/background_install/install_instructs/steps_mac.html#install-xcode-and-xquartz

If you are getting volumetric images like the ones shown in the APQC HTML, then @chauffeur_afni is able to run OK on your computer, because that is what makes those. There is no error message, so it is not clear why it wouldn’t create an image for what you ran. I wonder if the netpbm installation (needed to make PNG specifically on a MAC, and the APQC HTML uses JPGs) might not have completed properly? That might also depend on the gcc and glib above. After getting those installed, can you try this step again to make sure it completes successfully:
https://afni.nimh.nih.gov/pub/dist/doc/htmldoc/background_install/install_instructs/steps_mac.html#install-netpbm

–pt

ptaylor · November 27, 2022, 1:37pm

Just to note: in looking at the data, the contrast was a bit different than what might be expected; using the “nmi” cost function seems to sort out the EPI-anatomical alignment. So, changing:


-align_opts_aea -cost lpc+ZZ -check_flip \

to


-align_opts_aea -cost nmi -check_flip \

In most cases, starting with “lpc+ZZ” would still be recommended; but then there are alternatives that can be useful, as here.

We would also probably recommend using @SSwarper for the nonlinear alignment to template space (warping) and skullstripping (ss). This program would be run before afni_proc.py (AP), and its results provided with a couple command line options, as demonstrated here in SSW’s help:
https://afni.nimh.nih.gov/pub/dist/doc/htmldoc/programs/alpha/%40SSwarper_sphx.html#using-ssw-with-afni-proc-py

–pt