T1 distortion after running afni_proc.py

xiyuezh · June 12, 2023, 7:18pm

Hi experts,

I am running afni_proc.py for preprocessing some resting-state data. For some of the participants, T1 was distorted (as shown below):

The orignal T1:

After running afni_proc.py

Here's my code:

afni_proc.py \
      -subj_id "$i" \
      -dsets "$i".rs.applytopup.nii.gz \
      -copy_anat "$i".T1.padded180+orig.  \
      -blocks despike tshift align tlrc volreg blur mask scale regress \
      -tcat_remove_first_trs 0 \
      -align_opts_aea -cost lpc+ZZ -check_flip \
      -tlrc_base TT_N27+tlrc \
      -tlrc_NL_warp \
      -volreg_align_to MIN_OUTLIER  \
      -volreg_align_e2a \
      -volreg_tlrc_warp \
      -mask_apply group  \
      -mask_segment_anat yes  \
      -mask_segment_erode yes \
      -blur_size 6  \
      -regress_censor_motion 0.5   \
      -regress_censor_outliers 0.05 \
      -regress_apply_mot_types demean deriv                     \
      -regress_est_blur_epits                                   \
      -regress_est_blur_errts

Could you please help? Thank you in advance!

ptaylor · June 12, 2023, 9:19pm

Howdy-

I guess that is the nonlinear alignment to TT_N27 space going awry?

There should be a QC HTML that gets created, that would more directly show the final anatomical on template volume. Can you go to the results directory run:

afni_open -b QC_*/index.html

... and then click the "va2t" tab/button at the top? What does that look like?

The anatomical quality looks fine to start with. I wonder if the alignment got thrown off by the large amount of non-brain material present. Note that in general, we would suggest running @SSwarper prior to afni_proc.py, and then feed the nonlinear results to the latter. Any troubleshooting of nonlinear alignment (and skullstripping, the "SS" part of the @SSwarper program name) can be done ahead of time.

An example @SSwarper command might look like:

@SSwarper                                                                    \
    -base           "${template}"                                            \
    -subid          "${subj}"                                                \
    -input          "${dset_anat}"                                           \
    -odir           "${sdir_ssw}"

--pt

xiyuezh · June 13, 2023, 5:36pm

Hi pt,

Thank you for your reply. On the index.html, there's no tab "va2t". I only have mot, regr, warns, qsumm and final. Also I didn't get @SSwarper in the results folder. There are some temp files remained. I guess the afni_proc was not completely run in this case?

ptaylor · June 13, 2023, 5:56pm

Hmm, what is the output of:

afni_system_check.py -check_all

? I wonder if some snapshotting libraries are missing on that system (and that will also show what version of AFNI is being used, which will help).

--pt

xiyuezh · June 13, 2023, 6:41pm

Hi pt,

Here's the output. The 3dMVM failure is becasue of the version of Mac. I asked here before.

-------------------------------- general ---------------------------------
architecture: 64bit
cpu type: i386
system: Darwin
release: 18.7.0
version: Darwin Kernel Version 18.7.0: Tue Jun 22 19:37:08 PDT 2021; root:xnu-4903.278.70~1/RELEASE_X86_64
distribution: 10.14.6
number of CPUs: 8
apparent login shell: bash
shell RC file: .bashrc (exists)

--------------------- AFNI and related program tests ---------------------
which afni : /Users/xiyuezh/abin/afni
afni version : Precompiled binary macos_10.12_local: May 30 2023
: AFNI_23.1.07 'Publius Helvius Pertinax'
AFNI_version.txt : AFNI_23.1.07, macos_10.12_local, May 30 2023
which python : /usr/bin/python
python version : 2.7.16
which R : /usr/local/bin/R
R version : R version 4.2.2 (2022-10-31) -- "Innocent and Trusting"
which tcsh : /bin/tcsh

instances of various programs found in PATH:
afni : 1 (/Users/xiyuezh/abin/afni)
R : 1 (/Library/Frameworks/R.framework/Versions/4.2/Resources/bin/R)
python : 1 (/usr/bin/python)
python2 : 0
python3 : 1 (/Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9)
Xvfb : 1 (/opt/X11/bin/Xvfb)

** have python3 but not python2

testing ability to start various programs...
afni : success
suma : success
3dSkullStrip : success
uber_subject.py : success
3dAllineate : success
3dRSFC : success
SurfMesh : success
3dClustSim : success
3dMVM : FAILURE
Error in dyn.load(ll) :
unable to load shared object '/Users/xiyuezh/abin/R_io.so':
dlopen(/Users/xiyuezh/abin/R_io.so, 6): Library not loaded: /sw/Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libR.dylib
Referenced from: /Users/xiyuezh/abin/R_io.so
Reason: image not found
Calls: source ... withVisible -> eval -> eval -> set_R_io -> dyn.load
Execution halted

checking for R packages...
rPkgsInstall -pkgs ALL -check : success

R RHOME : /Library/Frameworks/R.framework/Resources

checking for $HOME files...
.afnirc : found
.sumarc : found
.afni/help/all_progs.COMP : found

------------------------------ python libs -------------------------------
** failed to load module PyQt4
-- PyQt4 is no longer needed for an AFNI bootcamp

++ module loaded: matplotlib.pyplot
module file : /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/matplotlib/pyplot.pyc
matplotlib version : 1.3.1
** need maptplotlib version 2.2+ for APQC

-- python binaries under /sw/bin:
/sw/bin/python2.7

-- python binaries under /usr/local/bin:
/usr/local/bin/python3 (sym link to /Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9)
/usr/local/bin/python3.9 (sym link to /Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9)

-------------------------------- env vars --------------------------------
PATH = /usr/local/fsl/bin:/opt/anaconda3/bin:/Library/Frameworks/Python.framework/Versions/3.9/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Users/xiyuezh/abin

PYTHONPATH =
R_LIBS =
LD_LIBRARY_PATH =
DYLD_LIBRARY_PATH (sub-shell) = :/opt/X11/lib/flat_namespace:/opt/X11/lib/flat_namespace
DYLD_FALLBACK_LIBRARY_PATH (sub-shell) =

----------------------------- eval dot files -----------------------------

** warning: .tcshrc does NOT seem to contain 'source .cshrc'
-- (csh and tcsh will use different files)
-- considered operations: path, flatdir, apsearch

-- note: followers should not need edits, so edit flags should be 0
(have 0 follower(s), which can be ignored)

dot file test : want 3 modifications across 3 files:
  file             path  flatdir  apsearch        follower
  ---------------  ----  -------  --------        --------
  .tcshrc          1     1        1               0     
  .cshrc           0     0        0               0     
  .bashrc          0     0        0               0     
------------------------------ data checks -------------------------------
data dir : found AFNI_data6 under $HOME (2976900M Avail)
top history: 20 Feb 2020 [rickr]: updated FT_analysis examples
data dir : found AFNI_demos under $HOME
top history: ...ct 2020 [taylorp]: updated scripts under FATCAT_DEMO
data dir : found suma_demo under $HOME
top history: ...s_New/data/Build_tmp on Mon Mar 4 11:56:45 EST 2013
data dir : found afni_handouts under $HOME
atlas : found TT_N27+tlrc under /Users/xiyuezh/abin

------------------------------ OS specific -------------------------------
XQuartz version : 2.8.1

which brew : /usr/local/bin/brew
brew version : Homebrew 4.0.22

++ found PyQt4 under /sw/lib/qt4-mac/lib/python2.7/site-packages
** seem to be using fink python2.7 but need python
consider: sudo ln -s /sw/bin/python2.7 /sw/bin/python
** warning: have fink PyQt4, but non-fink python /usr/bin/python
++ found 1 dylib files under '/opt/X11/lib/flat_namespace'
-- found 'libXt' dylib files:
/opt/X11/lib/flat_namespace/libXt.6.dylib
-- recent OS X, cheating to check DYLD_LIBRARY_PATH in cur shell 'bash'...
++ found evar DYLD_LIBRARY_PATH = :/opt/X11/lib/flat_namespace:/opt/X11/lib/flat_namespace
-- recent OS X, cheating to check DYLD_LIBRARY_PATH in shell 'tcsh'...
** env var DYLD_LIBRARY_PATH not set to contain /opt/X11/lib/flat_namespace

========================= summary, please fix: =========================

just be aware: login shell 'bash', but our code examples use 'tcsh'

AFNI programs show FAILURE

check for partial install of PyQt4

dot file test : want 3 modifications across 3 files:

consider installing gcc under homebrew

consider installing glib under homebrew

please set DYLD_LIBRARY_PATH to /opt/X11/lib/flat_namespace in tcsh

ptaylor · June 13, 2023, 6:57pm

OK, thanks, that looks mostly good, but there are a couple things to maybe suggest there:

if you install a newer version of matplotlib, you will be able to get better lineplot images in the APQC HTML; I can see that message about that here:
```
matplotlib version : 1.3.1
** need maptplotlib version 2.2+ for APQC
```
I think you can get that by running:
```
pip install -U matplotlib
```

because of this message:

please set DYLD_LIBRARY_PATH to /opt/X11/lib/flat_namespace in tcsh

You should probably run:

   echo '' >> ~/.tcshrc
   echo 'setenv DYLD_LIBRARY_PATH /opt/X11/lib/flat_namespace' >> ~/.tcshrc

If you go to a directory that has a NIFTI or BRIK/HEAD dataset, and run:

@chauffeur_afni -ulay DSET_NAME -prefix test_image

what happens? Is there a set of 3 images made called test_image*, or is there an error in the terminal?

--pt

xiyuezh · June 13, 2023, 7:27pm

Hello,
I updated matplotlib, but pip is linked to python3. I cannot make any change on python2. Dose AFNI linked with python2 or 3? I am trying to remove python3 to solve this.

ptaylor · June 13, 2023, 7:47pm

AFNI can run with either Python 2 or 3. The issue is Mac referring to its Python as python3.

Do you use homebrew to install things on your Mac?

-pt

xiyuezh · June 13, 2023, 7:50pm

Yes, I have homebrew

ptaylor · June 13, 2023, 8:07pm

It is possible to brew install python, but I myself prefer to use conda to set up an environment with what is needed; I find it simpler.

The quick instructions for that for AFNI are:
https://afni.nimh.nih.gov/pub/dist/doc/htmldoc/background_install/miniconda.html#set-up-conda-quick
and they include a step to set that environment to be activated by default in your shell:
Activate an env by default

That will give you a Python that works as just python in the terminal, and the example environments there have the few dependencies wanted by afni_proc.py.

--pt

xiyuezh · June 13, 2023, 8:41pm

when I tried

~/miniconda3/bin/conda init bash tcsh zsh

An error occurs:

Password:

>>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
Traceback (most recent call last):
  File "/Users/xiyuezh/miniconda3/lib/python3.10/site-packages/conda/exceptions.py", line 1132, in __call__
    return func(*args, **kwargs)
  File "/Users/xiyuezh/miniconda3/lib/python3.10/site-packages/conda/cli/main.py", line 69, in main_subshell
    exit_code = do_call(args, p)
  File "/Users/xiyuezh/miniconda3/lib/python3.10/site-packages/conda/cli/conda_argparse.py", line 122, in do_call
    return getattr(module, func_name)(args, parser)
  File "/Users/xiyuezh/miniconda3/lib/python3.10/site-packages/conda/cli/main_init.py", line 33, in execute
    return initialize(context.conda_prefix, selected_shells, for_user, args.system,
  File "/Users/xiyuezh/miniconda3/lib/python3.10/site-packages/conda/core/initialize.py", line 113, in initialize
    run_plan_elevated(plan2)
  File "/Users/xiyuezh/miniconda3/lib/python3.10/site-packages/conda/core/initialize.py", line 709, in run_plan_elevated
    result = subprocess_call(
  File "/Users/xiyuezh/miniconda3/lib/python3.10/site-packages/conda/gateways/subprocess.py", line 98, in subprocess_call
    stdout, stderr = process.communicate(input=stdin)
  File "/Users/xiyuezh/miniconda3/lib/python3.10/subprocess.py", line 1154, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/Users/xiyuezh/miniconda3/lib/python3.10/subprocess.py", line 1985, in _communicate
    input_view = memoryview(self._input)
TypeError: memoryview: a bytes-like object is required, not 'str'
$ /Users/xiyuezh/miniconda3/bin/conda init bash tcsh zsh

environment variables:
CIO_TEST=
CONDA_EXE=/Users/xiyuezh/miniconda3/bin/conda
CONDA_PYTHON_EXE=/Users/xiyuezh/miniconda3/bin/python
CONDA_ROOT=/Users/xiyuezh/miniconda3
CONDA_SHLVL=0
CURL_CA_BUNDLE=
DYLD_LIBRARY_PATH=:/opt/X11/lib/flat_namespace:/opt/X11/lib/flat_namespace
LD_PRELOAD=
PATH=/Users/xiyuezh/miniconda/bin:/usr/local/fsl/bin:/Users/xiyuezh/minicon
da3/condabin:/Library/Frameworks/Python.framework/Versions/3.9/bin:/us
r/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Users/xiyuezh/
abin
PYTHONPATH=/Users/my_user/code
REQUESTS_CA_BUNDLE=
SSL_CERT_FILE=
 active environment : None
        shell level : 0
   user config file : /Users/xiyuezh/.condarc
populated config files : /Users/xiyuezh/.condarc
conda version : 23.3.1
conda-build version : not installed
python version : 3.10.10.final.0
virtual packages : __archspec=1=x86_64
__osx=10.14.6=0
__unix=0=0
base environment : /Users/xiyuezh/miniconda3 (writable)
conda av data dir : /Users/xiyuezh/miniconda3/etc/conda
conda av metadata url : None
channel URLs : main/osx-64
main/noarch
r/osx-64
r/noarch
package cache : /Users/xiyuezh/miniconda3/pkgs
/Users/xiyuezh/.conda/pkgs
envs directories : /Users/xiyuezh/miniconda3/envs
/Users/xiyuezh/.conda/envs
platform : osx-64
user-agent : conda/23.3.1 requests/2.28.1 CPython/3.10.10 Darwin/18.7.0 OSX/10.14.6
UID:GID : 518:20
netrc file : None
offline mode : False

An unexpected error has occurred. Conda has prepared the above report.

If submitted, this report will be used by core maintainers to improve
future releases of conda.
Would you like conda to send this report to the core maintainers? [y/N]:
No report sent. To permanently opt-out, use
$ conda config --set report_errors false

xiyuezh · June 13, 2023, 8:58pm

Hi pt,

I updated matplotlib. Now it's

++ module loaded: matplotlib.pyplot
module file : /Users/xiyuezh/Library/Python/2.7/lib/python/site-packages/matplotlib/pyplot.pyc
matplotlib version : 2.2.5

Then when I input

echo '' >> ~/.tcshrc

I got:

/Users/xiyuezh/.tcshrc: Permission denied.

xiyuezh · June 16, 2023, 7:33pm

Hi Paul, @ptaylor

I tried runing @sswarpper before afni_proc.py. It takes a long time. Now the results look good. For example, the ve2t looks like this

Thank you so much for your help!

Xiyue

ptaylor · June 20, 2023, 4:30pm

I'm glad to hear things are working now!

I will note that @SSwarper is a wrapper for 3dAllineate and 3dQwarp in AFNI, and those programs were written using OpenMP for inherent parallelization. If you have multiple CPUs available on a machine, you can make use of them. In fact, by default, the AFNI programs with OpenMP under the hood will aim to make use of multiple CPUs. So, running alignment programs on multi-CPU machines will have benefits for time of processing.

The environment variable to set the max number of CPUs for OpenMP-ized programs to use is:
OMP_NUM_THREADS.

To see how many CPU's are available on a system:

afni_system_check.py -disp_num_cpu

To see how many the AFNI programs will use, either by default or by explicitly setting OMP_NUM_THREADS, run:

afni_check_omp

More details on this are provided in this AFNI Academy Bootcamp video:

ptaylor · June 20, 2023, 4:33pm

Also, for future readers, some of the setup issues on that Mac with conda got sorted out offline by:

making sure that ~/.bash_profile sourced ~/.bashrc
making sure that ~/.tcshrc sourced ~/.cshrc
This also helped all shells know where to look for AFNI, because the $PATH variable was set in ~/.bashrc and ~/.cshrc, specifically (as well as ~/.zshrc).

--pt

xiyuezh · June 20, 2023, 8:15pm

Hi Paul,

The results of both comand lines are 8. I think it means the 8 CPUs are already been used right?

Xiyue

ptaylor · June 22, 2023, 7:51am

Yes, the first command was about how many are available, and the second for how many are being used. Those need not be the same---the latter must just be less than or equal to the former---even though they are in this case.

--pt