ME-fMRI and RetroTS.py

axlander1 · April 10, 2023, 8:53pm

Hello Afniacs.

I am trying to analyze some multi-echo fMRI data and I’ve been stumbling over RetroTS.py. The first problem is that when I went to look at the help file online (https://afni.nimh.nih.gov/pub/dist/doc/htmldoc/programs/alpha/RetroTS.py_sphx.html#ahelp-retrots-py), the following message was displayed:

Traceback (most recent call last):
File “/home/afniHQ/afni.build/pub.dist/bin/linux_ubuntu_16_64/RetroTS.py”, line 28, in
from lib_RetroTS.PeakFinder import peak_finder
File “/home/afniHQ/afni.build/pub.dist/bin/linux_ubuntu_16_64/lib_RetroTS/PeakFinder.py”, line 29, in
from pylab import plot, subplot, show, text, figure

This is essentislly the same error I got when I ran RetroTS. I also got the following error:

from pylab import plot, subplot, show, text, figure
ModuleNotFoundError: No module named ‘pylab’

I’m running afni ver. 23.0.07 on a linux cluster ( Ubuntu 16.04.7 LTS)

My second issue is a question: How do I handle the number of slices per TR? The ME-fMRI data has 3 TEs (so there are 3 TEs/slice/TR) is the slice count n, or n*3?

Thanks for any help you can provide.

Alex

ptaylor · April 10, 2023, 9:12pm

Hi, Alex-

Wow, that is embarassing that that message appears in the help file from our build system… I will check that out, thanks for pointing that out.

So, that RetroTS.py program does have pylab as a module dependency. I would have thought installing matplotlib would install that as well, in most situations. Could you perphaps copy+paste the output of:


afni_system_check.py -check_all

on that system? And how would you manage Python module dependencies on that system? Do you use conda, or just module load things?

Minor but useful sidenote: There actually is an afni_proc.py demo of integrating ME-FMRI data with RetroTS.py physio processing, from this OHBM poster:
A multi-echo FMRI processing demo including TEDANA in afni_proc.py pipelines
Paul Taylor, Stephen J. Gotts, Adrian W. Gilmore, Joshua Teves, Richard C. Reynolds
https://afni.nimh.nih.gov/pub/dist/OHBM2022/OHBM2022_tayloretal_apmulti.pdf
… and the demo is downloadable with:


@Install_APMULTI_Demo1_rest

… with both HPC scripts for NIH’s Biowulf cluster (scripts_biowulf) and for desktop (scripts_desktop).

In the scripts_desktop_do_14_physio.tcsh, is this example command:


RetroTS.py -prefix ${subj}_${ses}_${task_label} \
           -r ${physio_resp}                    \
           -c ${physio_card}                    \
           -p 50 -n 33 -v 2.2

… where “-v 2.2” signifies a volumetric TR of 2.2s, and “-n 33” means that the volume has n=33 slices. The multi-echo aspect comes in when you process all the EPI information, combining them into a single value per TR (through, for example, optimal combination (OC) or tedana). In the demo, that FMRI processing is done using afni_proc.py and there are various ME-combination methods: OC, old Kundu et al. MEICA and newer MEICA group tedana (mtedana).

–pt

axlander1 · April 11, 2023, 5:40pm

Hi Paul.

Thanks for the tips and the links to the poster and demo for ME processing. I’ve put the results of the system check at the bottom. My responses are below.

Best,
Alex

So, that RetroTS.py program does have pylab as a module dependency. I would have thought
installing matplotlib would install that as well, in most situations.

I probably need to put the pylab module inside my virtual Python environment. I’m still figuring out how to configure all of that, and will work with the people who manage our cluster.

In the scripts_desktop_do_14_physio.tcsh, is this example command:


RetroTS.py -prefix ${subj}_${ses}_${task_label} \
 -r ${physio_resp}                    \
 -c ${physio_card}                    \
 -p 50 -n 33 -v 2.2

… where “-v 2.2” signifies a volumetric TR of 2.2s, and “-n 33” means that the volume has n=33 slices. The multi-echo aspect comes in when you
process all the EPI information, combining them into a single value per TR (through, for example,
optimal combination (OC) or tedana). In the demo, that FMRI processing is done using afni_proc.py
and there are various ME-combination methods: OC, old Kundu et al. MEICA and newer MEICA group
tedana (mtedana).

Thanks for the clarification on the structure of the RetroTS.py script. That now makes sense to me.

Could you perphaps copy+paste the output of:


> afni_system_check.py -check_all
>

on that system?

Here are the results of running afni_system_check.py -check_all

FinalScripts$ afni_system_check.py -check_all
-------------------------------- general ---------------------------------
architecture: 64bit ELF
cpu type: x86_64
system: Linux
release: 4.4.0-206-generic
version: #238-Ubuntu SMP Tue Mar 16 07:52:37 UTC 2021
distribution: debian stretch/sid
number of CPUs: 24
apparent login shell: bash
shell RC file: .bashrc (exists)

--------------------- AFNI and related program tests ---------------------
which afni : /usr/global/afni/23.0.07/afni
afni version : Precompiled binary linux_ubuntu_16_64: Mar 1 2023
: AFNI_23.0.07 ‘Commodus’
AFNI_version.txt : AFNI_23.0.07, linux_ubuntu_16_64, Mar 01 2023
which python : /remote_home/stevenal/myenvs/py3/bin/python
python version : 3.7.9
which R : /usr/bin/R
R version : R version 3.2.3 (2015-12-10) – “Wooden Christmas-Tree”
which tcsh : /usr/bin/tcsh

instances of various programs found in PATH:
afni : 3
/usr/global/afni/23.0.07/afni
/usr/lib/afni/bin/afni_wrapper
/usr/lib/afni/bin/afni
R : 1 (/usr/bin/R)
python : 2
/usr/global/python/versions/pypy3.7-7.3.3/bin/pypy3
/usr/bin/python2.7
python2 : 1 (/usr/bin/python2.7)
python3 : 2
/usr/global/python/versions/pypy3.7-7.3.3/bin/pypy3
/usr/bin/python3.5

testing ability to start various programs…
afni : success
suma : success
3dSkullStrip : success
uber_subject.py : success
3dAllineate : success
3dRSFC : success
SurfMesh : success
3dClustSim : success
3dMVM : success

checking for R packages…
rPkgsInstall -pkgs ALL -check : FAILURE

    oo Warning: 
       These packages are not installed on the computer: afex!
    These packages are not installed on the computer: lmerTest!
    These packages are not installed on the computer: paran!
    These packages are not installed on the computer: psych!
    These packages are not installed on the computer: brms!
    These packages are not installed on the computer: corrplot!
    These packages are not installed on the computer: metafor!

R RHOME : /usr/lib/R

checking for $HOME files…
.afnirc : found
.sumarc : found
.afni/help/all_progs.COMP : found

------------------------------ python libs -------------------------------
** failed to load module PyQt4
– PyQt4 is no longer needed for an AFNI bootcamp

** failed to load module matplotlib.pyplot
– matplotlib.pyplot is required

-------------------------------- env vars --------------------------------
PATH = /usr/global/afni/23.0.07:/remote_home/stevenal/myenvs/py3/bin:/usr/global/freesurfer/7.1.1/bin:/usr/global/freesurfer/7.1.1/fsfast/bin:/usr/global/freesurfer/7.1.1/tktools:/usr/global/fsl/6.0.4/bin:/usr/global/freesurfer/6.0.0/mni/bin:/usr/lib/fsl/5.0:/remote_home/stevenal/bin:/remote_home/stevenal/.local/bin:/remote_home/stevenal/abin:/usr/global/freesurfer/6.0.0/bin:/usr/global/freesurfer/6.0.0/fsfast/bin:/usr/global/freesurfer/6.0.0/tktools:/usr/global/camino/bin:/usr/global/ants/2.1.0-xenial/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/lib/afni/bin:/usr/global/bin:/usr/global/4dfp/bin:/usr/global/btk/Applications:/usr/global/btk/Utilities:/usr/global/btk/Scripts:/usr/global/btk/Viewer:/usr/global/caret/bin_linux64:/usr/global/dtiquery-1.1:/usr/global/dtitk/2.3.1/bin:/usr/global/dtitk/2.3.1/utilities:/usr/global/dtitk/2.3.1/scripts

PYTHONPATH =
R_LIBS =
LD_LIBRARY_PATH = /usr/lib/fsl/5.0

DYLD_LIBRARY_PATH =
DYLD_FALLBACK_LIBRARY_PATH =

----------------------------- eval dot files -----------------------------

** warning: .tcshrc does NOT seem to contain ‘source .cshrc’
– (csh and tcsh will use different files)
– considered opertaions: path, flatdir, apsearch
(not on a mac, should skip flatdir)

– note: followers should not need edits, so edit flags should be 0
(have 0 follower(s), which can be ignored)

dot file test : want 5 modifications across 3 files:

  file             path  flatdir  apsearch        follower
  ---------------  ----  -------  --------        --------
  .cshrc           1     0        1               0     
  .tcshrc          1     0        1               0     
  .bashrc          1     0        0               0

------------------------------ data checks -------------------------------
data dir : missing AFNI_data6
data dir : missing AFNI_demos
data dir : missing suma_demo
data dir : missing afni_handouts
atlas : found TT_N27+tlrc under /usr/global/afni/20.3.02
atlas : found TT_N27+tlrc under /usr/share/afni/atlases

------------------------------ OS specific -------------------------------
which apt-get : /usr/bin/apt-get
apt-get version : apt 1.2.32ubuntu0.2 (amd64)

========================= summary, please fix: =========================

just be aware: login shell ‘bash’, but our code examples use ‘tcsh’
have multiple versions of AFNI in PATH
missing R packages (see rPkgsInstall)
python library matplotlib is required
(see AFNI install docs for details)
dot file test : want 5 modifications across 3 files:
insufficient data for AFNI bootcamp
(see “Prepare for Bootcamp” on install pages)

ptaylor · April 11, 2023, 9:43pm

Cool.

You can set up miniconda/conda even on an HPC, since it doesn’t require sudo permissions. Here are notes related to conda and AFNI specifically:
https://afni.nimh.nih.gov/pub/dist/doc/htmldoc/background_install/miniconda.html

Basically, you create a directory of useful programs with dependencies in a very self-contained directory, say in your subfolder of your home directory. I think by just using the conda install recipes you will be set. It can be used to manage more than just Python, but these should be all you would need at the moment for Python 3.*, as noted in the notes above:
“matplotlib>=2.2.3” numpy scipy
… as pylab will come with matplotlib (and you would likely find matplotlib useful for other things, such as the afni_proc.py QC HTML).

–pt

axlander1 · April 12, 2023, 6:00pm

Thanks Paul. I was actually able to get the APMULTI demos to run by loading modules, but I think conda may be a more elegant solution. I’ll test that out as well. Thanks again for the quick responses and help.

Alex

ptaylor · April 13, 2023, 2:41am

Hi, Alex-

Module loading is certainly reasonable on an HPC system—to some extent, it is probably expected. You can likely even add those things to your scripts to run. Depending on how that system operates, the “scripts_biowulf/” directory might be useful, because it scripts HPC features like swarming jobs and using a temporary scratch disk to write to and copy back from. How directly useful those will be depends on the specific system, setup, etc.

I do find conda pretty broadly useful, too, though, to fill in missing links on various systems. It has the great benefit of really being quite modular—everything gets put into one place, and each environment has its own directory. If you are unhappy with an environment, it can really be purged (rather than some other package managers, where things are splashed around more, and more difficult to “undo” in a deep way).

Please feel free to ask any questions here; running


afni_system_check.py -check_all

is a good place to start with gauging how well things are going. And perhaps you shouldn’t even worry about some of the R dependencies to start–those basically matter at the group level.

–pt