calling afni python scrips through MATLAB

AFNI Message Board
1

Hi all,

we run most of our processing pipeline through MATLAB for several reasons, but a recent update to MATLAB has introduced an issue with the afni python scrips (e.g., 1d_tool.py).

for example,


unix(['1d_tool.py -help')

returns


env: python: No such file or directory

ans =

   127

but, when run in zsh, the expected help text is returned.

I know this isn’t an afni specific issue, but I’m curious – has anyone come across this issue?

Andrew

2

Hi, Andrew-

Caveat: I don’t use Matlab, so don’t have a deep answer here—maybe not even a helpful one, it might be best classified as “thinking out loud”.

I just tested running Matlab on our HPC cluster here, and running:


unix(['1d_tool.py -help'])

It worked successfully for me, so the issue doesn’t seem to be inherent to the program file.

My understanding is that the return code of 127 means that the system cannot find that command. You noted that in a zsh terminal, you can run “1d_tool.py” and see the program help. But I wonder if your Matlab is using a different shell, one that doesn’t have the AFNI binaries directory added to its path, perhaps (or, from that specific message, perhaps not even python)?

What is the output of each of the following for you:
A) to see what shell Matlab is using


unix(['echo $0'])

B) to see which python is there/available


unix(['which python'])

C) to see what version of python that is


unix(['python -V'])

D) the shebang of the Python programs in AFNI uses ‘/usr/bin/env …’, which in some system configurations can be different that the standard ‘python’ (because people set up aliases or other things):


unix(['/usr/bin/env python -V'])

–pt

3

The sh shell is used by default in Matlab. I tend to use the system() command instead of unix(), but I don’t think that will be critical here. Take a look at this previous post to see if the Matlab specific environment variables help.

https://afni.nimh.nih.gov/afni/community/board/read.php?1,158256,158279

4

Thanks for the quick reply.

Other AFNI scripts work. I run the following code to make sure AFNI and MATLAB are working together:


setenv('DYLD_FALLBACK_LIBRARY_PATH',[ '/Users/conradbn/abin:' getenv('DYLD_FALLBACK_LIBRARY_PATH') ])

% is the path to AFNI already set?
[afninotfound, unixoutput] = unix('3dcopy');
% if not, then we will look for it...
if afninotfound
    % is AFNI in /abin?
    [afninotfound, unixoutput] = unix('/abin/3dcopy');
    if ~afninotfound % found it!
        setenv('PATH', [getenv('PATH') ':/abin']);
        disp(sprintf('Adding /abin to Matlab''s path (for AFNI).'));
    end
end
if afninotfound
    % is AFNI in ~/abin?
    [afninotfound, unixoutput] = unix('~/abin/3dcopy');
    if ~afninotfound % found it!
        current_directory = pwd;
        eval('cd ~');
        afni_directory = [pwd '/abin'];
        setenv('PATH', [getenv('PATH') ':' afni_directory]);
        disp(sprintf(['Adding ' afni_directory ' to Matlab''s path (for AFNI).']));
        eval(['cd ' current_directory]);
        clear current_directory afni_directory;
    end
end
if afninotfound
    % is AFNI in ~/Documents/abin?
    [afninotfound, unixoutput] = unix('~/Documents/abin/3dcopy');
    if ~afninotfound % found it!
        current_directory = pwd;
        eval('cd ~');
        afni_directory = [pwd '/Documents/abin'];
        setenv('PATH', [getenv('PATH') ':' afni_directory]);
        disp(sprintf(['Adding ' afni_directory ' to Matlab''s path (for AFNI).']));
        eval(['cd ' current_directory]);
        clear current_directory afni_directory;
    end
end
if afninotfound
    % if AFNI is not in /abin nor ~/abin, then the user will have to add
    % the correct path to AFNI before this function will work.
    disp(sprintf('*****'));
    disp(sprintf('The required AFNI functions cannot be found.'));
    disp(sprintf('AFNI needs to be added to Matlab''s path.'));
    disp(sprintf('You can do this with the following command:'));
    disp(sprintf(' '));
    disp(sprintf('   setenv(''PATH'', [getenv(''PATH'') '':/abin'']);'));
    disp(sprintf(' '));
    disp(sprintf('where (as an example) ''/abin'' is the AFNI directory.'));
    disp(sprintf('Exiting...'));
    disp(sprintf('*****'));
    return;
end
clear afninotfound unixoutput;

To answer your questions:
The unix env is in zsh.


>> unix(['echo $0'])
/bin/zsh

ans =

     0

However, python is not recognized. This is interested because other versions of Matlab have worked.


>> unix(['which python'])
python not found

ans =

     1

>> unix(['python -V'])
zsh:1: command not found: python

ans =

   127

>> unix(['/usr/bin/env python -V'])
env: python: No such file or directory

ans =

   127

I’ve run these on a different computer with MATLAB 2019a and I get the expected returns that are provided from terminal.

I suspect I need to add a few lines of code to my start up script above to map python to the unix/system env in MATLAB?

5

Something like what you suggest sounds reasonable.

How does that system have Python installed (outside of Matlab, that is, just in native run-mode)? Are you using conda, or some kind of alias?

It might be useful to see:


afni_system_check.py -check_all

–pt

6

yes, I have python installed via miniconda3


-------------------------------- general ---------------------------------
architecture:         64bit 
system:               Darwin
release:              21.4.0
version:              Darwin Kernel Version 21.4.0: Mon Feb 21 20:34:37 PST 2022; root:xnu-8020.101.4~2/RELEASE_X86_64
distribution:         10.16
number of CPUs:       6
apparent login shell: zsh
shell RC file:        .zshrc (exists)

--------------------- AFNI and related program tests ---------------------
which afni           : /Users/andrewlynn/abin/afni
afni version         : Precompiled binary macos_10.12_local: Dec  3 2021 
                     : AFNI_21.3.10 'Trajan'
AFNI_version.txt     : AFNI_21.3.10, macos_10.12_local, Dec 03 2021
which python         : /Users/andrewlynn/miniconda3/bin/python
python version       : 3.9.7
which R              : /usr/local/bin/R
R version            : R version 3.6.3 (2020-02-29) -- "Holding the Windsock"
which tcsh           : /bin/tcsh

instances of various programs found in PATH:
    afni    : 1   (/Users/andrewlynn/abin/afni)
    R       : 1   (/Library/Frameworks/R.framework/Versions/3.6/Resources/bin/R)
    python  : 1   (/Users/andrewlynn/miniconda3/bin/python3.9)
    python2 : 0 
    python3 : 3 
      /Users/andrewlynn/miniconda3/bin/python3.9
      /Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10
      /usr/bin/python3

** have python3 but not python2

testing ability to start various programs...
    afni                 : success
    suma                 : success
    3dSkullStrip         : success
    uber_subject.py      : success
    3dAllineate          : success
    3dRSFC               : success
    SurfMesh             : success
    3dClustSim           : success
    3dMVM                : success

checking for R packages...
    rPkgsInstall -pkgs ALL -check : success

R RHOME : /Library/Frameworks/R.framework/Resources

checking for $HOME files...
    .afnirc                   : found
    .sumarc                   : found
    .afni/help/all_progs.COMP : found

------------------------------ python libs -------------------------------
** failed to load module PyQt4
-- PyQt4 is no longer needed for an AFNI bootcamp

** failed to load module matplotlib.pyplot
-- matplotlib.pyplot is not required, but is desirable

-- python binaries under /usr/local/bin:
    /usr/local/bin/python3 (sym link to /Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10)
    /usr/local/bin/python3.9 (sym link to /usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/bin/python3.9)

-------------------------------- env vars --------------------------------
PATH = /Users/andrewlynn/miniconda3/bin:/Users/andrewlynn/miniconda3/condabin:/Library/Frameworks/Python.framework/Versions/3.10/bin:/Applications/freesurfer/7.1.1/bin:/Applications/freesurfer/7.1.1/fsfast/bin:/Applications/freesurfer/7.1.1/mni/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Library/Apple/usr/bin:/Users/andrewlynn/abin

PYTHONPATH = 
R_LIBS = 
LD_LIBRARY_PATH = 
zsh compinit: insecure directories, run compaudit for list.
Ignore insecure directories and continue [y] or abort compinit [n]? y
DYLD_LIBRARY_PATH (sub-shell) = :/opt/X11/lib/flat_namespace
zsh compinit: insecure directories, run compaudit for list.
Ignore insecure directories and continue [y] or abort compinit [n]? DYLD_FALLBACK_LIBRARY_PATH (sub-shell) = 

------------------------------ data checks -------------------------------
data dir : missing AFNI_data6
data dir : missing AFNI_demos
data dir : missing suma_demo
data dir : missing afni_handouts
atlas    : found TT_N27+tlrc  under /Users/andrewlynn/abin

------------------------------ OS specific -------------------------------
XQuartz version      : 2.8.1

which brew           : /usr/local/bin/brew
brew version         : Homebrew 3.2.7

++ found PyQt4 under /usr/local/lib/python2.7/site-packages
** warning: have brew? PyQt4, but non-brew python /Users/andrewlynn/miniconda3/bin/python
-- consider installing gcc under homebrew
-- consider installing glib under homebrew
++ found 1 dylib files under '/opt/X11/lib/flat_namespace'
   -- found 'libXt' dylib files:
      /opt/X11/lib/flat_namespace/libXt.6.dylib
-- recent OS X, cheating to check DYLD_LIBRARY_PATH in cur shell 'zsh'...
zsh compinit: insecure directories, run compaudit for list.
Ignore insecure directories and continue [y] or abort compinit [n]? y
++ found evar DYLD_LIBRARY_PATH = :/opt/X11/lib/flat_namespace
-- recent OS X, cheating to check DYLD_LIBRARY_PATH in shell 'tcsh'...
** env var DYLD_LIBRARY_PATH not set to contain /opt/X11/lib/flat_namespace

=========================  summary, please fix:  =========================
*  just be aware: login shell 'zsh', but our code examples use 'tcsh'
*  insufficient data for AFNI bootcamp
   (see "Prepare for Bootcamp" on install pages)
*  please set DYLD_LIBRARY_PATH to /opt/X11/lib/flat_namespace in tcsh
7

adding the following code to the startup script fixed the problem:


pydir = '/Users/andrewlynn/miniconda3/bin';
setenv('PATH',[getenv('PATH') ':' pydir])

I’m now troubleshooting a different issue…

Thanks for your help!

8

Thanks for posting the system check output.

On a preliminary sidenote, the “Please Fix” section suggests doing something like copy+pasting the following:


echo "" >> ~/.cshrc
echo "setenv DYLD_LIBRARY_PATH /opt/X11/lib/flat_namespace" >> ~/.cshrc

… which should be totally unrelated to the present Python question.

Back to the Python stuff:
How do you start a conda environment in your normal shell usage? I think the problem lies with your matlab-called zsh doesn’t appear to have your conda environment started up.

Also, I would assume it would be easier to have the AFNI binaries added to your path from the shell setup (outside matlab), then to try to add/append path things in Matlab. But, based on your afni_system_check.py output, I guess that is the case? (Did you run that system_check from your matlab env, or from the terminal command line itself?)

–pt

9

Okeydoke.

–pt