tcsh vs. Miniconda

annschoe · January 18, 2021, 9:33pm

Good afternoon,

I have been getting this error message when I run the afni_system_check command:


------------------------------ python libs -------------------------------
++ module loaded: PyQt4
   module file : /usr/local/lib/python2.7/site-packages/PyQt4/__init__.pyc

** failed to load module matplotlib.pyplot
-- matplotlib.pyplot is not required, but is desirable

-- python binaries under /usr/local/bin:
    /usr/local/bin/python3 (sym link to /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/bin/python3.8)
    /usr/local/bin/python3.8 (sym link to /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/bin/python3.8)

I eventually figured out that a python 2.7 vs. 3.8 issue was at play, so I followed your advice to install Miniconda to better manage the multiple python versions on my system. This solved the problem beautifully, and when I ran the afni_system_check, I got the following message:


------------------------------ python libs -------------------------------
++ module loaded: PyQt4
   module file : /usr/local/lib/python2.7/site-packages/PyQt4/__init__.pyc

++ module loaded: matplotlib.pyplot
   module file : /Users/achoe2/miniconda3/envs/py27_afni_tiny/lib/python2.7/site-packages/matplotlib/pyplot.pyc

-- python binaries under /usr/local/bin:
    /usr/local/bin/python3 (sym link to /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/bin/python3.8)
    /usr/local/bin/python3.8 (sym link to /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/bin/python3.8)

I was pretty happy about this, until… I tried to run my AP scripts, which, unfortunately are all tcsh scripts – and as far as I can tell, conda doesn’t run on tcsh.

So at this point, I’m hoping that you could tell me that I am totally wrong, conda can be run using tcsh shell. But if this is not the case, would you please recommend how I should proceed? My bash shell scripting skills are quite minimal, and all the example scripts from the AFNI tutorials are tcsh scripts.

Sincerely,
Ann

ptaylor · January 19, 2021, 1:25am

Hi, Ann-

For some reason I don’t understand, conda folks make finishing their installation a pain with tcsh. It boggles my mind.

First, a grumble: Conda just sets up bash shell by default on installation, putting necessary pointer information in ~/bashrc on your system. In theory, the conda folks eventually “helped” make installations in non-bash shells easier by making the following command available (among a similar one for other shells):


conda init tcsh

This puts necessary path information into your ~/.*rc file for a given shell, so it knows where to find conda on your computer. The text it creates contains an if-condition between some comment lines with “conda initialize” in them, like this:


# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
[if condition stuff, syntax based on particular shell]
# <<< conda initialize <<<

… however, this does not set up your ~/.cshrc (which is what we use for tcsh setup) but instead goes for your ~/.tcshrc file (creating it if it doesn’t exist already). Why they don’t have “conda init csh”, I cannot say (tcsh and csh have essentially the same syntax, one derived from the other).

OK, what you can do to set things up:
See if you have ~/.tcshrc already, by copy+pasting:


\ls ~/.tcshrc

If that file does not exist, you will get an error message like “ls: cannot access …”. If it does exist, you will just see that file shown with full path. (This will slightly affect the steps taken, below.)

Now, run this:


conda init tcsh

… to create the conda path information in ~/.tcshrc.

If the file did exist from the start (from check above), then open it up (e.g., “afni_open -e ~/.tcshrc”), as well as the ~/.cshrc file (e.g., “afni_open -e ~/.cshrc”); copy just the lines that conda has created in ~/.tcshrc between the “# >>> conda initialize >>>” and “# <<< conda initialize <<<” comments—including those comments themselves—and paste them in your ~/.cshrc file. Then, open a new terminal and see if things work.

If the file did not exist from the start (from check above) then open it up, because it exists now (e.g., “afni_open -e ~/.tcshrc”), as well as the ~/.cshrc file (e.g., “afni_open -e ~/.cshrc”); copy the lines that conda has created in ~/.tcshrc between the “# >>> conda initialize >>>” and “# <<< conda initialize <<<” comments—including those comments themselves—and paste them in your ~/.cshrc file. Then, close those two files, and copy+paste this in a terminal:


\mv ~/.tcshrc ~/.tcshrc_backup

(which just renames it, so it won’t get sourced instead of ~/.cshrc). Now, open a new terminal and see if things work.

Wheeeeee.

–pt

annschoe · January 19, 2021, 4:59am

Hello pt,

Not Wheeeeeee

I have followed your instruction and conda now runs in tcsh (good). I am now getting error messages in both bash and tcsh (not so good). One thing of note is that in addition to following your instructions, I’ve also updated afni to the AFNI_21.0.04 ‘Titus’.

Here is the result of the newly failed afni_system_check, ran using bash


-------------------------------- general ---------------------------------
architecture:         64bit 
system:               Darwin
release:              19.6.0
version:              Darwin Kernel Version 19.6.0: Tue Nov 10 00:10:30 PST 2020; root:xnu-6153.141.10~1/RELEASE_X86_64
distribution:         10.15.7
number of CPUs:       8
apparent login shell: tcsh  (current shell is bash)
shell RC file:        .cshrc (exists)

--------------------- AFNI and related program tests ---------------------
which afni           : /Users/achoe2/abin/afni
afni version         : Precompiled binary macos_10.12_local: Jan 18 2021 
                     : AFNI_21.0.04 'Titus'
AFNI_version.txt     : AFNI_21.0.04, macos_10.12_local, Jan 18 2021
which python         : /Users/achoe2/miniconda3/bin/python
python version       : 3.8.5
which R              : /usr/local/bin/R
R version            : R version 3.6.3 (2020-02-29) -- "Holding the Windsock"
which tcsh           : /bin/tcsh

instances of various programs found in PATH:
    afni    : 1   (/Users/achoe2/abin/afni)
    R       : 1   (/Library/Frameworks/R.framework/Versions/3.6/Resources/bin/R)
    python  : 4 
      /Users/achoe2/miniconda3/bin/python3.8
      /Users/achoe2/miniconda3/envs/py27_afni_tiny/bin/python2.7
      /opt/sw/bin/python2.7
      /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
    python2 : 2 
      /Users/achoe2/miniconda3/envs/py27_afni_tiny/bin/python2.7
      /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
    python3 : 3 
      /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/bin/python3.8
      /Users/achoe2/miniconda3/bin/python3.8
      /usr/bin/python3


testing ability to start various programs...
    afni                 : success
    suma                 : success
    3dSkullStrip         : success
    uber_subject.py      : success
    3dAllineate          : success
    3dRSFC               : success
    SurfMesh             : success
    3dClustSim           : success
    3dMVM                : success

checking for R packages...
    rPkgsInstall -pkgs ALL -check : success

R RHOME : /Library/Frameworks/R.framework/Resources

checking for $HOME files...
    .afnirc                   : found
    .sumarc                   : found
    .afni/help/all_progs.COMP : found

------------------------------ python libs -------------------------------
++ module loaded: PyQt4
   module file : /usr/local/lib/python2.7/site-packages/PyQt4/__init__.py

** have PyQt4, but cannot load QtCore, QtGui; error is:

   **************************************************
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: dlopen(/usr/local/lib/python2.7/site-packages/PyQt4/QtCore.so, 2): Symbol not found: _PyInt_Type
  Referenced from: /usr/local/lib/python2.7/site-packages/PyQt4/QtCore.so
  Expected in: flat namespace
 in /usr/local/lib/python2.7/site-packages/PyQt4/QtCore.so
   **************************************************


** failed to load module matplotlib.pyplot
-- matplotlib.pyplot is not required, but is desirable

-- python binaries under /usr/local/bin:
    /usr/local/bin/python3 (sym link to /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/bin/python3.8)
    /usr/local/bin/python3.8 (sym link to /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/bin/python3.8)

-------------------------------- env vars --------------------------------
PATH = /usr/local/bin:/Users/achoe2/miniconda3/bin:/Users/achoe2/miniconda3/envs/py27_afni_tiny/bin:/Users/achoe2/miniconda3/condabin:/opt/sw/bin:/opt/sw/sbin:/Applications/freesurfer/7.1.1/bin:/Applications/freesurfer/7.1.1/fsfast/bin:/Applications/freesurfer/7.1.1/mni/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/opt/X11/bin:/Library/Apple/usr/bin:/Users/achoe2/abin:/Users/achoe2/abin

PYTHONPATH = /usr/local/lib/python2.7/site-packages

R_LIBS = 
LD_LIBRARY_PATH = 
DYLD_LIBRARY_PATH (sub-shell) = :/opt/X11/lib/flat_namespace
DYLD_FALLBACK_LIBRARY_PATH (sub-shell) = /Users/achoe2/abin

------------------------------ data checks -------------------------------
data dir : missing AFNI_data6
data dir : missing AFNI_demos
data dir : missing suma_demo
data dir : missing afni_handouts
atlas    : found TT_N27+tlrc  under .

------------------------------ OS specific -------------------------------
XQuartz version      : 2.7.11

which fink           : /opt/sw/bin/fink
fink version         : Package manager version: 0.45.2
which brew           : /usr/local/bin/brew
brew version         : Homebrew 2.7.4

++ found 'brew' at /usr/local/bin/brew
++ found PyQt4 under /usr/local/lib/python2.7/site-packages
** warning: have brew? PyQt4, but non-brew python /Users/achoe2/miniconda3/bin/python
++ found 1 dylib files under '/opt/X11/lib/flat_namespace'
   -- found 'libXt' dylib files:
      /opt/X11/lib/flat_namespace/libXt.6.dylib
-- recent OS X, cheating to check DYLD_LIBRARY_PATH in cur shell 'bash'...
++ found evar DYLD_LIBRARY_PATH = :/opt/X11/lib/flat_namespace
-- recent OS X, cheating to check DYLD_LIBRARY_PATH in shell 'tcsh'...
++ found evar DYLD_LIBRARY_PATH = /Applications/freesurfer/7.1.1/lib/gcc/lib:/opt/X11/lib/flat_namespace

=========================  summary, please fix:  =========================
*  check for partial install of PyQt4
*  insufficient data for AFNI bootcamp

And here is the previously successful output of afni_system_check, ran on bash, before the miniconda-on-tcsh change.


-------------------------------- general ---------------------------------
architecture:         64bit 
system:               Darwin
release:              19.6.0
version:              Darwin Kernel Version 19.6.0: Tue Nov 10 00:10:30 PST 2020; root:xnu-6153.141.10~1/RELEASE_X86_64
distribution:         10.15.7
number of CPUs:       8
apparent login shell: tcsh  (current shell is bash)
shell RC file:        .cshrc (exists)

--------------------- AFNI and related program tests ---------------------
which afni           : /Users/achoe2/abin/afni
afni version         : Precompiled binary macos_10.12_local: Jan 14 2021 
                     : AFNI_21.0.03 'Titus'
AFNI_version.txt     : AFNI_21.0.03, macos_10.12_local, Jan 14 2021
which python         : /Users/achoe2/miniconda3/envs/py27_afni_tiny/bin/python
python version       : 2.7.18
which R              : /usr/local/bin/R
R version            : R version 3.6.3 (2020-02-29) -- "Holding the Windsock"
which tcsh           : /bin/tcsh

instances of various programs found in PATH:
    afni    : 1   (/Users/achoe2/abin/afni)
    R       : 1   (/Library/Frameworks/R.framework/Versions/3.6/Resources/bin/R)
    python  : 3 
      /Users/achoe2/miniconda3/envs/py27_afni_tiny/bin/python2.7
      /opt/sw/bin/python2.7
      /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
    python2 : 2 
      /Users/achoe2/miniconda3/envs/py27_afni_tiny/bin/python2.7
      /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
    python3 : 2 
      /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/bin/python3.8
      /usr/bin/python3


testing ability to start various programs...
    afni                 : success
    suma                 : success
    3dSkullStrip         : success
    uber_subject.py      : success
    3dAllineate          : success
    3dRSFC               : success
    SurfMesh             : success
    3dClustSim           : success
    3dMVM                : success

checking for R packages...
    rPkgsInstall -pkgs ALL -check : success

R RHOME : /Library/Frameworks/R.framework/Resources

checking for $HOME files...
    .afnirc                   : found
    .sumarc                   : found
    .afni/help/all_progs.COMP : found

------------------------------ python libs -------------------------------
++ module loaded: PyQt4
   module file : /usr/local/lib/python2.7/site-packages/PyQt4/__init__.pyc

++ module loaded: matplotlib.pyplot
   module file : /Users/achoe2/miniconda3/envs/py27_afni_tiny/lib/python2.7/site-packages/matplotlib/pyplot.pyc

-- python binaries under /usr/local/bin:
    /usr/local/bin/python3 (sym link to /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/bin/python3.8)
    /usr/local/bin/python3.8 (sym link to /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/bin/python3.8)

-------------------------------- env vars --------------------------------
PATH = /Users/achoe2/miniconda3/envs/py27_afni_tiny/bin:/Users/achoe2/miniconda3/condabin:/usr/local/bin:/opt/sw/bin:/opt/sw/sbin:/Applications/freesurfer/7.1.1/bin:/Applications/freesurfer/7.1.1/fsfast/bin:/Applications/freesurfer/7.1.1/mni/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/opt/X11/bin:/Library/Apple/usr/bin:/Users/achoe2/abin:/Users/achoe2/abin

PYTHONPATH = /usr/local/lib/python2.7/site-packages

R_LIBS = 
LD_LIBRARY_PATH = 
DYLD_LIBRARY_PATH (sub-shell) = :/opt/X11/lib/flat_namespace
DYLD_FALLBACK_LIBRARY_PATH (sub-shell) = /Users/achoe2/abin

------------------------------ data checks -------------------------------
data dir : missing AFNI_data6
data dir : missing AFNI_demos
data dir : missing suma_demo
data dir : missing afni_handouts
atlas    : found TT_N27+tlrc  under .

------------------------------ OS specific -------------------------------
XQuartz version      : 2.7.11

which fink           : /opt/sw/bin/fink
fink version         : Package manager version: 0.45.2
which brew           : /usr/local/bin/brew
brew version         : Homebrew 2.7.4

++ found 'brew' at /usr/local/bin/brew
++ found 1 dylib files under '/opt/X11/lib/flat_namespace'
   -- found 'libXt' dylib files:
      /opt/X11/lib/flat_namespace/libXt.6.dylib
-- recent OS X, cheating to check DYLD_LIBRARY_PATH in cur shell 'bash'...
++ found evar DYLD_LIBRARY_PATH = :/opt/X11/lib/flat_namespace
-- recent OS X, cheating to check DYLD_LIBRARY_PATH in shell 'tcsh'...
++ found evar DYLD_LIBRARY_PATH = /opt/X11/lib/flat_namespace:/opt/X11/lib/flat_namespace

=========================  summary, please fix:  =========================
*  insufficient data for AFNI bootcamp

In both cases, the python 2.7 is activated via conda, but I see that the failed afni_system_check seems to be picking the base(?) regardless. Could this be the issue? I am also noticing that there are suddenly more instances of python, python2, and python3 after the change. Finally, I am also noticing something about non-brewed pyqt4…

help?

Sincerely,
Ann

ptaylor · January 19, 2021, 2:09pm

Hi, Ann-

There are indeed multiple instances of python:

the ones with “miniconda3” in the path are from your conda setup
the /System/Library/* one is from the basic Mac OS setup
the /opt/sw/* one is probably from macports, judging by path (I think that installs under /opt/; fink installs under /sw by default)
… but that might not be such a problem.

You can type “which python” in the command line to see excactly which python your current shell is using. The afni_system_check.py shows you that as well:

for current bash, it is: /Users/achoe2/miniconda3/bin/python
for earlier bash, it is: /Users/achoe2/miniconda3/envs/py27_afni_tiny/bin/python
I think that means that at present, your default conda environment python is just the “base” one (a Python 3.8 that liekly does not have any other modules loaded), whereas before it was one you created (a python 2.7, likely with a couple things listed for AFNI installation, judging by the name).

If you type “conda env list” in the terminal, it will show you all the environments you have to choose among; the one that is currently “active” has an asterisk just after its name. What is the output if you have “conda env list”?

To have a particular conda environment active by default in your shell, you can put “conda activate [name]” after the conda initialize block of code in the shell’s “~/.bashrc” or “~/.cshrc” file. For example, you could put “conda activate py27_afni_tiny”, or something with Python 3.* that has additional modules.

Another question: from your setup, it looks like you have tcsh as your default login shell, but you switch to using bash… would you prefer to have bash be your default login shell?

–pt

ps: for clarification, the “whee” was not to suggest that this was easy, but actually meant ironically that setting up conda with tcsh is so complicated…

annschoe · January 19, 2021, 4:33pm

Hello pt,

So here are my answers to your questions

When I type conda env list, the terminal shows that I have python 2.7 activated.
The reason I showed you the afni_sytem_check results produced in bash, not tcsh (my preferred shell) is that I wanted to let you do a side-by-side comparison of failed and successful results obtained from the same shell. This, because I never had a successful run of the afni_system_check under the tcsh.

And here is what I’ve discovered:

When I delete the following lines from the .cshrc file, conda no longer works in tcsh, but the afni_system_check again starts to run successfully in bash


# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
if ( -f "/Users/achoe2/miniconda3/etc/profile.d/conda.csh" ) then
    source "/Users/achoe2/miniconda3/etc/profile.d/conda.csh"
else
    setenv PATH "/Users/achoe2/miniconda3/bin:$PATH"
endif
# <<< conda initialize <<<

And here is the /Users/achoe2/miniconda3/etc/profile.d/conda.csh


setenv CONDA_EXE "/Users/achoe2/miniconda3/bin/conda"
setenv _CONDA_ROOT "/Users/achoe2/miniconda3"
setenv _CONDA_EXE "/Users/achoe2/miniconda3/bin/conda"
setenv CONDA_PYTHON_EXE "/Users/achoe2/miniconda3/bin/python"
echo "Copyright (C) 2012 Anaconda, Inc" > /dev/null
echo "SPDX-License-Identifier: BSD-3-Clause" > /dev/null

if (! $?_CONDA_EXE) then
  set _CONDA_EXE="${PWD}/conda/shell/bin/conda"
else
  if ("$_CONDA_EXE" == "") then
      set _CONDA_EXE="${PWD}/conda/shell/bin/conda"
  endif
endif

if ("`alias conda`" == "") then
    if ($?_CONDA_EXE) then
        # _CONDA_PFX is named so as not to cause confusion with CONDA_PREFIX
        # If nested backticks were possible we wouldn't use any variables here.
        set _CONDA_PFX=`dirname "${_CONDA_EXE}"`
        set _CONDA_PFX=`dirname "${_CONDA_PFX}"`
        alias conda source "${_CONDA_PFX}/etc/profile.d/conda.csh"
        # And for good measure, get rid of it afterwards.
        unset _CONDA_PFX
    else
        alias conda source "${PWD}/conda/shell/etc/profile.d/conda.csh"
    endif
    setenv CONDA_SHLVL 0
    if (! $?prompt) then
        set prompt=""
    endif
else
    setenv PATH "`dirname ${_CONDA_EXE}`:$PATH"
    switch ( "${1}" )
        case "activate":
            set ask_conda="`(setenv prompt '${prompt}' ; '${_CONDA_EXE}' shell.csh activate '${2}' ${argv[3-]})`" || exit ${status}
            eval "${ask_conda}"
            rehash
            breaksw
        case "deactivate":
            set ask_conda="`(setenv prompt '${prompt}' ; '${_CONDA_EXE}' shell.csh deactivate '${2}' ${argv[3-]})`" || exit ${status}
            eval "${ask_conda}"
            rehash
            breaksw
        case "install" | "update" | "upgrade" | "remove" | "uninstall":
            $_CONDA_EXE $argv[1-]
            set ask_conda="`(setenv prompt '${prompt}' ; '${_CONDA_EXE}' shell.csh reactivate)`" || exit ${status}
            eval "${ask_conda}"
            rehash
            breaksw
        default:
            $_CONDA_EXE $argv[1-]
            breaksw
    endsw
endif

So if this bit of code (conda.csh) is run on the tcsh shell, while the conda can now run(?) in tcsh, the conda no longer responds to the ‘conda activate py27_afni_tiny’ command. In fact, once the conda.csh is run, no matter which environment I activate, the ‘which python’ returns the same path (i.e., /Users/achoe2/miniconda3/bin/python). Prior to running conda.csh, if I activate the py27_afni_tiny, and ask ‘which python’, the path is correctly set for /Users/achoe2/miniconda3/envs/py27_afni_tiny/bin/python.

It pretty clear to me that something wicked is happening with conda.csh, but I don’t know how to fix it.

Wheeeeee (used in the same context as your previous wheee).

Thank you,
Ann

ptaylor · January 19, 2021, 9:25pm

Hi-

I think we have resolved the issue now.

I believe it was that PYTHONPATH was explicitly set in the ~/.bashrc file, pointing to an old (non-conda) location of Python on the system. Especially when only the base Python (which was version 3.*) was used, that was causing some confusing (because the older version pointed to was version 2.7).

Commenting out the PYTHONPATH value helped.

In each of the ~/.bashrc and ~/.cshrc, then, there was the “conda initialize” block of code for that shell, and then also a line following that with “conda activate [name of desired environment]”. Then things appear to be working.

Additional note: the PyQt4 issues in the above afni_system_check outputs are not important; we don’t use PyQt4 for anything at the moment (and it also has installations in Python ver 3., anyways).

–pt