Where does AFNI look for R

jwaltz · May 13, 2020, 7:07pm

I am running a 3dMVM script on a local machine, and it is taking a very long time. I want to run the next analysis on another node, but that node doesn’t have R. So I have to tell AFNI where to find R. Is that set by an environmental variable?

Thanks,

Jim Waltz

ptaylor · May 13, 2020, 7:43pm

Hi, Jim-

Typically, we add an environment variable in the ~/.bashrc or ~/.cshrc startup file.

For example, on my computer, all my R libraries are in ~/R, so I have the following line in my ~/.bashrc file:


export R_LIBS=$HOME/R

and the following in my ~/.cshrc:


setenv R_LIBS ~/R

Once you resource either of those files and/or open a new terminal, you should be good to go.

–pt

jwaltz · May 13, 2020, 9:48pm

Thanks. That helped. But now that AFNI can find R, I get a different error:

Fatal error: cannot create ‘R_TempDir’

Do you know where R/AFNI is trying to create ‘R_TempDir’, so I can request write permissions?

Thanks,

jw

ptaylor · May 13, 2020, 10:12pm

I don’t know about that.

Can you please add in your 3dMVM command, as well as the output of


afni_system_check.py -check_all

on that computer?

–pt

jwaltz · May 13, 2020, 10:23pm

No shortage of problems, trying to run AFNI on this machine:

sh TUIT_3dMVM_MU_DXxEXP_SZvNC.sh

(‘TUIT_3dMVM_MU_DXxEXP_SZvNC.sh’ is below)

afni_system_check.py -check_all
File “/data/brutus_data0/freeware/afni/afni_system_check.py”, line 279
print g_help_string
^
SyntaxError: Missing parentheses in call to ‘print’. Did you mean print(g_help_string)?

TUIT_3dMVM_MU_DXxEXP_SZvNC.sh

#! bin/sh

cd /data/brutus_data2/jwaltz/tuit/group.ana/orig_RelUnc

cd /data/brutus_data2/jwaltz/tuit/ana/group.ana/orig_MeanUnc

name=‘[MeanUncert#1_Coef]’

3dMVM -prefix TUIT_MU_DXxEXP_SZvNC.3dMVM
-mask ./killmask85_new-resam+tlrc
-bsVars ‘Dx*Exp’
-jobs 12
-dataTable
Subj Dx Exp InputFile
12352 HV Explore statsMU.12352+tlrc.${name}
12405 HV Explore statsMU.12405+tlrc.${name}
12519 HV Explore statsMU.12519+tlrc.${name}
12541 HV Explore statsMU.12541+tlrc.${name}
12546 HV Explore statsMU.12546+tlrc.${name}
12568 HV Explore statsMU.12568+tlrc.${name}
12677 HV Explore statsMU.12677+tlrc.${name}
12678 HV Explore statsMU.12678+tlrc.${name}
12691 HV Explore statsMU.12691+tlrc.${name}
12730 HV Explore statsMU.12730+tlrc.${name}
12734 HV Explore statsMU.12734+tlrc.${name}
12758 HV Explore statsMU.12758+tlrc.${name}
12770 HV Explore statsMU.12770+tlrc.${name}
13067 HV Explore statsMU.13067+tlrc.${name}
13095 HV Explore statsMU.13095+tlrc.${name}
13152 HV Explore statsMU.13152+tlrc.${name}
13210 HV Explore statsMU.13210+tlrc.${name}
13212 HV Explore statsMU.13212+tlrc.${name}
13253 HV Explore statsMU.13253+tlrc.${name}
12594 HV Nonexp statsMU.12594+tlrc.${name}
12669 HV Nonexp statsMU.12669+tlrc.${name}
12670 HV Nonexp statsMU.12670+tlrc.${name}
12671 HV Nonexp statsMU.12671+tlrc.${name}
12675 HV Nonexp statsMU.12675+tlrc.${name}
12684 HV Nonexp statsMU.12684+tlrc.${name}
12693 HV Nonexp statsMU.12693+tlrc.${name}
12700 HV Nonexp statsMU.12700+tlrc.${name}
12706 HV Nonexp statsMU.12706+tlrc.${name}
12733 HV Nonexp statsMU.12733+tlrc.${name}
12735 HV Nonexp statsMU.12735+tlrc.${name}
12752 HV Nonexp statsMU.12752+tlrc.${name}
13018 HV Nonexp statsMU.13018+tlrc.${name}
13096 HV Nonexp statsMU.13096+tlrc.${name}
13100 HV Nonexp statsMU.13100+tlrc.${name}
13199 HV Nonexp statsMU.13199+tlrc.${name}
12390 SZ Explore statsMU.12390+tlrc.${name}
12427 SZ Explore statsMU.12427+tlrc.${name}
12468 SZ Explore statsMU.12468+tlrc.${name}
12481 SZ Explore statsMU.12481+tlrc.${name}
12505 SZ Explore statsMU.12505+tlrc.${name}
12645 SZ Explore statsMU.12645+tlrc.${name}
12679 SZ Explore statsMU.12679+tlrc.${name}
12787 SZ Explore statsMU.12787+tlrc.${name}
12834 SZ Explore statsMU.12834+tlrc.${name}
12864 SZ Explore statsMU.12864+tlrc.${name}
12882 SZ Explore statsMU.12882+tlrc.${name}
12900 SZ Explore statsMU.12900+tlrc.${name}
12906 SZ Explore statsMU.12906+tlrc.${name}
12942 SZ Explore statsMU.12942+tlrc.${name}
12953 SZ Explore statsMU.12953+tlrc.${name}
13125 SZ Explore statsMU.13125+tlrc.${name}
13134 SZ Explore statsMU.13134+tlrc.${name}
12380 SZ Nonexp statsMU.12380+tlrc.${name}
12425 SZ Nonexp statsMU.12425+tlrc.${name}
12463 SZ Nonexp statsMU.12463+tlrc.${name}
12609 SZ Nonexp statsMU.12609+tlrc.${name}
12614 SZ Nonexp statsMU.12614+tlrc.${name}
12790 SZ Nonexp statsMU.12790+tlrc.${name}
12833 SZ Nonexp statsMU.12833+tlrc.${name}
12863 SZ Nonexp statsMU.12863+tlrc.${name}
12874 SZ Nonexp statsMU.12874+tlrc.${name}
13063 SZ Nonexp statsMU.13063+tlrc.${name}

ptaylor · May 14, 2020, 12:02am

Hi, Jim-

What version of AFNI is there?


afni -ver

My guess is that that is an old one; in the current version, the “print(g_help_string)” line has parentheses…

If you update AFNI on that machine, perhaps, that will help:


@update.afni.binaries -d

(and we can see if that will help the R issue as well, potentially?)

–pt

jwaltz · May 14, 2020, 1:47am

Precompiled binary linux_xorg7_64: May 10 2017 (Version AFNI_17.1.04)

jwaltz · May 14, 2020, 2:00am

Of note, though. I get no errors, if I run the same AFNI package from another node (running Ubuntu, instead of RedHat).

The advantage of using the RedHat node is that I can take advantage of more processors.

ptaylor · May 14, 2020, 11:44am

Hi, Jim-

I think 2017 was back when AFNI was written in cuneiform or Linear B… That is, it is a very old version (just past its 3rd birthday, in fact).

My guess is that your current node has Python 3 on it, and that version of afni_system_check.py (as well as other Python programs in AFNI) might won’t be happy with it; your other node might have Python 2 (though, I don’t know what version of AFNI with Ubuntu you have there). To check local python version, try “python -V”.

Regardless, I think on all your nodes, you should run:


@update.afni.binaries -d

so you have current AFNI; we can’t reverse engineer solutions for extremely old versions of programs. It will just replace whatever package you have (Ubuntu, xorg, etc.) with the most recent version of the same package, so it won’t swap you over to another type. Verify that your node has the most recent version running with:


afni -ver

Then run:


afni_system_check.py -check_all

… and we can get back to discussing the R stuff that you initially asked about.

–pt

jwaltz · May 14, 2020, 2:26pm

Ok, thanks a lot. Will talk to the peeps who manage our cluster. -jw

ptaylor · May 14, 2020, 4:34pm

Cool.

Also, as noted here:
https://afni.nimh.nih.gov/pub/dist/doc/htmldoc/background_install/install_instructs/steps_linux_Fed_RH.html#install-afni-binaries
for CentOS/Redhat 7, we do have a specific set of binaries that you might want to use:


@update.afni.binaries -package linux_centos_7_64 -do_extras

–pt

jwaltz · May 14, 2020, 5:11pm

Binaries updated, but not specifically for CentOS. I still get the same error about not being able to create ‘R_TempDir’.

Here is the output of afni_system_check.py -check_all:

-------------------------------- general ---------------------------------
architecture: 64bit
system: Linux
release: 3.10.0-862.el7.x86_64
version: #1 SMP Fri Apr 20 16:44:24 UTC 2018
distribution: CentOS Linux 7.5.1804 Core
number of CPUs: 72
apparent login shell: bash
shell RC file: .bashrc (exists)

--------------------- AFNI and related program tests ---------------------
which afni : /data/mprc_data1/freeware/afni/afni
afni version : Precompiled binary linux_xorg7_64: May 4 2020
: AFNI_20.1.06 ‘Otho’
AFNI_version.txt : AFNI_20.1.06, linux_xorg7_64, May 04 2020
which python : /data/mprc_data1/software/andaconda/bin/python
python version : 3.7.4
which R : /data/brutus_data0/software/R/R-3.0.0/bin/R
R version : R version 3.0.0 (2013-04-03) – “Masked Marvel”
which tcsh : /usr/bin/tcsh

instances of various programs found in PATH:
afni : 1 (/data/mprc_data1/freeware/afni/afni)
R : 1 (/data/brutus_data0/software/R/R-3.0.0/bin/R)
python : 2
/data/mprc_data1/software/andaconda/bin/python3.7
/usr/bin/python2.7
python2 : 1 (/usr/bin/python2.7)
python3 : 1 (/data/mprc_data1/software/andaconda/bin/python3.7)

testing ability to start various programs…
afni : success
suma : success
3dSkullStrip : success
uber_subject.py : success
3dAllineate : success
3dRSFC : success
SurfMesh : success
3dClustSim : success
3dMVM : success

checking for R packages…
rPkgsInstall -pkgs ALL -check : FAILURE
Fatal error: cannot create ‘R_TempDir’

R RHOME : /data/brutus_data0/software/R/R-3.0.0

checking for $HOME files…
.afnirc : found
.sumarc : found
.afni/help/all_progs.COMP : found

------------------------------ python libs -------------------------------
** failed to load module PyQt4
– PyQt4 is no longer needed for an AFNI bootcamp

++ module loaded: matplotlib.pyplot
module file : /data/mprc_data1/software/andaconda/lib/python3.7/site-packages/matplotlib/pyplot.py

-------------------------------- env vars --------------------------------
PATH = /data/mprc_data1/software/andaconda/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/data/brutus_data1/home/jwaltz//bin:/usr/lib/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/data/brutus_data0/freeware/fsl5/bin:/data/brutus_data0/home/jwaltz/ana/GeneralScripts:/data/mprc_data1/freeware/afni:/data/brutus_data0/software/R/R-3.0.0/bin

PYTHONPATH =
R_LIBS = /data/brutus_data1/home/jwaltz//R

LD_LIBRARY_PATH = :/data/brutus_data0/lib

DYLD_LIBRARY_PATH =
DYLD_FALLBACK_LIBRARY_PATH =

------------------------------ data checks -------------------------------
data dir : missing AFNI_data6
data dir : missing AFNI_demos
data dir : missing suma_demo
data dir : missing afni_handouts
atlas : found TT_N27+tlrc under /data/mprc_data1/freeware/afni

------------------------------ OS specific -------------------------------
which yum : /usr/bin/yum
yum version : 3.4.3

========================= summary, please fix: =========================

login shell ‘bash’, trusting user to translate code examples from ‘tcsh’
have python version 3.7.4, some programs need 2.7.x
missing R packages (see rPkgsInstall)
insufficient data for AFNI bootcamp

rickr · May 14, 2020, 5:17pm

Hi Jim,

Do you happen to be sitting in a directory that you do not own? Or do you now own the R_LIBS directory? To be sure, you could run:

whoami
ls -ld $R_LIBS
cd
afni_system_check.py -check_all

Thanks,

rick

jwaltz · May 14, 2020, 6:01pm

I appear to be the owner of both the folder that I sit in, and the owner that R sits in. I gave myself read/write privileges to both, but still I get the same error.

Here, the output of afni_system_check.py -check_all:
-------------------------------- general ---------------------------------
architecture: 64bit
system: Linux
release: 3.10.0-862.el7.x86_64
version: #1 SMP Fri Apr 20 16:44:24 UTC 2018
distribution: CentOS Linux 7.5.1804 Core
number of CPUs: 72
apparent login shell: bash
shell RC file: .bashrc (exists)

--------------------- AFNI and related program tests ---------------------
which afni : /data/mprc_data1/freeware/afni/afni
afni version : Precompiled binary linux_xorg7_64: May 4 2020
: AFNI_20.1.06 ‘Otho’
AFNI_version.txt : AFNI_20.1.06, linux_xorg7_64, May 04 2020
which python : /data/mprc_data1/software/andaconda/bin/python
python version : 3.7.4
which R : /data/brutus_data0/software/R/R-3.0.0/bin/R
R version : R version 3.0.0 (2013-04-03) – “Masked Marvel”
which tcsh : /usr/bin/tcsh

instances of various programs found in PATH:
afni : 1 (/data/mprc_data1/freeware/afni/afni)
R : 1 (/data/brutus_data0/software/R/R-3.0.0/bin/R)
python : 2
/data/mprc_data1/software/andaconda/bin/python3.7
/usr/bin/python2.7
python2 : 1 (/usr/bin/python2.7)
python3 : 1 (/data/mprc_data1/software/andaconda/bin/python3.7)

testing ability to start various programs…
afni : success
suma : success
3dSkullStrip : success
uber_subject.py : success
3dAllineate : success
3dRSFC : success
SurfMesh : success
3dClustSim : success
3dMVM : success

checking for R packages…
rPkgsInstall -pkgs ALL -check : FAILURE
Fatal error: cannot create ‘R_TempDir’

R RHOME : /data/brutus_data0/software/R/R-3.0.0

checking for $HOME files…
.afnirc : found
.sumarc : found
.afni/help/all_progs.COMP : found

------------------------------ python libs -------------------------------
** failed to load module PyQt4
– PyQt4 is no longer needed for an AFNI bootcamp

++ module loaded: matplotlib.pyplot
module file : /data/mprc_data1/software/andaconda/lib/python3.7/site-packages/matplotlib/pyplot.py

-------------------------------- env vars --------------------------------
PATH = /data/mprc_data1/software/andaconda/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/data/brutus_data1/home/jwaltz//bin:/usr/lib/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/data/brutus_data0/freeware/fsl5/bin:/data/brutus_data0/home/jwaltz/ana/GeneralScripts:/data/mprc_data1/freeware/afni:/data/brutus_data0/software/R/R-3.0.0/bin

PYTHONPATH =
R_LIBS = /data/brutus_data1/home/jwaltz//R

LD_LIBRARY_PATH = :/data/brutus_data0/lib:/data/brutus_data0/lib

DYLD_LIBRARY_PATH =
DYLD_FALLBACK_LIBRARY_PATH =

------------------------------ data checks -------------------------------
data dir : missing AFNI_data6
data dir : missing AFNI_demos
data dir : missing suma_demo
data dir : missing afni_handouts
atlas : found TT_N27+tlrc under /data/mprc_data1/freeware/afni

------------------------------ OS specific -------------------------------
which yum : /usr/bin/yum
yum version : 3.4.3

========================= summary, please fix: =========================

login shell ‘bash’, trusting user to translate code examples from ‘tcsh’
have python version 3.7.4, some programs need 2.7.x
missing R packages (see rPkgsInstall)
insufficient data for AFNI bootcamp

rickr · May 14, 2020, 7:23pm

How about:

df -hT /tmp

You could possibly set your own temp directory, e.g.

export TMPDIR=$HOME/R/tmp
mkdir $TMPDIR

rick

jwaltz · May 15, 2020, 5:54pm

Right. So I figured out that it was ‘/tmp’ that I needed read/write access to, and now I have read/write access to it.

Now I just get a series of errors where AFNI/R are looking for different files/libraries (e.g., the below).

Just seems to be a problem with our (new/old) installation of R.

So far, we have been hunting down those files/libraries, one by one. I don’t know how many there are.

oo Warning:
Failed to load R_io.so with this error message:

Error in dyn.load(ll) :
unable to load shared object ‘/data/mprc_data1/freeware/afni/R_io.so’:
libR.so: cannot open shared object file: No such file or directory
Calls: source … withVisible → eval → eval → set_R_io → dyn.load
Execution halted

‘R_io.so’ definitely exists, and ‘/data/mprc_data1/freeware/afni/’ is definitely on my path, so why would

jwaltz · May 20, 2020, 1:29am

Hi Guys-

I’m a little embarrassed to say that the problem was solved by installing R-4.0.0 and pointing R in the direction of the full set of libraries, with:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/data/brutus_data0/lib

I also had to copy over a newer version of “libstdc++.so.6”.

Now R and 3dMVM work.

Thanks for all your help.

-jw

ptaylor · May 20, 2020, 1:08pm

Hi, Jim-

No reason for sheepishness at all, esp. bouncing around between different computers, OSs and setups.

Glad it is sorted.

–pt

Where does AFNI look for R

TUIT_3dMVM_MU_DXxEXP_SZvNC.sh

cd /data/brutus_data2/jwaltz/tuit/group.ana/orig_RelUnc

So far, we have been hunting down those files/libraries, one by one. I don’t know how many there are.

Error in dyn.load(ll) : unable to load shared object ‘/data/mprc_data1/freeware/afni/R_io.so’: libR.so: cannot open shared object file: No such file or directory Calls: source … withVisible → eval → eval → set_R_io → dyn.load Execution halted

Error in dyn.load(ll) :
unable to load shared object ‘/data/mprc_data1/freeware/afni/R_io.so’:
libR.so: cannot open shared object file: No such file or directory
Calls: source … withVisible → eval → eval → set_R_io → dyn.load
Execution halted