uber_subject.py "cmd.ap.subj: No such file or directory."

nic · November 30, 2017, 11:25pm

Hi AFNI gurus and enthusiasts,

I’m running into a few puzzling, related problems with uber_subject.py.

After filling out the fields in the GUI, I click on the upper left document icon, which sets up the familiar folder structure and creates “cmd.ap.subj” in the subj folder. However, clicking on the right upper document icon produces an error message: “cmd.ap.subj: No such file or directory.”. An error log (“output.cmd.ap.test” stating “cmd.ap.test: No such file or directory.”) is created in the $top_dir (bc it looks for cmd.ap.subj there, instead of the subj folder). The $top_dir is always the shell login directory, i.e., it doesn’t matter from which directory I call the uber_subject.py GUI, it always sets the login directory as $top_dir.

Executing cmd.ap.test in the subj folder (the only file created) generates the expected proc script - in the $top_dir, not in the subj folder as usual. Making some minor changes to the proc script (e.g., adding full paths to input/result directories; same changes I’ve made to other proc scripts on other systems that worked without problems) and executing it results in early termination as stats.REML_cmd looks for X.xmat.1D in the $top_dir, not the subj folder. The proc script cds into the subj folder and executes stats.REML_cmd from there…so I don’t know why/how stats.REML_cmd looks in the $top_dir?! (There are problems with generating some of the review scripts/outputs down the line, too.)

Could this be related to using ~/.cshrc.local and ~/.cshrc pointing to some .cshrc.master (but I can’t see how anything in that file would lead to my problems)? I’m working on a remote server that is shared among many users.

Thank you very much for your help in advance!

Nic

#########
######### uber_subject.py output (cmd.ap.test) #########
#########
#!/usr/bin/env tcsh

created by uber_subject.py: version 0.41 (October 10, 2017)

creation date: Wed Nov 29 19:22:40 2017

set subject and group identifiers

set subj = subj
set gname = bh.9457

set data directories

set top_dir = /data1/nschwarz
set anat_dir = $top_dir/data/subj
set epi_dir = $top_dir/data/subj
set stim_dir = $top_dir/projects/cvr/analyses/bh/glm/scripts

run afni_proc.py to create a single subject processing script

afni_proc.py -subj_id $subj
-script proc.$subj -scr_overwrite
-blocks tshift align tlrc volreg blur mask scale regress
-copy_anat $anat_dir/subj_T1w+orig
-dsets $epi_dir/subj_BOLD_air_breathhold_mag+orig.HEAD
-tcat_remove_first_trs 0
-volreg_align_to MIN_OUTLIER
-volreg_align_e2a
-volreg_tlrc_warp
-blur_size 3.0
-regress_stim_times $stim_dir/stimtimes.txt
-regress_stim_labels
bh
-regress_basis ‘TENT(9,45,7)’
-regress_censor_motion 0.3
-regress_censor_outliers 0.1
-regress_reml_exec
-regress_compute_fitts
-regress_make_ideal_sum sum_ideal.1D
-regress_est_blur_epits
-regress_est_blur_errts

#########
######### System information #########
#########
-------------------------------- general ---------------------------------
architecture: 64bit ELF
system: Linux
release: 2.6.32-358.el6.x86_64
version: #1 SMP Tue Jan 29 11:47:41 EST 2013
distribution: Red Hat Enterprise Linux Server 6.4 Santiago
number of CPUs: 20
apparent login shell: tcsh
shell RC file: .cshrc (exists)

--------------------- AFNI and related program tests ---------------------
which afni : /cm/shared/abin/afni
afni version : Precompiled binary linux_openmp_64: Oct 12 2017
: AFNI_17.3.00
AFNI_version.txt : AFNI_17.3.00, linux_openmp_64, Oct 12 2017
which python : /cm/shared/anaconda/bin/python
python version : 2.7.14
which R : /cm/shared/anaconda/bin/R
R version : R version 3.3.1 (2016-06-21) – “Bug in Your Hair”
which tcsh : /bin/tcsh

instances of various programs found in PATH:
afni : 1 (/cm/shared/abin/afni)
R : 3
/cm/shared/anaconda2/bin/R
/usr/local/bin/R
/cm/shared/anaconda3/bin/R
python : 3
/cm/shared/anaconda2/bin/python2.7
/usr/bin/python
/cm/shared/anaconda3/bin/python3.6
python2 : 2
/cm/shared/anaconda2/bin/python2.7
/usr/bin/python
python3 : 1 (/cm/shared/anaconda3/bin/python3.6)

testing ability to start various programs…
afni : success
suma : success
3dSkullStrip : success
uber_subject.py : success
3dAllineate : success
3dRSFC : success
SurfMesh : success
3dClustSim : success

checking for R packages…
rPkgsInstall -pkgs ALL -check : FAILURE <— Will be updated soon

    oo Warning: 
       These packages are not installed on the computer: afex!
    These packages are not installed on the computer: phia!
    These packages are not installed on the computer: lme4!
    These packages are not installed on the computer: psych!

checking for $HOME files…
.afnirc : found
.sumarc : found
.afni/help/all_progs.COMP : found

------------------------------ python libs -------------------------------
++ module ‘PyQt4’ found at /cm/shared/anaconda/lib/python2.7/site-packages/PyQt4
++ module loaded: PyQt4

-------------------------------- env vars --------------------------------
PATH = /cm/shared/PALM:/cm/shared/micutil:/home/nschwarz/.local/bin:/cm/shared/mypytools:/cm/shared/bxh_xcede_tools-1.11.1-lsb30.x86_64/bin:/cm/shared/freesurfer-6.0.0/bin:/cm/shared/freesurfer-6.0.0/fsfast/bin:/cm/shared/freesurfer-6.0.0/tktools:/cm/shared/fsl-5.0.10/bin:/cm/shared/freesurfer-6.0.0/mni/bin:/cm/shared/fsl-5.0.10/bin:.:/home/nschwarz/bin:/home/nschwarz/bin/fsl:/cm/shared/fix:/cm/shared/ICA-AROMA:/cm/shared/workbench1.2.3/bin_rh_linux64:/cm/shared/ants-2.1.0:/cm/shared/spm12-standalone:/cm/shared/mricrogl:/cm/shared/mricron:/cm/shared/freesurfer/bin:/cm/shared/freesurfer/fsfast/bin:/cm/shared/freesurfer/tktools:/cm/shared/fsl/bin:/cm/shared/freesurfer/mni/bin:.:/cm/shared/anaconda/bin:/cm/shared/apps/sge/current/bin/UNSUPPORTED-lx-amd64:/cm/shared/fsl/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/ibutils/bin:/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/cm/shared/apps:/cm/shared/apps/sge/current/bin/lx-amd64:/cm/shared/apps/sge/current/utilbin/lx-amd64:/cm/shared/apps/MATLAB/R2014a/bin:/cm/shared/apps/SublimeText2:/cm/shared/abin:/cm/shared/anaconda2/bin:/cm/shared/anaconda3/bin:/cm/shared/abin/afni

PYTHONPATH =
R_LIBS =
LD_LIBRARY_PATH =
DYLD_LIBRARY_PATH =
DYLD_FALLBACK_LIBRARY_PATH =

------------------------------ data checks -------------------------------
data dir : missing AFNI_data6
data dir : missing AFNI_demos
data dir : missing suma_demo
data dir : missing afni_handouts
atlas : found TT_N27+tlrc under /cm/shared/abin

------------------------------ OS specific -------------------------------
which yum : /usr/bin/yum
yum version : 3.2.29

========================= summary, please fix: =========================

missing R packages (see rPkgsInstall)
insufficient data for AFNI bootcamp

rickr · December 4, 2017, 7:07pm

Hi Nic,

I have not been able to reproduce this cmd.ap.subj error,
and you are using the current version of uber_subject.py.

What is even more confusing is your comment about running
stats.REML_cmd. There should not be any consideration of
$top_dir for that.

There were some edits made in uber_subject.py around that
time, but you seem to be current. Is there any way that
you can get an update, just to be sure? It would be easy
to get and run your own version, if that would help. I
would be happy to help and resolve this question.

rick

nic · December 4, 2017, 7:36pm

Hi Rick,

Thank you for your answer!

I seem to have found a solution but don’t quite understand why this works. Apparently, I need to source these scripts from within the proc script (i.e., the script generated by uber_subject.py):

source stats.REML_cmd (I also tried to chmod for this script from within the proc (without sourcing it) but this didn’t help)
source ./@ss_review_basic > out.ss_review.$subj.txt (and comment out if ( -e @ss_review_basic ))
but I don’t need to source
@auto_tlrc -base TT_N27+tlrc -init_xform AUTO_CENTER -input ${subj}_T1w_ns+orig -no_ss

I’m not too versed with these things but could it have to do with the different script headers?

proc script: #!/bin/tcsh -xef
@ss_review_basic: #!/bin/tcsh
while
@auto_tlrc: #!/bin/tcsh -f

Thank you very much and have a lovely day,

Nic

nic · December 5, 2017, 3:42am

Hi Rick,

I’ve got another puzzling problem that may shed some light.

I’ve modified the 3dDeconvolve cmd in the original proc script to use a different stimtimes file and try a different response model (saved as a new script and chmod 777 newscript). This modified script copies the new stimtimes6.txt file into the stimuli directory (of a new subject folder) and uses it for the regular 3dDeconvolve. However, stats.REML_cmd looks for the old stimtimes.txt file. I tried both source stats.REML_cmd and ./stats.REML_cmd but neither worked.

Please see the original and modified 3ddeconvolve cmd below along with the terminal error message (and some history). The proc scripts are executed with tcsh -xef proc.csh |& tee out.log.txt.

Could it have something to do with the LD_LIBRARY_PATH? This is unset in my ~/.cshrc_master.

Thanks so much again!

Nic

ORIGINAL PROC SCRIPT:

3dDeconvolve -input pb04.$subj.r*.scale+tlrc.HEAD
-censor censor_${subj}_combined_2.1D
-polort 3
-num_stimts 7
-stim_times 1 stimuli/stimtimes.txt ‘TENT(9,45,7)’
-stim_label 1 bh
-stim_file 2 motion_demean.1D’[0]’ -stim_base 2 -stim_label 2 roll
-stim_file 3 motion_demean.1D’[1]’ -stim_base 3 -stim_label 3 pitch
-stim_file 4 motion_demean.1D’[2]’ -stim_base 4 -stim_label 4 yaw
-stim_file 5 motion_demean.1D’[3]’ -stim_base 5 -stim_label 5 dS
-stim_file 6 motion_demean.1D’[4]’ -stim_base 6 -stim_label 6 dL
-stim_file 7 motion_demean.1D’[5]’ -stim_base 7 -stim_label 7 dP
-iresp 1 iresp_bh.$subj
-fout -tout -bout -vout -x1D X.xmat.1D -xjpeg X.jpg -xsave
-x1D_uncensored X.nocensor.xmat.1D
-errts errts.${subj}
-bucket stats.$subj
…

– execute the 3dREMLfit script, written by 3dDeconvolve –

source stats.REML_cmd

ASSOCIATED stats.REML_cmd

3dDeconvolve -input pb04.sub-01.r01.scale+tlrc.HEAD -censor censor_sub-01_combined_2.1D -polort 3 -num_stimts 7 -stim_times 1 stimuli/stimtimes.txt ‘TENT(9,45,7)’ -stim_label 1 bh -stim_file 2 ‘motion_demean.1D[0]’ -stim_base 2 -stim_label 2 roll -stim_file 3 ‘motion_demean.1D[1]’ -stim_base 3 -stim_label 3 pitch -stim_file 4 ‘motion_demean.1D[2]’ -stim_base 4 -stim_label 4 yaw -stim_file 5 ‘motion_demean.1D[3]’ -stim_base 5 -stim_label 5 dS -stim_file 6 ‘motion_demean.1D[4]’ -stim_base 6 -stim_label 6 dL -stim_file 7 ‘motion_demean.1D[5]’ -stim_base 7 -stim_label 7 dP -iresp 1 iresp_bh.sub-01 -fout -tout -bout -vout -x1D X.xmat.1D -xjpeg X.jpg -xsave -x1D_uncensored X.nocensor.xmat.1D -errts errts.sub-01 -bucket stats.sub-01

3dREMLfit -matrix X.xmat.1D -input pb04.sub-01.r01.scale+tlrc.HEAD
-fout -tout -Rbuck stats.sub-01_REML -Rvar stats.sub-01_REMLvar
-Rerrts errts.sub-01_REML -verb $*

#########

MODIFIED PROC SCRIPT:

3dDeconvolve -input pb04.$subj.r*.scale+tlrc.HEAD
-censor censor_${subj}_combined_2.1D
-polort 3
-num_stimts 7
-stim_times 1 stimuli/stimtimes6.txt ‘BLOCK(15,1)’
-stim_label 1 bh
-stim_file 2 motion_demean.1D’[0]’ -stim_base 2 -stim_label 2 roll
-stim_file 3 motion_demean.1D’[1]’ -stim_base 3 -stim_label 3 pitch
-stim_file 4 motion_demean.1D’[2]’ -stim_base 4 -stim_label 4 yaw
-stim_file 5 motion_demean.1D’[3]’ -stim_base 5 -stim_label 5 dS
-stim_file 6 motion_demean.1D’[4]’ -stim_base 6 -stim_label 6 dL
-stim_file 7 motion_demean.1D’[5]’ -stim_base 7 -stim_label 7 dP
-iresp 1 iresp_bh.$subj
-fout -tout -bout -vout -x1D X.xmat.1D -xjpeg X.jpg -xsave
-x1D_uncensored X.nocensor.xmat.1D
-errts errts.${subj}
-bucket stats.$subj
…

– execute the 3dREMLfit script, written by 3dDeconvolve –

#source stats.REML_cmd
./stats.REML_cmd

ASSOCIATED stats.REML_cmd (which was sourced first but led to termination error and then tried running from current directory with same error)

3dDeconvolve -input pb04.sub-01.r01.scale+tlrc.HEAD -censor censor_sub-01_combined_2.1D -polort 3 -num_stimts 7 -stim_times 1 stimuli/stimtimes6.txt ‘BLOCK(15,1)’ -stim_label 1 bh -stim_file 2 ‘motion_demean.1D[0]’ -stim_base 2 -stim_label 2 roll -stim_file 3 ‘motion_demean.1D[1]’ -stim_base 3 -stim_label 3 pitch -stim_file 4 ‘motion_demean.1D[2]’ -stim_base 4 -stim_label 4 yaw -stim_file 5 ‘motion_demean.1D[3]’ -stim_base 5 -stim_label 5 dS -stim_file 6 ‘motion_demean.1D[4]’ -stim_base 6 -stim_label 6 dL -stim_file 7 ‘motion_demean.1D[5]’ -stim_base 7 -stim_label 7 dP -iresp 1 iresp_bh.sub-01 -fout -tout -bout -vout -x1D X.xmat.1D -xjpeg X.jpg -xsave -x1D_uncensored X.nocensor.xmat.1D -errts errts.sub-01 -bucket stats.sub-01

3dREMLfit -matrix X.xmat.1D -input pb04.sub-01.r01.scale+tlrc.HEAD
-fout -tout -Rbuck stats.sub-01_REML -Rvar stats.sub-01_REMLvar
-Rerrts errts.sub-01_REML -verb $*

###########################

ERROR MESSAGE WITH A BIT OF HISTORY:

3dDeconvolve -input pb04.sub-01.r01.scale+tlrc.HEAD -censor censor_sub-01_combined_2.1D -polort 3 -num_stimts 7 -stim_times 1 stimuli/stimtimes6.txt BLOCK(15,1) -stim_label 1 bh -stim_file 2 motion_demean.1D[0] -stim_base 2 -stim_label 2 roll -stim_file 3 motion_demean.1D[1] -stim_base 3 -stim_label 3 pitch -stim_file 4 motion_demean.1D[2] -stim_base 4 -stim_label 4 yaw -stim_file 5 motion_demean.1D[3] -stim_base 5 -stim_label 5 dS -stim_file 6 motion_demean.1D[4] -stim_base 6 -stim_label 6 dL -stim_file 7 motion_demean.1D[5] -stim_base 7 -stim_label 7 dP -iresp 1 iresp_bh.sub-01 -fout -tout -bout -vout -x1D X.xmat.1D -xjpeg X.jpg -xsave -x1D_uncensored X.nocensor.xmat.1D -errts errts.sub-01 -bucket stats.sub-01
++ 3dDeconvolve: AFNI version=AFNI_17.3.00 (Oct 12 2017) [64-bit]
++ Authored by: B. Douglas Ward, et al.
++ current memory malloc-ated = 242,879 bytes (about 243 thousand [kilo])
++ loading dataset pb04.sub-01.r01.scale+tlrc.HEAD
++ current memory malloc-ated = 547,983,083 bytes (about 548 million [mega])
++ STAT automask has 516409 voxels (out of 570000 = 90.6%)
++ Skipping check for initial transients
++ Input polort=3; Longest run=360.0 s; Recommended minimum polort=3 ++ OK ++
++ -stim_times using TR=0.75 s for stimulus timing conversion
++ -stim_times using TR=0.75 s for any -iresp output datasets
++ [you can alter the -iresp TR via the -TR_times option]
++ ** -stim_times NOTE ** guessing GLOBAL times if 1 time per line; LOCAL otherwise
++ ** GUESSED ** -stim_times 1 using LOCAL times
++ Number of time points: 480 (before censor) ; 472 (after)

Number of parameters: 11 [10 baseline ; 1 signal]
++ Memory required for output bricks = 1,165,080,000 bytes (about 1.2 billion [giga])
++ Wrote matrix image to file X.jpg
++ Wrote matrix values to file X.xmat.1D
++ ========= Things you can do with the matrix file =========
++ (a) Linear regression with ARMA(1,1) modeling of serial correlation:

3dREMLfit -matrix X.xmat.1D -input pb04.sub-01.r01.scale+tlrc.HEAD
-fout -tout -Rbuck stats.sub-01_REML -Rvar stats.sub-01_REMLvar
-Rerrts errts.sub-01_REML -verb

++ N.B.: 3dREMLfit command above written to file stats.REML_cmd
++ (b) Visualization/analysis of the matrix via ExamineXmat.R
++ (c) Synthesis of sub-model datasets using 3dSynthesize
++ ==========================================================
++ Wrote matrix values to file X.nocensor.xmat.1D
++ ----- Signal+Baseline matrix condition [X] (472x11): 3.25564 ++ VERY GOOD ++
++ ----- Signal-only matrix condition [X] (472x1): 1 ++ VERY GOOD ++
++ ----- Baseline-only matrix condition [X] (472x10): 3.24366 ++ VERY GOOD ++
++ ----- stim_base-only matrix condition [X] (472x6): 2.91534 ++ VERY GOOD ++
++ ----- polort-only matrix condition [X] (472x4): 1.01643 ++ VERY GOOD ++
++ +++++ Matrix inverse average error = 2.82774e-15 ++ VERY GOOD ++
++ Matrix setup time = 1.23 s
++ current memory malloc-ated = 1,714,499,644 bytes (about 1.7 billion [giga])
++ Calculations starting; elapsed time=6.400
++ voxel loop:0123456789.0123456789.0123456789.0123456789.0123456789.
++ Calculations finished; elapsed time=73.502
++ Smallest FDR q [1 Full_Fstat] = 1.25893e-13
++ Smallest FDR q [3 Run#1Pol#0_Tstat] = 9.99997e-16
++ Smallest FDR q [5 Run#1Pol#1_Tstat] = 7.03598e-14
++ Smallest FDR q [7 Run#1Pol#2_Tstat] = 1.40764e-12
++ Smallest FDR q [9 Run#1Pol#3_Tstat] = 8.68626e-13
++ Smallest FDR q [11 bh#0_Tstat] = 1.25892e-13
++ Smallest FDR q [12 bh_Fstat] = 1.25893e-13
++ Smallest FDR q [14 roll#0_Tstat] = 2.24815e-13
++ Smallest FDR q [15 roll_Fstat] = 2.24812e-13
++ Smallest FDR q [17 pitch#0_Tstat] = 3.75758e-13
++ Smallest FDR q [18 pitch_Fstat] = 3.75754e-13
++ Smallest FDR q [20 yaw#0_Tstat] = 2.6817e-12
++ Smallest FDR q [21 yaw_Fstat] = 2.6817e-12
++ Smallest FDR q [23 dS#0_Tstat] = 3.75509e-13
++ Smallest FDR q [24 dS_Fstat] = 3.75505e-13
++ Smallest FDR q [26 dL#0_Tstat] = 1.21213e-12
++ Smallest FDR q [27 dL_Fstat] = 1.21213e-12
++ Smallest FDR q [29 dP#0_Tstat] = 2.27109e-13
++ Smallest FDR q [30 dP_Fstat] = 2.27113e-13
++ Wrote bucket dataset into ./stats.sub-01+tlrc.BRIK

created 19 FDR curves in bucket header
++ Wrote iresp 3D+time dataset into ./iresp_bh.sub-01+tlrc.BRIK
++ Wrote 3D+time dataset into ./errts.sub-01+tlrc.BRIK
++ #Flops=1.03043e+11 Average Dot Product=19.7981
if ( 0 != 0 ) then
1d_tool.py -show_cormat_warnings -infile X.xmat.1D
tee out.cormat_warn.txt
– no warnings for correlation matrix (cut = 0.400) –
timing_tool.py -multi_timing stimuli/stimtimes.txt -tr 0.75 -warn_tr_stats
tee out.TENT_warn.txt
** failed to open 1D file ‘stimuli/stimtimes.txt’
** A1D: failed to read data file stimuli/stimtimes.txt
** (multi) failed to read timing from ‘stimuli/stimtimes.txt’

rickr · December 5, 2017, 2:19pm

Hi Nic,

To be sure, are you modifying the proc script that
was generated by afni_proc.py? That is not a good
idea. Those are long scripts, and it is difficult
to reconcile changes in one part of the script with
what happens elsewhere in the script.

To make an edit, change the afni_proc.py command,
run it to generate a new proc script, and then use
that to perform the analysis.

In this case, it looks like 3dDeconvolve is running
fine, but a subsequent 1d_tool.py command is failing,
because it does not know you have changed the timing
files.

Please try to make a modified afni_proc.py command,
and see if that works better.

rick