afni_proc.py: ERROR: mri_read_double_ascii: couldn't open file

Robin · February 18, 2020, 1:19pm

Hi, Robin-

I am going to split this into a new thread, as it is a separate question from here (where it was originally posted):
https://afni.nimh.nih.gov/afni/community/board/read.php?1,163213,163218#msg-163218

I have also edited the subject line to reflect the first error shown in the output.

Hope that is all reasonable.

(Heading into work now, will reply more anon, or perhaps someone else will chime in first.)

-pt

Sorry but it seems we have a new error post update…
This is the output of afni_system_check -check_all


afni_system_check.py -check_all
-------------------------------- general ---------------------------------
architecture:         64bit 
system:               Linux
release:              4.15.0-51-generic
version:              #55-Ubuntu SMP Wed May 15 14:27:21 UTC 2019
distribution:         Ubuntu 18.04 bionic
number of CPUs:       40
apparent login shell: bash
shell RC file:        .bashrc (exists)

--------------------- AFNI and related program tests ---------------------
which afni           : /usr/local/abin/afni
afni version         : Precompiled binary linux_openmp_64: Feb 14 2020 
                     : AFNI_20.0.09 'Galba'
AFNI_version.txt     : AFNI_20.0.09, linux_openmp_64, Feb 14 2020
which python         : /usr/bin/python
python version       : 2.7.15+
which R              : /usr/bin/R
R version            : R version 3.6.0 (2019-04-26) -- "Planting of a Tree"
which tcsh           : /usr/bin/tcsh

instances of various programs found in PATH:
    afni    : 1   (/usr/local/abin/afni)
    R       : 1   (/usr/bin/R)
    python  : 1   (/usr/bin/python2.7)
    python2 : 1   (/usr/bin/python2.7)
    python3 : 1   (/usr/bin/python3.6)


testing ability to start various programs...
    afni                 : success
    suma                 : success
    3dSkullStrip         : success
    uber_subject.py      : success
    3dAllineate          : success
    3dRSFC               : success
    SurfMesh             : success
    3dClustSim           : success
    3dMVM                : success

checking for R packages...
    rPkgsInstall -pkgs ALL -check : success

R RHOME : /usr/lib/R

checking for $HOME files...
    .afnirc                   : missing
    .sumarc                   : missing
    .afni/help/all_progs.COMP : found

------------------------------ python libs -------------------------------
++ module loaded: PyQt4
   module file : /usr/lib/python2.7/dist-packages/PyQt4/__init__.pyc

** failed to load module matplotlib.pyplot
-- matplotlib.pyplot is not required, but is desirable

-------------------------------- env vars --------------------------------
PATH = /usr/local/fsl/bin:/home/robka/.local/bin:/usr/local/freesurfer/bin:/usr/local/freesurfer/fsfast/bin:/usr/local/freesurfer/tktools:/usr/local/fsl/bin:/usr/local/freesurfer/mni/bin:/usr/local/fsl/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/abin:/usr/local/MATLAB/R2015b/bin:/usr/local/mricron_lx:/usr/local/xnat_img_download:/usr/local/stim_file_gen:/usr/local/mricron_old:/usr/local/xnat_resource_downloader:/opt/thinlinc/bin:/usr/local/abin:/usr/local/abin

PYTHONPATH = /usr/local/pylib_shared

R_LIBS = /usr/lib/R

LD_LIBRARY_PATH = /usr/lib/x86_64-linux-gnu/

DYLD_LIBRARY_PATH = 
DYLD_FALLBACK_LIBRARY_PATH = 

------------------------------ data checks -------------------------------
data dir : missing AFNI_data6
data dir : missing AFNI_demos
data dir : missing suma_demo
data dir : missing afni_handouts
atlas    : found TT_N27+tlrc  under /usr/local/abin

------------------------------ OS specific -------------------------------
which apt-get        : /usr/bin/apt-get
apt-get version      : apt 1.6.11 (amd64)

have Ubuntu system: Ubuntu 18.04 bionic

=========================  summary, please fix:  =========================
*  login shell 'bash', trusting user to translate code examples from 'tcsh'
*  please run: cp /usr/local/abin/AFNI.afnirc ~/.afnirc
*  please run: "suma -update_env" for .sumarc
*  insufficient data for AFNI bootcamp

When running 2 diferent pre-procs 1) Old one on old data that worked before and 2) A new analysis with updated pre-proc and new data we get FATAL ERRORs. This is on both of our Ubuntu 16 and Ubuntu18.

Pre-proc1)


-blocks despike tshift align tlrc volreg mask scale regress \
    -mask_segment_anat yes \
    -mask_segment_erode yes \
    -tshift_opts_ts -tpattern seq+z \
    -align_opts_aea -ginormous_move -deoblique on -cost lpc+zz \
    -tcat_remove_first_trs 0 \
    -tlrc_base MNI_avg152T1+tlrc \
    -volreg_tlrc_warp \
    -volreg_align_to MIN_OUTLIER \
    -volreg_align_e2a \
    -regress_censor_outliers 0.1 \
    -regress_apply_mot_types demean deriv \
    -regress_stim_types AM1 \
    -regress_basis 'dmBLOCK' \
    -regress_stim_times $stim/* \

Output error of pre-proc 1:


3dSkullStrip -orig_vol -input ./__tt_T1+orig -prefix ./__tt_T1_ns
*+ WARNING:   If you are performing spatial transformations on an oblique dset, 
  such as ./__tt_T1+orig.BRIK,
  or viewing/combining it with volumes of differing obliquity,
  you should consider running: 
     3dWarp -deoblique 
  on this and  other oblique datasets in the same session.
 See 3dWarp -help for details.
++ Oblique dataset:./__tt_T1+orig.BRIK is 4.693148 degrees from plumb.
#Script is running (command trimmed):
  3dinfo ./__tt_T1_ns+orig | \grep 'Data Axes Tilt:'|\grep 'Oblique'
#++ Dataset /fenix/users/irepe/scn9a/M_sub01_run1.results/__tt_T1_ns+orig is not oblique
#Script is running (command trimmed):
  3dinfo ./vr_base_min_outlier+orig | \grep 'Data Axes Tilt:'|\grep 'Oblique'
#++ Dataset /fenix/users/irepe/scn9a/M_sub01_run1.results/vr_base_min_outlier+orig is ***oblique****
#Script is running:
  3dAttribute DELTA /fenix/users/irepe/scn9a/M_sub01_run1.results/__tt_T1_ns+orig
#++ Spacing for anat to oblique epi alignment is 0.868056
#Script is running (command trimmed):
  3dcopy ./__tt_T1_ns+orig ./__tt_T1_ns_ob; @Align_Centers -base ./vr_base_min_outlier+orig -dset ./__tt_T1_ns_ob+orig -no_cp
++ 3dcopy: AFNI version=AFNI_20.0.09 (Feb 14 2020) [64-bit]
*+ WARNING:   If you are performing spatial transformations on an oblique dset, 
  such as ./vr_base_min_outlier+orig.BRIK,
  or viewing/combining it with volumes of differing obliquity,
  you should consider running: 
     3dWarp -deoblique 
  on this and  other oblique datasets in the same session.
 See 3dWarp -help for details.
++ Oblique dataset:./vr_base_min_outlier+orig.BRIK is 8.172770 degrees from plumb.
++ 3drefit: AFNI version=AFNI_20.0.09 (Feb 14 2020) [64-bit]
++ Authored by: RW Cox
++ Processing AFNI dataset ./__tt_T1_ns_ob+orig
 + deoblique
++ 3drefit processed 1 datasets
else: endif not found.
#Script is running (command trimmed):
  cat_matvec ./__tt_T1_ns_ob_shft.1D -I > ./__tt_T1_ns_shft_I.1D
** ERROR: mri_read_double_ascii: couldn't open file ./__tt_T1_ns_ob_shft.1D
** ERROR: THD_read_dvecmat: can't read matrix+vector from './__tt_T1_ns_ob_shft.1D'
** FATAL ERROR: Can't read matrix from './__tt_T1_ns_ob_shft.1D'

Pre-proc2)


-blocks despike tshift align tlrc volreg mask regress \
    -align_opts_aea \
    -giant_move \
    -tcat_remove_first_trs 0 \
    -volreg_align_to MIN_OUTLIER \
    -tlrc_NL_warp \
    -volreg_tlrc_warp \
    -regress_censor_motion 0.3 \
    -regress_censor_outliers 0.1 \
    -regress_apply_mot_types demean deriv \
    -regress_stim_types AM1 \
    -regress_basis 'dmBLOCK' \
    -regress_stim_times $stim/* \

Output error pre-proc 2:


Performing center alignment with @Align_Centers

++ 3dcopy: AFNI version=AFNI_20.0.09 (Feb 14 2020) [64-bit]
++ 3drefit: AFNI version=AFNI_20.0.09 (Feb 14 2020) [64-bit]
++ Authored by: RW Cox
++ Processing AFNI dataset ./anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft+orig
 + deoblique
++ 3drefit processed 1 datasets
else: endif not found.
ls: cannot access 'anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft.1D': No such file or directory
** ERROR: mri_matrix_evalrpn('&read4x4Xform() &write(stdout:) ') at '&read4x4Xform()': can't read file ''
mv: missing destination file operand after 'anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_44.1D'
Try 'mv --help' for more information.
++ 3drename: AFNI version=AFNI_20.0.09 (Feb 14 2020) [64-bit]
++ THD_rename_dataset_files: rename anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft+orig.HEAD -> __ats_tmp__anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft+orig.HEAD
++ THD_rename_dataset_files: rename anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft+orig.BRIK -> __ats_tmp__anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft+orig.BRIK
** THD_rename_dataset_files: old header anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft+acpc.HEAD doesn't exist!
** THD_rename_dataset_files: old header anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft+tlrc.HEAD doesn't exist!
if: Missing file name.
#**ERROR Failed in affine step
** ERROR - script failed

With the afni_system_check output beeing good(?) I’m a bit confused?

rickr · February 18, 2020, 8:11pm

Hi Robin,

I think the problem in @Align_Centers should be fixed now. Can you wait for a new build this evening, or would it be nice to get the fix sooner?

Thanks,

rick

Robin · February 19, 2020, 6:49am

Since I’m in Sweden my day was almost over when you replied =).

Just let me know when I can download the new binaries! Did you manage to reproduce the error?

Thanks a bunch for all this amazing support.

Robin · February 19, 2020, 8:38am

We tested the new binaries this morning and running the script I attached worked, we can complete the pre-proc and the output stats looks reasonable! Thank you for a fast fix.

However, there is still a FATAL ERROR that without “GOFORIT 6” probably would have stopped the pre-proc.

The error is still under @Align_Centers:


Performing center alignment with @Align_Centers

++ 3dcopy: AFNI version=AFNI_20.0.10 (Feb 18 2020) [64-bit]
++ 3drefit: AFNI version=AFNI_20.0.10 (Feb 18 2020) [64-bit]
++ Authored by: RW Cox
++ Processing AFNI dataset ./anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft+orig
 + deoblique
++ 3drefit processed 1 datasets
++ 3drename: AFNI version=AFNI_20.0.10 (Feb 18 2020) [64-bit]
++ THD_rename_dataset_files: rename anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft+orig.HEAD -> __ats_tmp__anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft+orig.HEAD
++ THD_rename_dataset_files: rename anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft+orig.BRIK -> __ats_tmp__anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft+orig.BRIK
** THD_rename_dataset_files: old header anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft+acpc.HEAD doesn't exist!
** THD_rename_dataset_files: old header anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft+tlrc.HEAD doesn't exist!
++ 3dWarp: AFNI version=AFNI_20.0.10 (Feb 18 2020) [64-bit]
++ Authored by: RW Cox
++ 3dcopy: AFNI version=AFNI_20.0.10 (Feb 18 2020) [64-bit]
** FATAL ERROR: Output dataset ./pre.anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA+orig.HEAD already exists! - EXIT
** Program compile date = Feb 18 2020

First It complains about
** THD_rename_dataset_files: old header anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft+acpc.HEAD doesn’t exist!
** THD_rename_dataset_files: old header anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft+tlrc.HEAD doesn’t exist!

Followed by a fatal error regarding that the dataset pre.anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA+orig.HEAD already exist (I think when it tried to run 3dcopy). To me its seems like this dataset was created at some point in the Align process and then it tried to save a new file with the same name. But since the file exist no other function will fail and we can complete the analysis. I think only you who know the back end code can figure out this one =).

rickr · February 19, 2020, 2:52pm

Hi Robin,

Okay, that’s good news and not-so-good news… I did reproduce that ‘missing endif’ error, but not this ‘already exists’ one.

To be sure, are you running the proc from a clean directory, where the ‘results’ tree does not yet exist?

Would you be able to send me email with the proc script and the output.proc.* files attached? Your afni_proc.py command is working for me on the class data, so more detail would be helpful. I think you can click on my name to get the email address.

Sorry this is being troublesome!

rick

P.S. If you do send email and want to address the GOFORIT question, please feel free to include a tgz package of the stimuli directory.

EDIT: Okay, hold the phone. Going back to the previous thread, I see that you have already uploaded data and output. Going through that now, thanks! …

rickr · February 19, 2020, 3:12pm

Okay the uploaded pre_proc.sh ran well (though I removed the tcsh syntax execution command (using |&), and added the afni_proc.py option, -execute, to remove bash vs. tcsh concerns).

Note that this ran completely through, including both registration and linear regression. So there is just the one question of whether you ran it when there was no ‘results’ directory already present. Is that the case?

Thanks,

rick

Robin · February 19, 2020, 4:06pm

Thanks Rick!

Yes, this was our first thought too so we made a clean directory. Just to confirm I re-ran it with 2 new folders called “affine” and “NL” where one has the -tlrc_NL_warp setting and the other don’t.

My current AFNI-version:


afni -version
Precompiled binary linux_ubuntu_16_64: Feb 18 2020 (Version AFNI_20.0.10 'Galba')

My ubuntu-version


lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.2 LTS
Release:	18.04
Codename:	bionic

Both these clean new directories, using the script I sent you with and without tlrc_NL_warp runt to completion. But I realized that only the NL script gave me the error message:


#++ Aligning /home/robka/upload/NL/test_NL.results/awpy/base.nii data to /home/robka/upload/NL/test_NL.results/awpy/anat.un.nii data
#Script is running (command trimmed):
  @auto_tlrc -base ./base.nii -input ./anat.un.nii -suffix .aff -no_ss -no_pre -init_xform CENTER 
Copying NIFTI volume to ./anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA
++ 3dcopy: AFNI version=AFNI_20.0.10 (Feb 18 2020) [64-bit]

Performing center alignment with @Align_Centers

++ 3dcopy: AFNI version=AFNI_20.0.10 (Feb 18 2020) [64-bit]
++ 3drefit: AFNI version=AFNI_20.0.10 (Feb 18 2020) [64-bit]
++ Authored by: RW Cox
++ Processing AFNI dataset ./anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft+orig
 + deoblique
++ 3drefit processed 1 datasets
++ 3drename: AFNI version=AFNI_20.0.10 (Feb 18 2020) [64-bit]
++ THD_rename_dataset_files: rename anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft+orig.HEAD -> __ats_tmp__anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft+orig.HEAD
++ THD_rename_dataset_files: rename anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft+orig.BRIK -> __ats_tmp__anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft+orig.BRIK
** THD_rename_dataset_files: old header anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft+acpc.HEAD doesn't exist!
** THD_rename_dataset_files: old header anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA_shft+tlrc.HEAD doesn't exist!
++ 3dWarp: AFNI version=AFNI_20.0.10 (Feb 18 2020) [64-bit]
++ Authored by: RW Cox
++ 3dcopy: AFNI version=AFNI_20.0.10 (Feb 18 2020) [64-bit]
[7m** FATAL ERROR:[0m Output dataset ./pre.anat.un_XYZ_OYmeqfZ3JqF7n9LNC7gDAA+orig.HEAD already exists! - EXIT
** Program compile date = Feb 18 2020
Center distance of 0.000032 mm
Padding ...

The script I sent you was with no -tlrc_NL_warp option so that means if you get it wihen you add the -tlrc_NL_warp option we are on the same page?
Also I did find the anat_final (warped dataset) to look a little more grainy than usual but perhaps that’s a concidense or due to me remote login.

Robin · February 19, 2020, 4:21pm

I looked into the T1-image quality thing too.

I personally think (look at cerebellum) that the T1_al_junk (left) looks better than the anat_final.test image (right). I want to remember those warped T1s (anat_final and anat_final_w_skull) looking more clear. We do use -volreg_align_e2a .

Doesn’t it look kind of low-res? Input T1 is > 1mm reslution isotropic and and anat_final.test is 2x2x2 mm. In our other analysis pipelines (very similar scripts) anat_final usually is of 1x1x1 resolution. Perhaps this is unrelated but better to mention it. Attching an older anat_final too, the original T1 there had 1x1x1 resolution and anat-final had 1x1x1 as well.

rickr · February 19, 2020, 4:44pm

Hi Robin,

Okay, I will test with the NL_warp option.

Regarding the anat resolution, it goes down to 2 mm^3 because you are using a 2 mm template, MNI_avg152T1+tlrc.

Importantly, that is a low-res blurry template, and as such, we would never recommend it for non-linear registration (or even affine, for that matter). NL-reg should use a fairly high-quality template. The current practice is to actually run @SSwarper (to align to the MNI152_2009_template_SSW.nii.gz template) for skull-stripping and standard space warping before running afni_proc.py. Then the output of that can be give to afni_proc.py via -tlrc_NL_warped_dsets. See example 6b:

afni_proc.py -show_example 'example 6b'

But anyway, I will run this with -tlrc_NL_warp and let you know how it goes.

rick

Robin · February 19, 2020, 5:06pm

Thanks Rick! You beat me to it. Found that the script used the MNIavg template.

We usually use the MNI152_2009_template.nii.gz as the basis. But this new approch looks interesting. I guess we should implement that. Thanks!

rickr · February 19, 2020, 5:59pm

Hi Robin,

Okay, I hunted down the “FATAL ERROR: Output dataset … already exists!” message. It turns out that that is not actually an error condition. That scenario needs to be trapped, so that 3dcopy does not run and make an error. We will look into it.

But it seems like your process is working!

rick

Robin · February 19, 2020, 6:01pm

Awesome! Thanks!

You guys were, as always, super helpful and quick!

rickr · February 19, 2020, 6:10pm

Okay, that inappropriate FATAL ERROR should not appear anymore.

Sorry for the confusion!

rick