Error when running 3dMSS: "The content under -dataTable is not rectangular!"

AFNI version info (afni -ver):
AFNI_21.1.07 'Domitian'

First I want to thank you all for creating this software and being so active on this forum. I really appreciate it.

Here's my issue: when I run the below script 3dMSS_test.sh (inspired by example 5 from the 3dMSS documentation:

3dMSS -prefix output -jobs 16 \
	-lme 'pain+stimulus+s(TENT,k=9)+s(TENT,k=9,by=pain)' \
	-ranEff 'list(subject=~1)' \
	-qVars 'pain,stimulus,TENT' \
	-prediction @HRF.table \
	-dataTable @extended_df_final.txt

I get the following error:

Read 20 items 
Read 7567 items 
Error in if (len%%wd != 0) errex.AFNI(paste("The content under -dataTable is not rectangular !", : 
argument is of length zero"
Calls: process.MSS.opts
Execution halted

I tried following the advice from this thread (appending \ to each line and creating different subject ids for different conditions), and this thread (using file_tool -test -infile … to check for bad formatting → returns "script has 0 bad characters")


Here is a snippet from extended_df_final.txt (the actual .txt file contains a consistent use of \t, whereas here it is formatted differently for readability):

path			subject			pain		stimulus		TENT
/path/to/brick[0]	sub-1432	chronicbackpain	taste	0
/path/to/brick[1]	sub-1432	chronicbackpain	taste	1
/path/to/brick[2]	sub-1432	chronicbackpain	taste	2
...
/path/to/brick[9]	sub-1432	chronicbackpain	tasteless	0
/path/to/brick[10]	sub-1432	chronicbackpain	tasteless	1
/path/to/brick[11]	sub-1432	chronicbackpain	tasteless	2
...
/path/to/brick[0]	sub-1234 healthy taste 0
/path/to/brick[1]	sub-1234 healthy taste 1
/path/to/brick[2]	sub-1234 healthy taste 2
...

Repeating with TR 0-8 for each subject and each condition taste and tasteless. brick is the bucket output of 3dDeconvolve which contains beta maps for each TENT function for both taste and tasteless conditions (I can provide that script as well if needed). The output from 3dDeconvolve makes sense (or so I think)--after using 3dbucket to aggregate subbricks, I'm able to look at the HRF for both taste and tasteless conditions for each voxel in the brain.

I'm not sure what to do to resolve the "rectangular" error. I've written code to ensure that all of the rows are filled without success (the file seems rectangular).

Any help would be greatly appreciated!

I should also clarify my design: subjects either have chronicbackpain or are healthy. In the MRI, they receive a stimulus that has taste or a control tasteless stimulus. Our TR is 1.

I read the documentation more thoroughly, which said that the last column in dataTable needs to be InputFile which contains the paths to the beta maps. I also binarized pain and stimulus to zeros and ones. Those changes got rid of the "rectangular" error from before.

However, now there is a new error:
image

I looked through all previous posts on the AFNI discussion board but none of the solutions worked for me.

First, I recommend updating your AFNI to the latest version, as your current version is about three years old. I assume that 'pain' is a between-subjects factor and 'stimulus' is a within-subjects factor. Could you clarify what specific effects you intend to analyze using 3dMSS?

Gang Chen

Hi Gang,

Thank you for your reply!

Thank you for pointing out that my version of AFNI is outdated--I will request to update it. It is on a shared server so I cannot update it myself.

Yes, pain is between-subjects and stimulus is within-subject. We are trying to compare the HRF between chronic pain and healthy subjects when they are tasting a sweet drink (vs a tasteless control stimulus). We're trying to do something similar to Fig 5 in your 2023 paper, where you compare the HRF of BP patients to that of healthy controls during the selective attention task.

I recommend conducting three separate analyses:

  1. Taste Condition Analysis: Focus solely on the taste condition. Encode the pain factor with -0.5 for one group and 0.5 for the other. Use the following model specification:

    -lme 's(TENT,k=9)+s(TENT,k=9,by=pain)' \
    -ranEff 'list(subject=~1)' \
    -qVars 'pain,TENT' \
    -prediction @HRF.taste.table \
    

    This will allow you to estimate the hemodynamic response (HDR) for the taste condition.

  2. Tasteless Condition Analysis: Repeat the same steps for the tasteless condition. This will provide you with the estimated HDR for the tasteless condition.

    -lme 's(TENT,k=9)+s(TENT,k=9,by=pain)' \
    -ranEff 'list(subject=~1)' \
    -qVars 'pain,TENT' \
    -prediction @HRF.tasteless.table \
    
  3. Contrast Analysis: Analyze the contrast between the taste and tasteless conditions by obtaining the contrast for each of the nine time points. Then, follow the same procedure as in step 1 using the following model:

    -lme 's(TENT,k=9)+s(TENT,k=9,by=pain)' \
    -ranEff 'list(subject=~1)' \
    -qVars 'pain,TENT' \
    -prediction @HRF.contrast.table \
    

    This step will help assess the statistical evidence for the contrast between the taste and tasteless conditions.

Let me know if this is clear.

Gang Chen

1 Like

Crystal clear. Thanks again, Gang!

Hi Gang,

One more question: I successfully ran the first two analyses comparing the HRF b/w pain patients and healthy controls for under taste and tasteless conditions separately but I am not sure how to run the contrast.

Specifically, I'm not sure whether compute the contrast at the group or individual level. I assumed that group level would be better, so I computed 9*2 new BRIKs of the form contrast_tn_pain(less).orig for each TR (0-8). These were created by using 3dcalc to subtract the output of 3dMSS_taste and the output of 3dMSS_tasteless after indexing by the sub-bricks referring to each TR. I'm not sure where to place these contrasts in the model (or even if this is the right thing to be doing).

I ran 3dMSS for taste and tasteless by subsetting -dataFrame to include only taste or tasteless trials, respectively, and then removed the stimulus column. My HRF.table looked the same in both cases, with columns for label (g1t1, g1t2, ... g1t8, g2t1, ...), pain (0.5, 0.5, ... -0.5, -0.5), and TENT (0, 1, 2, ... 8, 0, 1, ... 8).

Since the bash script is the same for the contrast analysis (i.e., no changes to the lme model or qVars), I don't know where to "tell" the program about the contrast. It seems like I should change the -prediction or -dataTable, but I'm not sure which one to change or how to change it. I tried adding the contrast_tn_pain(less).orig files as a new column in the HRF.table, but that did not work.

Thanks again for all your help!

Caleb

Hi Caleb,

I successfully ran the first two analyses comparing the HRF b/w pain patients and healthy controls for under taste and tasteless conditions separately

Have you reviewed the estimated HDRs for both conditions for each of the two groups? Do they appear reasonable to you?

I'm not sure whether compute the contrast at the group or individual level.

The contrast between the two conditions for 3dMSS input should be computed at the individual level, not at the group level. This means that both the input data table and the prediction table for 3dMSS should follow the same structure as used in the previous two analyses for the respective conditions. Does this approach seem clear to you?

Gang Chen

Hi Gang,

I ran that analysis after computing the individual level contrasts and it worked! The output of all three analyses seem reasonable and the effect in each case is much more impressive than analyses where we used the canonical HRF. Thank you so much! This tool is incredible.

One more question: when I'm viewing the statistical maps, I'm unable to choose an underlay that would allow for easier visualization of the anatomical features of the brain. When I try to choose an MNI template as the underlay, for example, after I change the overlay to the output of 3dMSS, it removes the MNI template automatically and goes back to the "default" underlay (not sure if it's default in general or just for the 3dMSS output). I pasted a screenshot below. I tried aligning the output file to MNI but that did not work either. Do you know how to get rid of this? Sorry if it seems like a silly question--this is my first time using AFNI

I can jump in about the underlay/overlay question.

A reference space template (that is not in an "original" subject space) has info in the header that marks it as a special, whether in NIFTI or BRIK/HEAD format. MNI template space datasets do/should have this. The AFNI GUI will not show a dataset with "reference space" header info with a dset in original/native subject space.

So, the current situation probably means that this dataset: does not have a qform_code or sform_code appropriate for a reference dataset (if NIFTI), or space name appropriate for a reference/non-original space dataset (if BRIK/HEAD). Let's verify that. What is the output of:

# if nifti
nifti_tool -disp_hdr field qform_code field sform_code -infiles DSET_STAT

# if brik/head or nifti
3dinfo -space -av_space DSET_STAT

?

--pt

Hi ptaylor,

Thank you for your reply.

Here is the output when I run the first part of the code you provided:


When I run it with output_contrast or output_contrast+orig.HEAD I get the same result. output_contrast+orig.HEAD definitely exists in this directory so I'm not sure what this error means.

And here is the output for the second line:
image

For reference, output_contrast is the output of 3dMSS.

Thanks again!

Hi-

Those first errors are to be expected if the dataset is in BRIK/HEAD format (i.e., ending with .BRIK* and .HEAD) rather than NIFTI (i.e., ending with .nii or .nii.gz). That is not a problem. It's just why there were the comments "if nifti" etc. were above them.

OK, so the header info for the stats dataset says that the file is in original/native subject space. How were these processed prior to this? Was there alignment to a template space performed, or not? According to this header info, it would not be appropriate to overlay these dsets on a template volume, but instead on something like subject anatomical or subject EPI. If the latter is not the case, we can deal with it, but that is a question that requires knowing about the processing and what the final space from that was.

--pt

Hi pt,

I think that alignment was not performed properly. I am doing it again with FSL but at the same time trying to use AFNI's @SSwarper because it looks more convenient. But when I try to call it, I get this error:

I tried adding the AFNI directory to my path like in this suggestion but it does not help. Do you know what might be going wrong?

I thought it might be because I am running a version of AFNI from 2021, but when I look into the AFNI directory, the script for @SSwarper is there. Do you know what this error might mean?

Also I can make a separate thread if that would be better since this is completely different from the original issue haha

Hm, what is the output of:

afni_system_check.py -check_all

?

And did you follow AFNI install instructions for any of the variety of OSs here?

--pt

I'm not sure how it was installed because I'm running it through a shared cluster.

Here is the output from afni_system_check

[cmahlen@bhg0016 ses-bsl]$ afni_system_check.py -check_all
-------------------------------- general ---------------------------------
architecture: 64bit ELF
system: Linux
release: 3.10.0-1160.95.1.el7.x86_64
version: #1 SMP Fri Jun 23 08:44:55 EDT 2023
distribution: Red Hat Enterprise Linux Server 7.9 Maipo
number of CPUs: 24
** GPSS command failure for: ps h -o pid,ppid,user,comm -p 9401

error output:

** cannot detect shell: empty process stack
** GPSS command failure for: ps h -o pid,ppid,user,comm -p 9401

error output:

** cannot detect shell: empty process stack
apparent login shell: None
shell RC file: NONE (does not exist)

--------------------- AFNI and related program tests ---------------------
which afni : /software/afni/21.1.07/afni
afni version : Precompiled binary linux_openmp_64: May 11 2021
: AFNI_21.1.07 'Domitian'
AFNI_version.txt : AFNI_21.1.07, linux_openmp_64, May 11 2021
which python : /usr/bin/python
python version : 2.7.18
which R : /software/r/4.1.1/b1/bin/R
R version : R version 4.1.1 (2021-08-10) -- "Kick Things"
which tcsh : /usr/bin/tcsh

instances of various programs found in PATH:
afni : 1 (/gpfs/fs1/sfw2/afni/21.1.07/afni)
R : 1 (/gpfs/fs1/sfw2/r/4.1.1/b1/bin/R)
python : 1 (/usr/bin/python2.7)
python2 : 1 (/usr/bin/python2.7)
python3 : 1 (/usr/bin/python3.6)

testing ability to start various programs...
afni : success
suma : success
3dSkullStrip : success
uber_subject.py : success
3dAllineate : success
3dRSFC : success
SurfMesh : success
3dClustSim : success
3dMVM : success

checking for R packages...
rPkgsInstall -pkgs ALL -check : success

R RHOME : /software/r/4.1.1/b1/lib64/R

checking for $HOME files...
.afnirc : missing
.sumarc : missing
.afni/help/all_progs.COMP : missing

------------------------------ python libs -------------------------------
** failed to load module PyQt4
-- PyQt4 is no longer needed for an AFNI bootcamp

++ module loaded: matplotlib.pyplot
module file : /software/miniconda/2.7.18/lib/python2.7/site-packages/matplotlib/pyplot.pyc

-------------------------------- env vars --------------------------------
PATH = /software/hdf5/1.8.18/b1/bin:/software/gsl/2.5/b1/bin:/software/udunits2/2.2.20/usr/bin:/software/jdk/11.0.10/bin:/software/sqlite/3210000/bin:/software/proj/6.1.0/bin:/software/gdal/3.1.3/bin:/software/geos/3.7.2/bin:/software/gcc/9.1.0/libexec:/software/gcc/9.1.0/bin:/software/fftw3/3.3.10/b3/bin:/software/r/4.1.1/b1/bin:/software/afni/21.1.07:/software/libmng/2.0.3/b1/bin:/software/libpng/1.6.32/bin:/software/cuda/9.1/usr/local/cuda-9.1/bin:/software/fsl/6.0.5.1/bin:/software/slurm/current/bin:/software/firefox/current:/software/circ/bin:/xcatpost:/usr/bin:/bin:/sbin:/usr/sbin:/mnt/sysimage/bin:/mnt/sysimage/usr/bin:/mnt/sysimage/usr/sbin:/mnt/sysimage/sbin:/sbin:/usr/sbin:/xcatpost:/usr/lpp/mmfs/bin:/opt/ibutils/bin:/software/afni/21.1.07/afni

PYTHONPATH =
R_LIBS =
LD_LIBRARY_PATH = /software/hdf5/1.8.18/b1/lib:/software/mpfr/4.1.0/b1/lib:/software/gmp/6.2.1/b1/lib:/software/librsvg/2.40.20/usr/lib64:/software/gsl/2.5/b1/lib:/software/udunits2/2.2.20/usr/lib64:/software/jdk/11.0.10/lib:/software/sqlite/3210000/lib:/software/proj/6.1.0/lib:/software/gdal/3.1.3/lib:/software/miniconda/2.7.18/lib:/software/geos/3.7.2/lib:/software/gcc/9.1.0/lib64:/software/fftw3/3.3.10/b3/lib:/software/r/4.1.1/b1/lib64:/software/r/4.1.1/b1/lib64/R/lib:/software/libpng/1.2.50/usr/lib64:/software/libmng/2.0.3/b1/lib:/software/libpng/1.6.32/lib:/software/cuda/9.1/usr/local/cuda-9.1/lib64:/lib64:/software/slurm/current/lib64:/software/slurm/current/lib

DYLD_LIBRARY_PATH =
DYLD_FALLBACK_LIBRARY_PATH =

------------------------------ data checks -------------------------------
data dir : missing AFNI_data6
data dir : missing AFNI_demos
data dir : missing suma_demo
data dir : missing afni_handouts
atlas : found TT_N27+tlrc under /software/afni/21.1.07

------------------------------ OS specific -------------------------------
which yum : /usr/bin/yum
yum version : There was a problem importing one of the Python modules

========================= summary, please fix: =========================

  • just be aware: login shell 'None', but our code examples use 'tcsh'
  • please run: cp /software/afni/21.1.07/AFNI.afnirc ~/.afnirc
  • please run: "suma -update_env" for .sumarc
  • please run: apsearch -update_all_afni_help
  • insufficient data for AFNI bootcamp

There are some odd text outputs that system output; they have shown up once before in a thread, but I'm not sure how problematic those are.

I think you might have added the path+program to your path:

software/afni/21.1.07/afni

... rather than just the path:

software/afni/21.1.07

("which afni" outputs both the path and program name.) Could you try adjusting that, and opening a new terminal, and see if that works?

I am not sure that will resolve it, because tcsh is your login shell, and I do already see elsewhere in the $PATH output just the /software/afni/21.1.07 path.

Might have to ask the mighty @rickr for his opinion here...

On a side note, but which might be relevant: that version of AFNI is also quite old, and should really be updated. I expect there are improvements to @SSwarper since then (as well as other programs), and also we have an updated program sswarper2 that is preferred in its stead (almost all same options, and same output structure).

--pt

1 Like

While the 'ps' error is odd, even moreso is "basename: command not found". Just to see, what is the output from:

which basename
ps h -o pid,ppid,user,comm | head -n 3

Thanks,

  • rick

Hi Rick,

The output to which basename is
/usr/bin/basename

The output to the second line is this:
image

So given that those both work here, but not in a sub-shell, it seems likely that at least one of your dot files messes up your PATH. To test your shells, would you please run these commands and report the output:

sh
basename hi/there
exit

bash
basename hi/there
exit

tcsh
basename hi/there
exit

zsh
basename hi/there
exit

It seems unlikely for zsh/tcsh to be needed here, but let's be reasonably complete.

  • rick

Here is the output: