3dISC-sharing Error ("The content under -dataTable is not rectangular !", and the solution

[Manqi001]

Hi, I just want to share my problem and the solution with people who could meet the same problem.

I am newcomer to 3dISC and tried to perform the LME of two groups.

firstly, I wrote "ISC.sh" as follow:
3dISC -prefix ISC2a -jobs 12
-mask myMask+tlrc
-model 'grp+(1|Subj1)+(1|Subj2)'
-gltCode ave '1 0 -0.5'
-gltCode G11 '1 1 0'
-gltCode G12 '1 0 1'
-gltCode G22 '1 -1 -1'
-gltCode G11vG22 '0 2 1'
-gltCode G11vG12 '0 1 -2'
-gltCode G12vG22 '0 1 2'
-gltCode ave-G12 '0 0 -1.5'
-dataTable @data.txt \

in the "data.txt", I arranged my data as follow:

Subj1 Subj2 grp InputFile
s01 s02 G11 s01_02+tlrc
s01 s03 G11 s01_03+tlrc
……
s04 s06 G22 s04_06+tlrc
s05 s06 G22 s05_06+tlrc

When I run the ISC.sh in Linux terminal, error was reported:

3dISC -prefix ISC2a
Error in if (len%%wd != 0) errex.AFNI(paste("The content under -dataTable is not rectangular !", :
参数长度为零(the parameter length is zero)
Calls: process.ISC.opts
停止执行(stop execution)
……

The next day I read the paper “Inter-subject correlation during long narratives reveals widespread neural correlates of reading ability”, and Dr. Jangraw shared his code on Github that inspired me : this error occurred probably because I didn't align the variables.

Later, I just unified the spacing (number of spaces) between variables in each row in data.txt, then the 3dISC execution was smoothly.

[ptaylor]

Thanks for mentioning this.

I would be surprised if the number of spaces mattered, as long as there was at least one space between each column. I wonder if a space was missing in the iniital table?

Personally, I do find uniform spacing is useful for me at a human-level, though, so I can see my data more easily and visually check things. So, that would certainly seem helpful.

--pt

[Manqi001]

Hi ptaylor, I think the number of spaces matters. I am not familiar with the discuss message board……I dont know why, after posting, all spaces in the gap were reduced to only one space.

in fact, the data.txt is as follow:
Subj1 Subj2 grp InputFile \ #the numbers of spaces between variables are 1, 4, 6, and 4
s01 s02 G11 s01_02+tlrc #the numbers of spaces between variables are 5, 6, 6, and 4
s01 s03 G11 s01_03+tlrc
s01 s04 G12 s01_04+tlrc
s02 s03 G11 s02_03+tlrc
s02 s04 G12 s02_04+tlrc
s03 s04 G12 s03_04+tlrc
……

Hope this helps.

[ptaylor]

Hi-

The spaces disappeared in your initial post because they were not put between backticks, to put them into "preformatted text"; highlighting and clicking the "</>" button at the top of the text field would do the same. Please see here:

typing about code or terminal text: please use backticks (and avoid screenshotting text)

Also, a fun feature of the codeblock backticks is that you can specify the highlighting language within the codeblock. Just put the name of the available script/programming language in the top line, where the opening backticks are: here, I started the codeblock with: ``` bash #!/bin/bash # comment varA=$1 varB=$2 echo "++ The variables are: ${varA} ${varB}" here, I started the codeblock with: ``` python # comment varA = 'some string' varB = '''another string''' print("++ The variables…

When putting what you posted in backticks, this is what it looks like:

• Initial post, first code block:

3dISC -prefix ISC2a -jobs 12 
-mask myMask+tlrc                     \
-model  'grp+(1|Subj1)+(1|Subj2)'     \
-gltCode ave     '1 0 -0.5'           \
-gltCode G11     '1 1 0'              \
-gltCode G12     '1 0 1'              \
-gltCode G22     '1 -1 -1'            \
-gltCode G11vG22 '0 2 1'              \
-gltCode G11vG12 '0 1 -2'             \
-gltCode G12vG22 '0 1 2'              \
-gltCode ave-G12 '0 0 -1.5'           \
-dataTable @data.txt                         \

• Initial post, second code block:

Subj1 Subj2    grp      InputFile       \
s01     s02      G11      s01_02+tlrc     \
s01     s03      G11      s01_03+tlrc     \
……
s04     s06      G22      s04_06+tlrc     \
s05     s06      G22      s05_06+tlrc

I will send you an email, and you can send us the first and second text files to take a look. I don't think the number of spaces should matter when reading in a file, but seeing the file will help. Note that when you have a datatable file, you also should not need the backslash at the end of the line: \. That "continuation of line character" is not necessary there.

Also, it might be helpful to update your AFNI. Recently, @discoraj added in nice functionality to be more specific about where non-rectangularity occurs in a datatable (like, reporting the line number). It will also give some helpful comments on datatable info, too.

--pt

[Manqi001]

I found that the most important change I made was to modify the way I used 3dISC commands and data. I have packaged the code&data and attached a description (1lookfisrtXXX.txt) and send them to your mailbox. I would really appreciate it if you help me check them.
thanks,
manqi

[Manqi001]

hi everyone who are viewing this post, I have to clarify the error "-dataTable is not rectangular" was a subtle issue of having a bad character (the backslashes) in my initial text file, rather than a question of equal spacing. The subtle issue about the backslashes can happen sometimes with certain text editors (esp. on macOS), or copying from a PDF, or from other ways.
Thanks for the help from ptaylor. I contacted ptaylor by personal email and he help me check my initial and final codes & files.
Hope this will help you if you met the same problem.

[ptaylor]

Yes, just to add more detail here about how troubleshooting this proceeded:

When first running the script "ISC.txt" that basically just contained the 3dISC command, this is the output shown:

$ tcsh ISC.txt 
updating R_LD_LIBRARY_PATH ...
Error in if (len%%wd != 0) errex.AFNI(paste("The content under -dataTable is not rectangular !",  : 
  argument is of length zero
Calls: process.ISC.opts
Execution halted
-mask: Command not found.
-model: Command not found.
-gltCode: Command not found.
-gltCode: Command not found.
-gltCode: Command not found.
-gltCode: Command not found.
-gltCode: Command not found.
-gltCode: Command not found.
-gltCode: Command not found.
-gltCode: Command not found.
-dataTable: Command not found.
...

I found the "Command not found" aspect perplexing. I thought that there might be a space to the right of the continuation of line characters \ that were (appropriately) at the end of the lines, but there didn't appear to be any there. And nothing else looked erroneous in the script either.

So, being a naturally suspicious person, I decided to run Rick's convenient file_tool program on the script. This was the command and output:

$ file_tool -show_bad_all -infiles ISC.txt 
ISC.txt has 0 bad characters
 
ISC.txt file type: DOS
consider: file_tool -show_file_type -infile ISC.txt -prefix FIXED.txt
file 'ISC.txt' has bad backslashes:
   bad '\' line    1 :     3dISC -prefix ISC2a -jobs 12                \
   bad '\' line    2 :           -mask 3mm_standard.nii                \
   bad '\' line    3 :           -model  'grp+(1|Subj1)+(1|Subj2)'     \
   bad '\' line    4 :           -gltCode ave     '1 0 -0.5'           \
   bad '\' line    5 :           -gltCode G11     '1 1 0'              \
   bad '\' line    6 :           -gltCode G12     '1 0 1'              \
   bad '\' line    7 :           -gltCode G22     '1 -1 -1'            \
   bad '\' line    8 :           -gltCode G11vG22 '0 2 1'              \
   bad '\' line    9 :           -gltCode G11vG12 '0 1 -2'             \
   bad '\' line   10 :           -gltCode G12vG22 '0 1 2'              \
   bad '\' line   11 :           -gltCode ave-G12 '0 0 -1.5'           \
   bad '\' line   12 :           -dataTable                            \
...
consider: file_tool -show_bad_backslash -infile ISC.txt -prefix FIXED.txt

Indeed, there were bad characters lurking there, and this is a subtle kind of issue to have. Fortunately, file_tool can check for bad characters that happen---like, when copy+pasting from a PDF or DOC/DOCX file, or from an email text, or even using certain text editors (looking at you, Apple/Mac operating system, for producing these sometimes!). These are very subtle issues that are hard for a user/researcher to see by eye, and can happen basically unbeknownst to the person writing the script.

So, the lesson is, if you ever get weird/unexpected behavior from a script, especially where an option on its own line is "unrecognized", perhaps run the file through file_tool, as shown above.

This also shows the benefits of using a script for any command and analysis: one can check it later, as well as share it easily with other folks, either in a paper/supplement or in a Message Board exchange for troubleshooting.

--pt