ETAC, fatal errors - perhaps due to compression?

Hey folks, more fun and exciting testing of ETAC has led to a few issues.

Issue the first:
The error appears to be related to file compression, which I have set as active in .afnirc. Relevant text:


++ 3dMultiThresh: AFNI version=AFNI_18.2.04 (Jul  6 2018) [64-bit]
++ -mthresh dataset parameters
 +      clustering NN=2  thresholding=1-sided  4 thresholds  2 hpows
++ MultiThresh cluster FOM threshold method = cdf 90.0%
++ Output dataset group_etac/hc/pre_pain_only_etac.ETACtmask.1pos.B0.0.nii.gz
++ Output dataset group_etac/hc/pre_pain_only_etac.ETACamask.1pos.B0.0.nii.gz
14413
++ 3dMultiThresh: AFNI version=AFNI_18.2.04 (Jul  6 2018) [64-bit]
++ -mthresh dataset parameters
 +      clustering NN=2  thresholding=1-sided  4 thresholds  2 hpows
++ MultiThresh cluster FOM threshold method = cdf 90.0%
++ Output dataset group_etac/hc/pre_pain_only_etac.ETACtmask.1pos.B5.0.nii.gz
++ Output dataset group_etac/hc/pre_pain_only_etac.ETACamask.1pos.B5.0.nii.gz
18675
++ processing 1 input datasets...
++ padding all datasets by 0 (for dilations)
** FATAL ERROR: failed to open mask dataset 'group_etac/hc/pre_pain_only_etac.ETACtmask.1pos.*.nii'
** Program compile date = Jul  6 2018
++ 3dbucket: AFNI version=AFNI_18.2.04 (Jul  6 2018) [64-bit]
** ERROR (nifti_image_read): failed to find header file for 'group_etac/hc/pre_pain_only_etac.ETACamask.1pos.*.nii'
can't open dataset group_etac/hc/pre_pain_only_etac.ETACamask.1pos.*.nii
++ 3dttest++ ----- merging 2 blur cases to make neg 1-sided activation mask

Specifically, it says it writes the *ETACtmask.1pos file as a .nii.gz, but then cannot load it later. I checked, they are definitely there, and definitely compressed.

Would an fix be resetting the compression environment variable and then running (beautiful and slow) ETAC in that terminal window? Want to make sure I know the right ones to set.

Issue the second

My ETAC run output, including the basic 3dttest++ maps are +orig as far as AFNI can tell. I cannot overlay them onto the AFNI templates. This is not an issue when I use 3dMEMA on the same data, and all input, including the group mask are TLRC. Any thoughts? Could this be related to my above compression obsession?

I’ll see if I can do a bit more testing, in time. Thanks for any help!

I found time to check on this first issue - changing the AFNI_COMPRESSOR environment variable in the .afnirc such that it was commented back out again fixed things. I now have lovely ETAC output. Setting the AFNI_COMRPESSOR to nothing right before running my etac script (aka export AFNI_COMPRESSOR=; tcsh cmd.etac,pain_on_brain) also worked just fine, and does not require any .afnirc editing.

The second issue still remains, data think they are ‘orig’, which can be corrected by running 3drefit afterwards - not sure what could be happening there. This is with the most updated version of AFNI (updated today). No sign of any warnings of anything that explain why orig space is being used for output…

To end on a positive note, the 5percent files are produced now, albeit it with the warning that some were not available.

I’m having a lovely conversation with myself here, but I think I have figured out the source of the second issue as well, that is, my ETAC output being in “orig” space.

I was fiddling with @chauffeur_afni, and having trouble overlaying my statistics onto a (tlrc) template brain. The afni viewer has no problem doing so, as the stats BRIK is identified as +tlrc, however, running 3dinfo shows that the stays are in ORIG template space.

I believe this is due to the data being multiecho, specifically importing tedana output as the combined output in afni_proc.
AFNI passes the volreg datasets to the specified tedana.py, and in my case these are volreg+tlrc, with 3dinfo reporting Template space:MNI. All is as it should be here.
The tedana output is in .nii format, and despite being copied into afni as +tlrc data (*combine+tlrc), 3dinfo reports Template Space: ORIG.

To fix this in my existing data, I can just run 3drefit and set the space to tlrc, and everything finally works!

A long term solution would be a 3drefit call on either the nii data output by tedana, or on the combine dataset that is created, checking out course that it is +tlrc before doing so. Another solution may be found in AFNI_NIFTI_VIEW being set to “tlrc” in .afnirc (have not yet tested…).

Any thoughts from the AFNI gurus on this?

Re. @chauffeur_afni: you can paste what you are trying with teh error involved, but also note that some examples (with the most modern @chauffeur_afni) now exist here:
https://afni.nimh.nih.gov/pub/dist/doc/htmldoc/tutorials/auto_image/auto_@chauffeur_afni.html

Re:

I was fiddling with @chauffeur_afni, and having trouble overlaying my statistics onto a (tlrc) template brain. The afni viewer has no problem doing so, as the stats BRIK is identified as +tlrc, however, running 3dinfo shows that the stays are in ORIG template space.

→ it surprises me that it would be any different. @chauffeur* just runs AFNI in a virtual environment. I don’t see how that same problem wouldn’t happen in the viewer.

Also, if your stats have been calculated in an aligned space to “tlrc”, then I don’t see why they wouldn’t have the +tlrc space value… If you want, you can upload the dataset and I can take a look at that aspect.

–pt

There was no error, technically - I just ended up with an ugly figure, and noticed that when @chauffeur was running AFNI, that the view switched to Talairach and then to original as it placed the overlay. This led to the figures being underlayless. Examining the temp files with 3dinfo revealed the different spaces. @Chaffeur had correctly copied the underlay (MNI 2009 template), but when taking the stats file it wrote a .nii that was in orig space.

The stats filename had +tlrc, as that was placed by afni_proc, but it seems like that was just limited to the -view and not -space, as seen in the the 3dinfo report for that file:


$ 3dinfo pb05.sub-8019sham_ses-preTMS.r01.combine+tlrc.
++ 3dinfo: AFNI version=AFNI_18.2.05 (Jul 25 2018) [64-bit]

Dataset File:    pb05.sub-8019sham_ses-preTMS.r01.combine+tlrc
Identifier Code: AFN_XnN-OjaGPD4LsMp30DVHSg  Creation Date: Fri Jul 20 19:03:57 2018
Template Space:  ORIG
Dataset Type:    Echo Planar (-epan)
Byte Order:      LSB_FIRST [this CPU native = LSB_FIRST]
Storage Mode:    BRIK
Storage Space:   970,933,040 (971 million [mega]) bytes
Geometry String: "MATRIX(-2.5,0,0,95,0,-2.5,0,131.75,0,0,2.5,-77):77,92,77"
Data Axes Tilt:  Plumb
Data Axes Orientation:
  first  (x) = Left-to-Right
  second (y) = Posterior-to-Anterior
  third  (z) = Inferior-to-Superior   [-orient LPI]
R-to-L extent:   -95.000 [R] -to-    95.000 [L] -step-     2.500 mm [ 77 voxels]
A-to-P extent:   -95.750 [A] -to-   131.750 [P] -step-     2.500 mm [ 92 voxels]
I-to-S extent:   -77.000 [I] -to-   113.000 [S] -step-     2.500 mm [ 77 voxels]
Number of time steps = 445  Time step = 1.35000s  Origin = 0.00000s
  -- At sub-brick #0 '#0' datum type is float:            0 to       17004.8
  -- At sub-brick #1 '#1' datum type is float:            0 to       17074.4
  -- At sub-brick #2 '#2' datum type is float:            0 to         17388
** For info on all 445 sub-bricks, use '3dinfo -verb' **

----- HISTORY -----
[dowdlelt@hypnos: Fri Jul 20 19:03:57 2018] {AFNI_18.2.04:linux_ubuntu_16_64} 3dcopy tedana_r01/TED.r01/dn_ts_OC_T1c.nii pb05.sub-8019sham_ses-preTMS.r01.combine
[dowdlelt@hypnos: Fri Jul 20 19:04:13 2018] {AFNI_18.2.04:linux_ubuntu_16_64} 3drefit -view tlrc pb05.sub-8019sham_ses-preTMS.r01.combine+orig


The tedana processing, which just writes .nii files, screwed up that information (header is lost), and when it was copied back in with, the header information was updated, but not the space. It is easy enough to do a recursive call on my data with 3dRefit. I already do it for all of the tedana output, just need to extend it to the stats datasets.

If I can tease apart anything else I’ll update. It is strange though, AFNI has no problem overlaying the combined data, but when it is copied (I guess?) and written as a .nii again, things go wrong.

Well, if a program (tedana in this case) is messing up the header info, then, well… the header info might be messed up.

There are two properties here: space and av_space; the latter is the “view” one. For example, the output of this 3dinfo command on the TT_N27 is given here:


3dinfo -prefix -space -av_space /data/REF_TEMPLATES_AFNI/TT_N27+tlrc.
      TT_N27	TLRC	+tlrc

Sounds like you want to change the space. I will cite the 3drefit help for both, which are conveniently neighboring:


-view code      Changes the 'view' to be 'code', where the string 'code'
                  is one of 'orig', 'acpc', or 'tlrc'.
               ** WARNING: The program will also change the .HEAD and .BRIK
                  filenames to match.  If the dataset filenames already
                  exist in the '+code' view, then this option will fail.
                  You will have to rename the dataset files before trying
                  to use '-view'.  If you COPY the files and then use
                  '-view', don't forget to use '-newid' as well!
               ** WARNING2: Changing the view without specifying the new 
                  might lead to conflicting information. Consider specifying
                  the space along with -view
  -space spcname  Associates the dataset with a specific template type, e.g.
                  TLRC, MNI, ORIG. The default assumed for +tlrc datasets is
                  'TLRC'. One use for this attribute is to use MNI space
                  coordinates and atlases instead of the default TLRC space.
               ** See WARNING2 for -view option.

… note those WARNING2 comments (which are missing a word, yes: “without specifying the new [space] might lead to conflicting information…”, but hopefullyi the meaning is clarified in the next sentence).

See in your 3dinfo output the line:


Template Space:  ORIG

which conflicts with the “view” property. I think your sitation is exactly what is being WARNING2ed against.

Note: I don’t think a figure can be “underlayless”. It probably just just selecting the offending olay itself, or perhaps some other file in that space (which may not even appear in teh FOV, potentially).

–pt

underlayless was the wrong word, but yes - it was using itself as an underlay with very bad intensity ranges. I desire my blobs to be on brains, not empty black space! Unfortunate that this processing strips away all the friendly AFNI information, but it is at least correctable.

It looks like afni_proc used 3drefit on the data, but only changed the view and not the space, according to the limited history that came with the combine+tlrc file. I’ll run 3drefit based on my volreg data (tlrc view, but MNI space) that was used as input to tedana - tedana does not perform any spatial shifts, thankfully.

Appreciate the clarification - happy to have figured out the source of all these problems finally.

Well, the “bad ranges” are just default: 2%-98% in the viewed slice, I think, unless an environment variable is set otherwise. NB: that is typically reasonable-ish for an anatomical, which is often a desirable underlay, as you have noted.

What is the tedana program causing these issues? That person can be pinged about such changes occurring unhappily, if the program is changing header information badly.

By “spatial shifts”, do you mean no changes to the origin+orient information are happening?

–pt

Yes, the proc script updates the view but not that space.
That is an oversight on my part, I will fix it.

Thanks!

  • rick

Thanks for following up - I’ve got nice figures now, after correcting on space and playing a bit more with chauffeur.

Tedana does a lot of processing with python. I am not sure that carrying over AFNI header information is a priority :frowning: Fortunately, it is effectively a last step for me, and I keep the volreg output around for further QC. I do believe it has been noticed by them however, so there is a hope. And yes, it doesn’t move the data around at all. Rick’s fix should resolve the problem nicely, as correcting the space has definitely got things working for me.

Thanks, and keep up all the awesome!

Thanks for this - I’ll keep trying to break things (well intentioned, of course).

Good luck keeping up the destructive rampage. :slight_smile:

BTW, I put that fix in yesterday, but did not do a build.
Would you like a binary update, or are you grabbing
things from github? If the latter, it is basically an update
to db_mod.py.

Thanks,

  • rick

I’ll just wait on it - but I did see it on github, and look forward to testing it out in the near future. Thanks again