Extracting voxel-wise data with 3dmaskdump and recombing it with 3dUndump

Hi,

this is going to be a bit complicated, but I try to describe my problem as simple as possible.

I extracted voxel-based time-series from AFNI functional scans (3x3x3 mm voxels) using 3dmaskdump. Subsequently, I loaded the extracted time-series into Python, and computed one measurement for each voxel (meaning I get one value per voxel, such as 7.54 or 2.13).

Furthermore, I also extracted the 3D coordinates of the template I am using (Schaefer-Yeo-AFNI-2021 Atlas 1000 parcels version) after resampling this atlas to 3x3x3 mm voxels with 3dresample, so that the anatomical voxel size matches the functional voxel size. Here is the 3dresample code for the atlas:


3dresample \
-input /volumes.../Schaefer_7N_1000_ribbon.nii.gz \ # AFNI atlas mentioned above
-master /volumes/errts.Subject1.anaticor+tlrc \ # Preprocessed functional run
-prefix Resample_Yeo7N_1000ribbon.nii

Here is the code that I used to extract the voxel-based coordinates after resampling the atlas:


3dmaskdump \
-mask $network.nii \ # Was part of a for loop to select several networks
-noijk \
-xyz \
-quiet \
Resample_Yeo7N_1000ribbon.nii \ # The 3x3x3 mm resampled Schaefer-Yeo-AFNI-2021 Atlas 1000 parcels version
> Cor_${network}.1D # Save xyz coordinates to .1D files

Next, I computed a measurement in Python, loading the 3dmaskdump extracted voxel time-series into Python.
Furthermore, I also loaded the the resampled atlas 3D coordinates into Python.
Using Python, I then saved the results in the following format, where the results are saved row-wise in a .1D file


x y z measurement_result

This means that the first index is the x, the second the y, and the third the z 3D space voxel coordinate. Finally, the fourth index is the result of the measurement.
Only spaces were added between x y z and the voxel value, nothing else.

I then compared the coordinates extracted by 3dmaskdump and the voxel coordinates + result value by Python. Both files exactly match, meaning that the voxel coordinates (xyz) are exactly the same, from the first to the last voxel.

Next, I wanted to transform my voxel coordinate plus measurement result .1D files into NIfTI files. Therefore, I used 3dUndump. I loaded the .1D files saved by Python (the previous step above) and wanted to save those .1D files as NIfTI files using 3dundump Here is my code:


3dUndump \
-xyz \
-prefix someprefix \
-master Resample_Yeo7N_1000ribbon.nii \  # The 3x3x3 mm resampled Schaefer-Yeo-AFNI-2021 Atlas 1000 parcels version
-mask $dir_rois/$network.nii \ # The specific networks I chose (using a for loop here)
acw_${network}.1D # The input files, again part of a for loop

Now, in this step using 3dUndump comes the problem. For many (although not for all) voxels AFNI prints the following error:

[quote=“*+ WARNING: File /volumes/…/acw_Global.1D line 14331: y coord=-97.5 is outside -96.003 … 132.003
*+ WARNING: File/volumes/…/acw_Global.1D line 14332: y coord=-97.5 is outside -96.003 … 132.003
*+ WARNING: /volumes/…/acw_Global.1D line 14333: y coord=-97.5 is outside -96.003 … 132.003
*+ WARNING: /volumes/…/acw_Global.1D line 14334: y coord=-97.5 is outside -96.003 … 132.003
*+ WARNING: /volumes/…/acw_Global.1D line 14335: y coord=-97.5 is outside -96.003 … 132.003
*+ WARNING: /volumes/…/acw_Global.1D line 14990: y coord=-97.5 is outside -96.003 … 132.003”]
[/quote]

I shortened AFNI’s output above. Loading the resulting NIfTI file of 3dUndump into AFNI or MRIcroGL looks ok, I do not see any flaws there.
However, when I plot those NIfTI result by 3dUndump via nilearn (Python), I can see that especially the posterior part of the brain seems to lack any values or is not plotted, it just plots a pure greenish area, the real values are somehow not placed on the brain’s surface.

I am wondering if that problem is related to the error by 3dUndump above and not to Python itself. I assume that the voxels’ affected by the WARNINGs above are somehow displaced in 3D space, meaning that nilearn in Python cannot properly find and place all voxels within the NIfTI file produced by AFNI’s 3dUndump.


The attached picture shows the result of the nilearn plot that loads the NIfTI file produced by 3dUndump. The picture also shows the result of the same brain when loading the voxel coordinates plus my measurement results into Python, and plotting the same brain as 3D scatter plot (instead of using the AFNI’s 3dUndump created NIfTI file).

You can clearly see tha the nilearn brain (that is using the NIfTI output by AFNI’s 3dUndump) lacks ~50% of data for the brain surface, where, instead, it only shows a massively green surface which is clearly wrong. At the same time, the 3d scatter plot that is not using AFNI’s 3dUndump shows a perfectly fine result.

Can you guys detect a mistake in my AFNI scripts shown above? What am I doing wrong in the AFNI-related scripts so that 3dUndump prints warnings for many voxels that are out of space?

Here is a picture between the 3dundump output (above, first row) and the resampled anatomical template (second row).

The mismatch is clear between those files, hence the mismatch occurs within 3dundump, because in all previous processing steps, the voxel coordinates are exaclty the same.
I wonder what needs to be changed in the 3dundump script?

Hi, Philipp-

Maybe the best place to start is running this:


3dinfo -same_all_grid -prefix DSET1 DSET2 DSET3 ...

on all the dsets you are extracting data from and writing. That will point out where you have a mismatch of grids.

–pt

Dear Paul,

thank you.
I read on Andy’s Brain Blog (Andy's Brain Blog) that AFNI’s xyz coordinates correspond to MNI152’s x -y z.
More precisely, the positive y value in AFNI’s xyz coordinates corresponds to a negative y value in MNI152 (and vice versa).

So I went back to my Python script that initially loads the output of 3dmaskdump. In Python, I multiplied the y coordinates (extracted from AFNI) with -1.


y = y * -1

Then, I ran 3dUndump again, but with the multiplied y-coordinate values.
Now, the code you provided me yields no mismatch:


3dinfo \
-same_all_grid \
-prefix Extracted_ACW_Awake.nii Resample_Yeo7N_1000ribbon.nii

prints


NO-DSET	NO-DSET	NO-DSET	NO-DSET	NO-DSET	NO-DSET
0	0	0	0	0	Resample_Yeo7N_1000ribbon.nii

The attached image below shows a visual comparison between both files. The visual inspection shows that there are still some slight mismatches.
The cortical grey matter looks more thick in the resampled atlas version than in the functional measurement-based result (of course, I used the same 1000 parcels ribbon version in both cases!).
Based on the information that I provided so far, do you have any idea where the slight mismatches can come from?

At least the data looks to be identically positioned in space, after multiplying the y-coordinates with -1.

Thanks so far.

Hi, Philipp-

Many times we use 3dinfo on just a single dataset as input. The command I gave you should be run on multiple datasets at once. Can you please run:


3dinfo \
-same_all_grid \
-prefix \
Extracted_ACW_Awake.nii Resample_Yeo7N_1000ribbon.nii \
DSET2 \
DSET3 \
....

where you fill in all your other datasets being used?

thanks,
pt

I think the whole problems stems from the radiological vs. neurological coordinates.
My goal is to display AFNI preprocessed data on the MNI 152 template.

To make it short: would the correct solution to the problem be to simply extract voxel-based data using 3dmaskdump via adding the option


-nbox 3 3 3

to 3dmaskdump?

As I understand, -nbox 3 3 3 would extract the voxel-based data in the neurological space matching the MNI152 template, and the 3 3 3 specification would match AFNI’s 3x3x3 mm functional and preprocessed voxel size that I have.
Subsequently, I would no longer have to play around with changing the coordinates (either x, y, or z) in Python.

Paul, what do you think?

Sorry, I just saw your answer, will test that now.

Paul,

the two .nii datasets that I used are:

“Extracted_ACW_Awake.nii” and “Resample_Yeo7N_1000ribbon.nii”. Hence I use the code, where both files are in the same folder:


3dinfo \
-same_all_grid \
-prefix \
Extracted_ACW_Awake.nii \
Resample_Yeo7N_1000ribbon.nii

which prints


NO-DSET	NO-DSET	NO-DSET	NO-DSET	NO-DSET	NO-DSET
0	0	0	0	0	Resample_Yeo7N_1000ribbon.nii

Is that correct?

Hi, Philipp-

That first dataset has not been entered correctly, since the program cannot find it, hence the NO-DSET messages. Please make sure that is the correct name and/or directory—path information can be included to locate it, too.

–pt

Paul, sorry, it was a long day for me. I indeed made a typo, hence AFNI could not find one file.

Here is the new output:


Resample_Yeo7N_1000ribbon.nii
1	1	1	1	1	         ACW_Awake_Global.nii
1	1	1	1	1	Resample_Yeo7N_1000ribbon.nii

Note that this output (the ACW_Awake_Global.nii file) was generated after I multiplied the y-coordinates with -1 in Python, leaving the x and z coordinates untouched. By untouched, I mean that I extracted those with the “-xyz” command in 3dmaskdump.

Hi, Philipp-

The output:


1	1	1	1	1	         ACW_Awake_Global.nii
1	1	1	1	1	Resample_Yeo7N_1000ribbon.nii

… suggests that these two files are on the same grid.

I wouldn’t do any ad hoc coordinate adjustments in Python. That would mean that header information could easily become incorrect, and that is possibly happening here? I am not sure—but I don’t see any reason to introduce a coordinate change within Python.

As far as I see it, you have file of xyz coords and N values per line that you read into Python:


x0 y0 y0 vala valb valc vald ....
x1 y1 y1 vala valb valc vald ....
x2 y2 y2 vala valb valc vald ....
...

so you should just be able to output then output a file that has the exact same coords in each row and the new quantity for each row:


x0 y0 y0 py_out0
x1 y1 y1 py_out1
x2 y2 y2 py_out2
...

Then use 3dUndump on this new dataset; I don’t think you should even need to use a mask in that command, since you only have values within the original mask, anyways.


However, I am a little confused about your analysis here. Do you have 2 different “input” files on which you are running 3dmaskdump—one being a 4D dataset, and one being a template or atlas, from which you want coordinates? I don’t see why you would want coordinates from a separate dataset.

So, perhaps could you first explain what your starting point information is (one 4D dataset of time series that is in standard space, and then a template or atlas in the same standard space?), and what you want to do with each piece of information? That will help.

thanks,
pt

Hi Paul,

I will try to specifiy what my aim is. I compute the temporal autocorrelation (AC) for each voxel. How do I do that?

  1. I extract the voxel-based time-series from the preprocessed AFNI errts. file (aligned to MNI152) using 3dmaskdump into one .1D file per subject.
  2. Next, I load this .1D file into Python. I compute the AC for every voxel and save those voxel values in a big list.
  3. I take the mean across all subjects for each voxel, also in Python.

Now I would like to plot that AC “map” on a 3D surface brain.

  1. I also loaded the voxel coordinates from a resampled Schaefer-Yeo AFNI template that I build myself. How did I build that file myself? I downloaded the data that you provide here: https://afni.nimh.nih.gov/pub/dist/atlases/SchaeferYeo/ Using that data, I created the 7-networks 1000 parcels version with AFNI myself. This creation is 99% correct, I also plotted the brain in Python, I inspected it in AFNI, all looks perfect. This is the file called “Resample_Yeo7N_1000ribbon.nii” in my previous posts, because I “downsampled” this file to the 3x3x3 mm voxel resolution after I created it, so that it matches the functional resolution.
  2. Next, I create a new .1D file using Python. In this file, I save the new information as follows: the first row starts with the first voxel coordinate, that is, it starts with x y and z. Next, the 4th index gets the AC value, such as 4.55 (a single value!). Hence, every row contains 4 numbers, the x y z coordinates plus the AC value.
  3. I load this .1D file with 3dUndump to create a new NIfTI file that, in turn, can be loaded back into Python by nilearn, finally allowing to plot this “AC map” on a 3D brain surface.

I should add that the extracted time-series in the first step (1) are already extracted using the different networks, that it, with a for loop for different masks in 3dmaskdump. Hence, the coordinates between those extracted time-series and the template have to be the same, there should be no mismatch, as the previous test showed. However, there is only no mismatch if I mirror the y coordinates in Python by multiplying them with -1.

Hi, Philipp-

Thanks for that explanation.

When you refer to the Scaefer-Yeo dataset, I guess that means that atlas (ROI map of regions), right? That is different than a template.

But the main point is you want to use this ROI information with the errts file (or more specifically autocorrelation derived from errts file) information? Can you please check that your Resample_Yeo7N_1000ribbon.nii and errts* datasets are on the same grid, using the “3dinfo -same_all_grid …” information?

Also, I don’t see why you want to get coordinate information from Resample_Yeo7N_1000ribbon.nii and replace that information from the earlier 1D files. If the errts data are not on the same grid as the Resample*ribbon.nii file, then it should not be used to replace coordinates. If those two files are on the same grid, then replacing coordinates is unnecessary and might just lead to a scripting issue. Actually, I don’t see how Resample_Yeo7N_1000ribbon.nii is used at all here.

–pt

Hi Paul,

here is the output:


1	1	1	1	1	              ACW_Awake_Global.nii
1	1	1	1	1	     Resample_Yeo7N_1000ribbon.nii
1	1	1	1	1	     errts.Subject1_Awake.anaticor

And you are right. I guess that I can skip using the “Resample_Yeo7N_1000ribbon.nii” here, and directly use the coordinates from the .errts file.

However, now I tell you something interesting.
When I don’t mirror the y coordinates in Python, that is, when I don’t run y = y * -1 on them, I again get the


*+ WARNING: File /volumes/sandisk/fmri/dataset_3/processed/Extracted_ACW_Awake/acw_Global.1D line 14333: y coord=-97.5 is outside -96.003 .. 132.003
*+ WARNING: File /volumes/sandisk/fmri/dataset_3/processed/Extracted_ACW_Awake/acw_Global.1D line 14334: y coord=-97.5 is outside -96.003 .. 132.003
*+ WARNING: File /volumes/sandisk/fmri/dataset_3/processed/Extracted_ACW_Awake/acw_Global.1D line 14335: y coord=-97.5 is outside -96.003 .. 132.003
*+ WARNING: File /volumes/sandisk/fmri/dataset_3/processed/Extracted_ACW_Awake/acw_Global.1D line 14336: y coord=-97.5 is outside -96.003 .. 132.003
*+ WARNING: File /volumes/sandisk/fmri/dataset_3/processed/Extracted_ACW_Awake/acw_Global.1D line 14987: y coord=-97.5 is outside -96.003 .. 132.003
*+ WARNING: File /volumes/sandisk/fmri/dataset_3/processed/Extracted_ACW_Awake/acw_Global.1D line 14988: y coord=-97.5 is outside -96.003 .. 132.003
*+ WARNING: File /volumes/sandisk/fmri/dataset_3/processed/Extracted_ACW_Awake/acw_Global.1D line 14989: y coord=-97.5 is outside -96.003 .. 132.003
*+ WARNING: File /volumes/sandisk/fmri/dataset_3/processed/Extracted_ACW_Awake/acw_Global.1D line 14990: y coord=-97.5 is outside -96.003 .. 132.003

error for maaaaannyyy voxels in AFNI’s 3dUndump. However, even when I now run the 3dinfo grid check on those files, that is, including the 3dUndump NIfTI file that uses the original y-coordinates (and not the ones mirrored in Python), I get the following output:


1	1	1	1	1	              ACW_Awake_Global.nii
1	1	1	1	1	     Resample_Yeo7N_1000ribbon.nii
1	1	1	1	1	     errts.Subject1_Awake.anaticor

Isn’t that strange?

Ok, I think that I found the problem.
What a mess, but let me explain, I now remember another detail that I did here.

I extracted the voxel-based time series with 3dmaskdump using the option “-no ijk” so that the resulting .1D files do NOT include the voxel coordinates.
Why? Because I wanted to run interpolation in Python, and it was easier for me to script it this way.

Subsequently, I also extracted the voxel coordinates from the atlas file that I created.
Now, comparing the voxel coordinates between the atlas and the 3dmaskdump extracted errts files shows me that the voxel coordinates are NOT the same.
Sorry! I feel like an idiot.

I thought that both voxel coordinates should match, since the preprocessed errts files used the MNI152 template.

I guess that is the core of the problem.

Edit: Paul, forget what I wrote above. If I extract all voxel coordinates from the preprocessed errts. file, of course I will get many voxels that do not match the atlas, since the errts file includes all voxels, whereas the atlas/mask/roi does not. Furthermore, 3dinfo showed that “Resample_Yeo7N_1000ribbon.nii” (my atlas file) an the errts file do match.I have no idea anymore where the problem comes from, I am fully lost now.

Paul,

I think that I maybe solved the problem now. The problem was that I extracted the networks’ coordinates with the 3dmaskdump option


-xyz

enabled.
By “networks coordinates” I mean the coordinates of the 7 AFNI Schaefer-Yeo networks that I initially created, plus a combination of all 7 networks together, where I added the 7 single networks together with 3dcalc because I also wanted to investigate the complete cerebral cortex in addition to the 7 individual networks.

Hence, the voxel coordinates were extracted in the radiological convention.
Conversely, when one extracts AFNI preprocessed time-series data with 3dmaskdump without adding the option “-xyz”, the extracted voxel coordinates by 3dmaskdump are in the neurological (ijk) convention, and not in the radiological (xyz) convention, correct?

Hence, there is a mismatch between ijk and xyz, or between the functional data extracted by 3dmaskdump and the network coordinates that I extracted with with using -xyz. Thats why adding y = y-1 and all those fuzzy steps were required in Python.


Now, when I no longer applied “xyz” in the 3dmaskdump script where I extracted the voxel coordinates from the single 7 networks plus the combination of all 7 sevenworks (labeled “global”), no mismatch, warning messages in 3dUndump, whatsoever occur anymore. Also, I do not have to mirror the x, y, or z axis coordinates in Python.

Furthermore, checking all kinds of files with


3dinfo \
-same_all_grid \
-prefix \
ACW_Awake_Global.nii \ # the new NIFTI file created by 3dUndump
Global.nii # the combination of the 7 individual networks, where the 7 individual networks were combined using 3dcalc

yields no mismatch. Also, a visual inspection of both files (ACW_Awake_Global.nii vs. Global.nii) in AFNI shows that all voxels are 100% aligned in space, that no voxels diverge between the two nifti files, and that everything seems to be exactly in place.

Even though I sort of created those files using a inconvenient way, because I extracted the voxel coordinates from the network files (such as from “Global.nii”), the error or mistake simply was that I initially added the option “-xyz” precisely in this step. And this messed everything up.

Please let me know what you think, even though I understand this thread is hard to follow now, because it all was a big fuzzy mess.

Hi, Philipp-

“ijk” refers to integer-based indices for locating oneself in an ordered collection, like a list/array. These numbers are have no units, and go from 0,…, N-1, where “N” is the size of the collection along the given dimension. These are independent of voxel size.

“xyz” refers to physical coordinates of the dset, each of which has units of millimeters, depends on voxel sizes, and are generally floating point valued. In one of my least favorite aspects of neuroimaging data processing, there are maaaany conventions for what signs mean on these: “-4” in the x-axis might mean left or right, depending on convention, and many people still don’t report these correctly in papers. Sigh. Life is hard.

In general, I would not try to mix these two conventions: equality/matching with floating point numbers is not a fun task numerically. Different datasets that technically “match” or are on the same grid might have different rounding errors in floating point coordinates, for example. The dataset orientation can play a confusing role, as well, such as moving between RAI-DICOM or LPI-SPM conventions. That is why I recommended verifying that dsets that were going to have information merged in Python should be checked for matching first, and then just use the coordinates of one dset. The header information being correct is key, and Python won’t really deal with that header info—it is a recipe for confusion, instability and possibly disaster.

–pt

Paul,

thanks you for the detailed answer, this is much appreciated. A final question.
Say I am interested in extracting AFNI functional data that already has 3x3x3 mm voxels using 3dmaskdump in the neurological convention.
In this case, I would add the option


-nbox 3 3 3 

to the 3dmaskdump script, correct? The extracted data would then correspond to the neurological or LPI-SPM convention, unless, of course, I would mess up something later in Python, But at least from the AFNI side of things, all should be clear.

Hi Philipp,

The -xbox/-dbox/-nbox options take x y z coordinates, which without ranges have nothing to do with the voxel sizes.

However, if you want the -xbox to be a range of voxels, let’s say of size +/- 3mm, and centered at coordinate 10,20,30, for example (to include 27 voxels around that central point), you could use ranges of -/+ 3mm around each central coordinate. I will leave the -xyz option in there to display your specific coordinates.

3dmaskdump -xyz -xbox 7:13 17:23 27:33 DATASET

However, if 10, 20, 30 is not exactly at the center of a voxel, this will not output 27 voxels. And often the centers are not even exact in base 2 decimal, so you might have to add epsilons, e.g.

3dmaskdump -xyz -xbox 6.9:13.1 16.9:23.1 26.9:33.1 DATASET

Note that the epsilons can be generous, until they overlap with the next voxel center.

  • rick