Dear experts,
I used the Draw_ROI_Plugin to draw 13 independent ROIs, and then create a region-of-interest mask of 13 different values(1 to 13). Before using this mask for further analysis, i would like to seperate each ROI to double check it.
for example, I want to seperate the ROI whose value is 4, using the following code
“3dcalc -a ROI13_Mask.nii.gz -expr ‘equals(a,4)’ -prefix ROI13_4.nii.gz”
Confusingly, I can view the ROI13_Mask in AFNI, and can see different values related to different ROIs. However, when it turns to a single region (the ROI13_4.nii.gz), there is something wrong when I try to view it in AFNI .
++WARNING: nifti_read_buffer(path to file):
data bytes needed = 311296
data bytes input = 311280
number missing = 16 (set to 0)
** NIFTI load bricks: cannot read brick 0 from ‘path to file’
Can anyone else help me solve it or give me some suggestions, I have been trapped for a long time.
How is the initial NIFTI dset created (e.g. do you specify ROI13_Mask.nii.gz with the SaveAs button in the drawing plugin)?
Are any other operations performed in the between creation and the 3dcalc command?
These 13 ROIs were grouped into the original mask by 3dcalc “a1+b2+…m*13”. However, this original mask didn’t fill in the brain well, leaving much empty space especially in the marginal area, so I chose to use the Draw Plungin to fill it——“Choose dataset for copying”- “Save” botton, get a COPY file, and then run the following code:
“3dcalc -a COPY_WFU_Frontal_Sup_L_3mm.nii.gz -b WFU_Frontal_Sup_L_3mm.nii.gz -expr ‘a+b’ -prefix aaa_Frontal_Sup_L.nii.gz
3dcalc -a aaa_Frontal_Sup_L.nii.gz -b SB92_noBrainstemCereb.nii.gz -expr ‘a*b’ -prefix Frontal_Sup_L_Edited1.nii.gz”
For each ROI, I repeated this process several times to get a dream one, which filled the border completely.
So far, no problem arose. Each ROI can be viewed in the AFNI. However, when I used the previous method, 3dcalc “a1+b2+…m13" ,to make up a new mask. This mask also couldn’t be viewed in the AFNI, and had the same “data bytes needed” error. Something wrong must had happened at this time. But I ignored it and found another way to avoide this error. Specifically, I calculated 3-4 ROIs each time (e.g. 3dcalc "a1+b2+c3”), and then added them together. By this way, I got the ROI13_Mask.nii.gz.
I don’t know what happen here, but I can view the ROI13_Mask.nii.gz in the AFNI, and the value related with each ROI is correct. When I tried to separate each ROI, that error arose again, as I had mentioned before.
-Here is the output of “nifti_tool -disp_nim -infiles ROI13_4.nii.gz”
Besides, I also ran the “nifti_tool -disp_nim -infiles ROI13_Mask.nii.gz”. The output of it is almost the same, except that the valve of iname_offset is 5568464 and the esize of ext_list is 5568112.
Thanks for the update, Wind. And sorry that I did not get back to this earlier. I was going to suggest “nifti_tool -rm_ext all” on the NIFTI files, but wanted to see what was happening, first.
Following up on the nifti_read_buffer error. I am getting it sporadically when I run multiple processes in parallel. Rerunning the same command (one process) works fine. I wonder if this is an OS limitation.
A sample output is shown below, the ellipses in the path are mine to make the output more readable:
cheers,
Z
++ 3dcopy: AFNI version=AFNI_22.1.14 (Jun 24 2022) [64-bit]
++ WARNING: nifti_read_buffer(/…/ADNIpet/NIFTI/035_S_7021/Baseline.tauPET.I1547562.2.2022-02-17.nii.gz):
data bytes needed = 49222656
data bytes input = -1
number missing = 49222657 (set to 0)
** NIFTI load bricks: cannot read brick 2 from ‘/…/ADNIpet/NIFTI/035_S_7021/Baseline.tauPET.I1547562.2.2022-02-17.nii.gz’
e[7m** FATAL ERROR:e[0m Can’t load dataset ‘/…/ADNIpet/NIFTI/035_S_7021/Baseline.tauPET.I1547562.2.2022-02-17.nii.gz’: is it complete?
** Program compile date = Jun 24 2022
It looks like it’s not reading that dataset at all. That might mean the OS can’t find it because it hasn’t finished being written yet. Consider adding a small delay before this 3dcopy command. Another approach is to make the reading and writing faster. pigz is the parallelized version of gzip that might help. AFNI format and NIFTI format work differently, so an AFNI format solution with .BRIK.gz might not show this problem. Chris Rorden came up with this improvement to the znzlib library used by NIFTI writing.
I actually just opened up a similar topic, and intriguingly this does appear to be OS or machine-dependent behavior: I’m seeing the error on biowulf but not on MacOS.
I am intrigued by the delay suggestion and it got me wondering whether it is possible to trap for this error within AFNI’s I/O and have the operation sleep for a bit before trying again, but only when needed. I suspect this problem occurs more often when using network file systems and when running parallel processes with file I/O. Setting a delay at the script level would slow everything down to fix a few occurrences.
I don’t mean to be piling on the team’s plate, but some guidance on how to best fix this might help the next intrepid developer to implement it.
I think I’ve seen some mysterious issues like this on high-use network drives, and some sleep never hurt anyone. I suppose a programmatic pre-read sleep might help or a post-write sleep. What program wrote that dataset that becomes the input for the 3dcopy? You could wrap 3dcopy in a script to include a delay before and after, maybe making the amount of sleep different if the input and output is compressed. That might be a good test to see if this is actually useful. You could also try turning off compression completely when running this on a server by unsetting AFNI_COMPRESSOR and setting AFNI_AUTOGZIP to NO.
I am afraid the sleep option is not practical for me – not sure where to put it as the problem occurs at multiple spots. I will need to dig deeper to find out how to replicate the problem with regularity.
To be clear, all the read-write operations carried out by each process are sequential. Think a large number of scripts running on each subject separately and outputs are in separate directories. For the timing to help, it would have to be that afni programs return before the write operation is complete. Does this occur when using compression?
AFNI format datasets are opened and closed with popen and pclose, but NIFTI format datasets are handled with the znzlib, which uses the zlib function gzopen, gzclose and a variety of gz… functions for writing. I suspect there is some odd buffering problem on the distributed file system that might be handled with a combination of flush and sleeps before and after gzclose. That’s all in one file, znzlib.c, or in nifti2_io.c, so a rebuild with the small changes and tests might reveal something. Also trying with the AFNI format instead or with Chris R’s modifications might work.
Just curious if there are some network properties
that we could also use to debug things:
You’re using MacOS?
YES
What protocol is the network share? NFS? SMB?
SMB
AFS?
How big are the files?
~ 5-10 Mb
Can you try unsetting AFNI_COMPRESSOR and
setting AFNI_AUTOGZIP to NO?
Unfortunately this will take some work because multiple scripts are written to expect nii.gz. I will try it out when my bandwidth allows
I will try to debug with your suggestions in mind next time I have a chance to reproduce the problem. As I mentioned to Peter, there are a lot of scripts being invoked and they all assume nii.gz output so modifications for compression options will take time to implement and test.
Let’s hold off on digging deeper until either I or others who encounter this problem can provide more hints.
All the best,
Ziad
Daniel Glen Wrote:
AFNI format datasets are opened and closed with
popen and pclose, but NIFTI format datasets are
handled with the znzlib, which uses the zlib
function gzopen, gzclose and a variety of gz…
functions for writing. I suspect there is some odd
buffering problem on the distributed file system
that might be handled with a combination of flush
and sleeps before and after gzclose. That’s all in
one file, znzlib.c, or in nifti2_io.c, so a
rebuild with the small changes and tests might
reveal something. Also trying with the AFNI format
instead or with Chris R’s modifications might
work.
The error has reared up again. This time on a 3dTstat command in a different dataset and without scripts running in parallel.
For whatever it is worth: The numbers below ++ WARNING are identical to the previously reported error. I suspect they reflect i/o buffer size and initialization.
All the best,
Ziad
++ 3dTstat: AFNI version=AFNI_22.1.14 (Jun 24 2022) [64-bit]
++ Authored by: KR Hammett & RW Cox
++ WARNING: nifti_read_buffer(…/Proc/011_S_4105/nalv.060922_1212/V1.tsal.mid.nii.gz):
data bytes needed = 49222656
data bytes input = -1
number missing = 49222657 (set to 0)
** NIFTI load bricks: cannot read brick 3 from ‘…/Proc/011_S_4105/nalv.060922_1212/V1.tsal.mid.nii.gz’
e[7m** FATAL ERROR:e[0m Unable to compute output dataset!
** Program compile date = Jun 24 2022
Since this is smb, try adjusting the buffer size in smb.conf. This was still in a single script that did things sequentially, and 3dTstat was one of the steps?
If you can reproduce this with just a single script, that’s great. The problem may be intermittent based on network traffic and file server activity. I would try both increases and decreases in the samba cache. This was from a quick google search :
The
National Institute of Mental Health (NIMH) is part of the National Institutes of
Health (NIH), a component of the U.S. Department of Health and Human
Services.