"data bytes needed" error

AFNI Message Board
1

Dear experts,
I used the Draw_ROI_Plugin to draw 13 independent ROIs, and then create a region-of-interest mask of 13 different values(1 to 13). Before using this mask for further analysis, i would like to seperate each ROI to double check it.

for example, I want to seperate the ROI whose value is 4, using the following code
“3dcalc -a ROI13_Mask.nii.gz -expr ‘equals(a,4)’ -prefix ROI13_4.nii.gz”

Confusingly, I can view the ROI13_Mask in AFNI, and can see different values related to different ROIs. However, when it turns to a single region (the ROI13_4.nii.gz), there is something wrong when I try to view it in AFNI .

++WARNING: nifti_read_buffer(path to file):
data bytes needed = 311296
data bytes input = 311280
number missing = 16 (set to 0)
** NIFTI load bricks: cannot read brick 0 from ‘path to file’

Can anyone else help me solve it or give me some suggestions, I have been trapped for a long time.

Thanks very much

Wind

2

Hi Wind,

That is very peculiar!

How is the initial NIFTI dset created (e.g. do you specify ROI13_Mask.nii.gz with the SaveAs button in the drawing plugin)?
Are any other operations performed in the between creation and the 3dcalc command?

What is the output of:

nifti_tool -disp_nim -infiles ROI13_4.nii.gz
  • rick
3

I am extremely grateful for your replying!

To make it clear for you, I would like to outline the process. Hope this is helpful.

The whole process is made up of following steps:

  1. Download 13 different ROIs in matlab-WFU Human atlas-AAL (e.g. Frontal_Sup_L.nii), and then, for each ROI run the following code

"3dcalc -prefix WFU_Frontal_Sup_L.nii.gz -a Frontal_Sup_L.nii -expr 'step(a-0)'
3dresample -master SB92_noBrainstemCereb.nii.gz -dxyz 3 3 3 -prefix WFU_Frontal_Sup_L_3mm.nii.gz -input WFU_Frontal_Sup_L.nii.gz
3drefit -space MNI WFU_Frontal_Sup_L_3mm.nii.gz"

  1. These 13 ROIs were grouped into the original mask by 3dcalc "a1+b2+...m*13". However, this original mask didn't fill in the brain well, leaving much empty space especially in the marginal area, so I chose to use the Draw Plungin to fill it——"Choose dataset for copying"- "Save" botton, get a COPY file, and then run the following code:

"3dcalc -a COPY_WFU_Frontal_Sup_L_3mm.nii.gz -b WFU_Frontal_Sup_L_3mm.nii.gz -expr 'a+b' -prefix aaa_Frontal_Sup_L.nii.gz
3dcalc -a aaa_Frontal_Sup_L.nii.gz -b SB92_noBrainstemCereb.nii.gz -expr 'a*b' -prefix Frontal_Sup_L_Edited1.nii.gz"

For each ROI, I repeated this process several times to get a dream one, which filled the border completely.

  1. So far, no problem arose. Each ROI can be viewed in the AFNI. However, when I used the previous method, 3dcalc "a1+b2+...m13" ,to make up a new mask. This mask also couldn't be viewed in the AFNI, and had the same "data bytes needed" error. Something wrong must had happened at this time. But I ignored it and found another way to avoide this error. Specifically, I calculated 3-4 ROIs each time (e.g. 3dcalc "a1+b2+c3"), and then added them together. By this way, I got the ROI13_Mask.nii.gz.

I don't know what happen here, but I can view the ROI13_Mask.nii.gz in the AFNI, and the value related with each ROI is correct. When I tried to separate each ROI, that error arose again, as I had mentioned before.

-Here is the output of "nifti_tool -disp_nim -infiles ROI13_4.nii.gz"

----------------------------------------------------------------------------------------------------------------------
header file 'ROI13_4.nii.gz', num_fields = 63, fields:

all fields:
  name                offset  nvals  values
  ------------------- ------  -----  ------
  ndim                   0      1    3
  nx                     8      1    64
  ny                    16      1    76
  nz                    24      1    64
  nt                    32      1    1
  nu                    40      1    1
  nv                    48      1    1
  nw                    56      1    1
  dim                   64      8    3 64 76 64 1 1 1 1
  nvox                 128      1    311296
  nbyper               136      1    1
  datatype             140      1    2
  dx                   144      1    3.0
  dy                   152      1    3.0
  dz                   160      1    3.0
  dt                   168      1    0.0
  du                   176      1    0.0
  dv                   184      1    0.0
  dw                   192      1    0.0
  pixdim               200      8    0.0 3.0 3.0 3.0 0.0 0.0 0.0 0.0
  scl_slope            264      1    0.0
  scl_inter            272      1    0.0
  cal_min              280      1    0.0
  cal_max              288      1    0.0
  qform_code           296      1    4
  sform_code           300      1    4
  freq_dim             304      1    0
  phase_dim            308      1    0
  slice_dim            312      1    0
  slice_code           316      1    0
  slice_start          320      1    0
  slice_end            328      1    0
  slice_duration       336      1    0.0
  quatern_b            344      1    0.0
  quatern_c            352      1    -0.0
  quatern_d            360      1    0.0
  qoffset_x            368      1    -94.5
  qoffset_y            376      1    -130.5
  qoffset_z            384      1    -76.5
  qfac                 392      1    1.0
  qto_xyz              400     16    3.0 -0.0 0.0 -94.5 0.0 3.0 -0.0 -130.5 0.0 0.0 3.0 -76.5 0.0 0.0 0.0 1.0
  qto_ijk              528     16    0.333333 0.0 0.0 31.5 -0.0 0.333333 0.0 43.5 0.0 -0.0 0.333333 25.5 0.0 0.0 0.0 1.0
  sto_xyz              656     16    3.0 -0.0 -0.0 -94.5 -0.0 3.0 -0.0 -130.5 0.0 0.0 3.0 -76.5 0.0 0.0 0.0 1.0
  sto_ijk              784     16    0.333333 0.0 0.0 31.5 0.0 0.333333 0.0 43.5 -0.0 -0.0 0.333333 25.5 0.0 0.0 0.0 1.0
  toffset              912      1    0.0
  xyz_units            920      1    2
  time_units           924      1    0
  nifti_type           928      1    1
  intent_code          932      1    0
  intent_p1            936      1    0.0
  intent_p2            944      1    0.0
  intent_p3            952      1    0.0
  intent_name          960     16    
  descrip              976     80    
  aux_file            1056     24    
  fname               1080      1    'ROI13_4.nii.gz'
  iname               1088      1    'ROI13_4.nii.gz'
  iname_offset        1096      1    607359616
  swapsize            1104      1    0
  byteorder           1108      1    1
  data                1112      1    (raw data of unknown type)
  num_ext             1120      1    1
  ext_list            1128      1    ecode = 4, esize = 607359248, edata = <?xml
------------------------------------------------------------------------------------------------------------------

Besides, I also ran the "nifti_tool -disp_nim -infiles ROI13_Mask.nii.gz". The output of it is almost the same, except that the valve of iname_offset is 5568464 and the esize of ext_list is 5568112.

Thank you again very much.

Wind

4

Thanks so much!

This problem had been solved by the command “3drefit -denote ROI.nii.gz”, opreating on each ROI.

Best wishes!

Wind

5

Thanks for the update, Wind. And sorry that I did not get back to this earlier. I was going to suggest “nifti_tool -rm_ext all” on the NIFTI files, but wanted to see what was happening, first.

Anyway, I am glad you figured it out.

Thanks,

  • rick
6

Dear friends,

Following up on the nifti_read_buffer error. I am getting it sporadically when I run multiple processes in parallel. Rerunning the same command (one process) works fine. I wonder if this is an OS limitation.

A sample output is shown below, the ellipses in the path are mine to make the output more readable:

cheers,
Z

++ 3dcopy: AFNI version=AFNI_22.1.14 (Jun 24 2022) [64-bit]
++ WARNING: nifti_read_buffer(/…/ADNIpet/NIFTI/035_S_7021/Baseline.tauPET.I1547562.2.2022-02-17.nii.gz):
data bytes needed = 49222656
data bytes input = -1
number missing = 49222657 (set to 0)
** NIFTI load bricks: cannot read brick 2 from ‘/…/ADNIpet/NIFTI/035_S_7021/Baseline.tauPET.I1547562.2.2022-02-17.nii.gz’
e[7m** FATAL ERROR:e[0m Can’t load dataset ‘/…/ADNIpet/NIFTI/035_S_7021/Baseline.tauPET.I1547562.2.2022-02-17.nii.gz’: is it complete?
** Program compile date = Jun 24 2022

7

It looks like it’s not reading that dataset at all. That might mean the OS can’t find it because it hasn’t finished being written yet. Consider adding a small delay before this 3dcopy command. Another approach is to make the reading and writing faster. pigz is the parallelized version of gzip that might help. AFNI format and NIFTI format work differently, so an AFNI format solution with .BRIK.gz might not show this problem. Chris Rorden came up with this improvement to the znzlib library used by NIFTI writing.

https://github.com/neurolabusc/znzlib

8

I actually just opened up a similar topic, and intriguingly this does appear to be OS or machine-dependent behavior: I’m seeing the error on biowulf but not on MacOS.

9

Thank you, Daniel.

I will try pigz and see if things get better.

I am intrigued by the delay suggestion and it got me wondering whether it is possible to trap for this error within AFNI’s I/O and have the operation sleep for a bit before trying again, but only when needed. I suspect this problem occurs more often when using network file systems and when running parallel processes with file I/O. Setting a delay at the script level would slow everything down to fix a few occurrences.
I don’t mean to be piling on the team’s plate, but some guidance on how to best fix this might help the next intrepid developer to implement it.

Cheers,
Z

10

I think I’ve seen some mysterious issues like this on high-use network drives, and some sleep never hurt anyone. I suppose a programmatic pre-read sleep might help or a post-write sleep. What program wrote that dataset that becomes the input for the 3dcopy? You could wrap 3dcopy in a script to include a delay before and after, maybe making the amount of sleep different if the input and output is compressed. That might be a good test to see if this is actually useful. You could also try turning off compression completely when running this on a server by unsetting AFNI_COMPRESSOR and setting AFNI_AUTOGZIP to NO.

11

Thank you, Daniel.

I am afraid the sleep option is not practical for me – not sure where to put it as the problem occurs at multiple spots. I will need to dig deeper to find out how to replicate the problem with regularity.
To be clear, all the read-write operations carried out by each process are sequential. Think a large number of scripts running on each subject separately and outputs are in separate directories. For the timing to help, it would have to be that afni programs return before the write operation is complete. Does this occur when using compression?

All the best,
Ziad

12

Just curious if there are some network properties that we could also use to debug things:

  1. You’re using MacOS?
  2. What protocol is the network share? NFS? SMB? AFS?
  3. How big are the files?
  4. Can you try unsetting AFNI_COMPRESSOR and setting AFNI_AUTOGZIP to NO?
13

AFNI format datasets are opened and closed with popen and pclose, but NIFTI format datasets are handled with the znzlib, which uses the zlib function gzopen, gzclose and a variety of gz… functions for writing. I suspect there is some odd buffering problem on the distributed file system that might be handled with a combination of flush and sleeps before and after gzclose. That’s all in one file, znzlib.c, or in nifti2_io.c, so a rebuild with the small changes and tests might reveal something. Also trying with the AFNI format instead or with Chris R’s modifications might work.

14

Hi Peter!

Responses inline below.

All the best,
Ziad

Peter Molfese Wrote:

Just curious if there are some network properties
that we could also use to debug things:

  1. You're using MacOS?
    YES
  2. What protocol is the network share? NFS? SMB?
    SMB
    AFS?
  3. How big are the files?
    ~ 5-10 Mb
  4. Can you try unsetting AFNI_COMPRESSOR and
    setting AFNI_AUTOGZIP to NO?
    Unfortunately this will take some work because multiple scripts are written to expect nii.gz. I will try it out when my bandwidth allows
15

Thank you, Daniel.

I will try to debug with your suggestions in mind next time I have a chance to reproduce the problem. As I mentioned to Peter, there are a lot of scripts being invoked and they all assume nii.gz output so modifications for compression options will take time to implement and test.

Let's hold off on digging deeper until either I or others who encounter this problem can provide more hints.

All the best,
Ziad

Daniel Glen Wrote:

AFNI format datasets are opened and closed with
popen and pclose, but NIFTI format datasets are
handled with the znzlib, which uses the zlib
function gzopen, gzclose and a variety of gz...
functions for writing. I suspect there is some odd
buffering problem on the distributed file system
that might be handled with a combination of flush
and sleeps before and after gzclose. That's all in
one file, znzlib.c, or in nifti2_io.c, so a
rebuild with the small changes and tests might
reveal something. Also trying with the AFNI format
instead or with Chris R's modifications might
work.

16

I spoke too soon.

The error has reared up again. This time on a 3dTstat command in a different dataset and without scripts running in parallel.

For whatever it is worth: The numbers below ++ WARNING are identical to the previously reported error. I suspect they reflect i/o buffer size and initialization.

All the best,
Ziad

++ 3dTstat: AFNI version=AFNI_22.1.14 (Jun 24 2022) [64-bit]
++ Authored by: KR Hammett & RW Cox
++ WARNING: nifti_read_buffer(…/Proc/011_S_4105/nalv.060922_1212/V1.tsal.mid.nii.gz):
data bytes needed = 49222656
data bytes input = -1
number missing = 49222657 (set to 0)
** NIFTI load bricks: cannot read brick 3 from ‘…/Proc/011_S_4105/nalv.060922_1212/V1.tsal.mid.nii.gz’
e[7m** FATAL ERROR:e[0m Unable to compute output dataset!
** Program compile date = Jun 24 2022

17

Since this is smb, try adjusting the buffer size in smb.conf. This was still in a single script that did things sequentially, and 3dTstat was one of the steps?

18

Hi Daniel,

In line below:

Cheers,
Z

Daniel Glen Wrote:

Since this is smb, try adjusting the buffer size
in smb.conf.
Any tips for how that is done and whether I should adjust up or down?

This was still in a single script
that did things sequentially, and 3dTstat was one
of the steps?
Correct.

19

If you can reproduce this with just a single script, that's great. The problem may be intermittent based on network traffic and file server activity. I would try both increases and decreases in the samba cache. This was from a quick google search :

Cache data before flushing to disk:

write cache size = 2097152
min receivefile size = 16384
getwd cache = true

https://coderwall.com/p/2ufa0g/fix-samba-read-and-write-performance-issues

20

Hi @dglen @rickr ,

I faced this issue today, and it looks like the file is read before its written.

++ WARNING: nifti_read_buffer():
data bytes needed = 771680
data bytes input = -1
number missing = 771681 (set to 0)

How do I do a delay in the script ?
can I just use a "sleep 1s" after the file is written to prevent this error ?