Potential Bug 3dQwarp and Red Hat Enterprise Linux Server release 7.3 (Maipo)

AFNI Message Board
1

Hi Afni Experts,
I wanted to touch base as I encountered an error using 3dQwarp (via auto_warp.py) that seems specific to Red Hat. The cluster runs Red Hat 7.3 and I tried both AFNI Precompiled binary linux_openmp_64: Jun 23 2017 (Version AFNI_17.1.12) and Precompiled binary linux_openmp_64: Apr 12 2017 (Version AFNI_17.1.01). I had AFNI installed by IT (root access) using the directions (https://afni.nimh.nih.gov/pub/dist/doc/htmldoc/background_install/install_instructs/steps_linux_Fed_RH.html) and I installed a separate local copy with the same error.

When running auto_warp.py : auto_warp.py -base tpl.nii.gz -input T1.brain2tpl.nii.gz -warp_dxyz 1 -qw_opts -penfac .01 -minpatch 13 -allinfast -Qfinal
I received the following error:
++ +++++++++++ Begin warp optimization: base=./base.rw.nii source=./anat.rw.qw_Allin.nii
++ AFNI warpomatic: 193 x 232 x 208 volume ; autobbox = 20…172 21…210 16…181 [clock= 31s 114ms]
lev=0 1…191 0…231 0…201: [first cost=-0.66608] … done [cost:-0.66608==>-0.81349]
lev=1 patch=145x175x157 [clock= 7m 27s 972ms]…:[cost=-0.84611]:… done [cost:-0.81349==>-0.85062 ; 16 patches optimized, 0 skipped]
lev=2 patch=109x131x117 [clock= 9m 37s 771ms]… done [cost:-0.85062==>-0.92735 ; 23 patches optimized, 4 skipped]
lev=3 patch=81x99x89 [clock= 11m 17s 130ms]… done [cost:-0.92735==>-1.00095 ; 38 patches optimized, 26 skipped]
lev=4 patch=61x73x67 [clock= 12m 31s 953ms]… done [cost:-1.00095==>-1.08824 ; 75 patches optimized, 50 skipped]
lev=5 patch=47x55x49 [clock= 13m 36s 968ms]… done [cost:-1.08824==>-1.19102 ; 192 patches optimized, 151 skipped]
lev=6 patch=35x41x37 [clock= 14m 44s 912ms]… done [cost:-1.19102==>-1.30727 ; 456 patches optimized, 354 skipped]
lev=7 patch=25x31x27 [clock= 15m 58s 739ms]… done [cost:-1.30727==>-1.43121 ; 1165 patches optimized, 1032 skipped]
lev=8 patch=19x23x21 [clock= 17m 22s 849ms]… done [cost:-1.43121==>-1.55941 ; 2563 patches optimized, 2350 skipped]
lev=9 patch=13x13x13 [clock= 19m 13s 740ms]…#**ERROR Failed in warping step
** ERROR - script failed

The only time I DO NOT receive an error is if I put the minpatch is set > 25. I used afni_system_check -check_all and did not see anything obviously missing.

I tested the exact same software (copied and installed on Ubuntu 14.04) and subject on a local cluster using Ubuntu and did not have any issues when using minpatch of 19 or less. I wanted to see if there are any reported issues that are related to red hat / 3dQwarp , or libraries from the install instructions (as IT installed with root access for me) that could explain this error.

Thanks,
Ajay

2

It is very hard to say, with the limited error message output by auto_warp.py. We can’t test this run here, since we don’t have a similar system. One possibility is that the program ran out of memory and was killed by Linux (which is nearly silent when it does something so terrible). This situation might be possible if the two different computers have different amounts of RAM. It did get down to patch=19x23x21, indicating that minpatch 19 might work.

You can try adding the options
-verb -qw_opts -verb
to the auto_warp.py command, to get more messages output by the program while it runs.

3

Hi Bob,
Just as a follow up, even though the login and compute nodes had the same specs, it seems that the login node limited ram per user to 5GB and failed while the compute nodes I would use the specified 24gb and it worked perfectly. So looks like a memory issue after all.

Thanks,
Ajay

4

Thanks for the follow up, Ajay.

For the record, there is now a CentOS 7 package available,
though I don’t know if that would be more useful than openmp_64
in any way…

But it is at pub/dist/tgz/linux_centos_7_64.tgz if you want it.

  • rick
5

Hi Rick,
I wanted to find out if the Centos 7 version is also openmp or not?

Secondly, in running 3dNwarpApply I had the following errors on the red hat system when the exact same command/files passed on my ubuntu system…any thoughts?

3dNwarpApply -interp NN -nwarp ‘tmp.anat2MNI_WARP.nii tmp.anat2MNI.aff.Xat.1D anat.brain2tpl.aff6.1D INV(T2.brain2B0.aff6.1D) nl_WARP.nii’ -iwarp -master B0ref_resamp.nii.gz -source ./DWI.weight+tlrc. -prefix weight22 -verb >> 3dNwarpApply_err1.txt
++ 3dNwarpApply: AFNI version=AFNI_17.1.01 (Apr 12 2017) [64-bit]
++ Authored by: Zhark the Warped
++ -master dataset is ‘B0ref_resamp.nii.gz’
++ opened source dataset ‘./DWI.weight+tlrc.’
++ Processing -nwarp ++ Enter IW3D_read_nwarp_catlist( tmp.anat2MNI_WARP.nii tmp.anat2MNI.aff.Xat.1D anat.brain2tpl.aff6.1D INV(T2.brain2B0.aff6.1D) nl_WARP.nii )

  • Open dataset warp tmp.anat2MNI_WARP.nii
  • max displacments = 20.1844 14.3053 11.4142
  • Open matrix file tmp.anat2MNI.aff.Xat.1D
  • max shifts = 0.551628 2.81892 5.4792
  • Open matrix file anat.brain2tpl.aff6.1D
  • max shifts = 0.420547 2.80645 14.4052
  • Open matrix file INV(T2.brain2B0.aff6.1D)
  • max shifts = 2.08339 11.8799 32.1879
  • Open dataset warp nl_WARP.nii
  • max displacments = 5.69117 10.5456 4.94413
  • found geometry mismatch: orig=MATRIX(-1,0,0,96,0,-1,0,132,0,0,1,-93):193,231,208 new=MATRIX(1,0,0,-117.0035,0,1,0,-120.5573,0,0,1,-62.79667):230,236,173
  • — Totalized max displacments = 28.9311 42.3562 68.4307

++ IW3D_set_geometry_nwarp_catlist: padding = 46

  • results: master_geomstring = MATRIX(1.5,0,0,-116.5035,0,1.5,0,-120.0573,0,0,1.5,-46.54667):153,153,96 actual_geomstring = MATRIX(1.5,0,0,-185.5035,0,1.5,0,-189.0573,0,0,1.5,-115.5467):245,245,188
  • – warp #0 geometry = MATRIX(-1,0,0,96,0,-1,0,132,0,0,1,-93):193,231,208
  • regridding warp #0 to geometry = MATRIX(1.5,0,0,-185.5035,0,1.5,0,-189.0573,0,0,1.5,-115.5467):245,245,188
  • – warp #4 geometry = MATRIX(1,0,0,-117.0035,0,1,0,-120.5573,0,0,1,-62.79667):230,236,173
  • regridding warp #4 to geometry = MATRIX(1.5,0,0,-185.5035,0,1.5,0,-189.0573,0,0,1.5,-115.5467):245,245,188
    +++++ initial structure of Nwarp_catlist: Nwarp ; Matrix[1] ; Matrix[1] ; Matrix[1] ; Nwarp ;
    ++ – IW3D_reduce_nwarp_catlist – start loop
  • looking at nwc[0]
  • … #0 and #1 are neighbors
  • looking at nwc[1]
  • … #1 and #2 are neighbors
  • … IW3D_reduce_nwarp_catlist: Matrix[1]-Matrix[1] ii=1 jj=2
  • looking at nwc[2]
  • … #2 and #3 are neighbors
  • … IW3D_reduce_nwarp_catlist: Matrix[1]-Matrix[1] ii=2 jj=3
  • looking at nwc[3]
  • … #3 and #4 are neighbors
  • – loop reduction operation count = 2
    ++ – IW3D_reduce_nwarp_catlist – start loop
  • looking at nwc[0]
  • … #0 and #3 are neighbors
  • IW3D_reduce_nwarp_catlist: warp-Matrix[1] ii=0 jj=3
  • looking at nwc[3]
  • … #3 and #4 are neighbors
  • … IW3D_reduce_nwarp_catlist: warp-warp ii=3 jj=4
  • ii=3: geometry = MATRIX(1.5,0,0,-185.5035,0,1.5,0,-189.0573,0,0,1.5,-115.5467):245,245,188
  • jj=4: geometry = MATRIX(1.5,0,0,-185.5035,0,1.5,0,-189.0573,0,0,1.5,-115.5467):245,245,188
  • – loop reduction operation count = 2
    ++ – IW3D_reduce_nwarp_catlist – start loop
  • nwc[0] is doubly NULL
  • nwc[1] is doubly NULL
  • nwc[2] is doubly NULL
  • nwc[3] is doubly NULL
  • – loop reduction operation count = 0
    +++++ final structure of Nwarp_catlist: (NULL) ; (NULL) ; Nwarp ;
    ++ Warping: [nvar=1]ab{nwarp_catlist[0] [NULL entry?] [NULL entry?] [warp]} + – invert max|AA|=64.153511
    • init nstep=8 qq=1/2^8=0.003906
    • init step 1
    • init step 2
    • init step 3
    • init step 4
    • init step 5
    • init step 6
    • init step 7
    • init step 8
    • start iterations: normAA=36.756603 inewtfac=0.248055
    • iterate 1 nrat=0.027847
    • iterate 2 nrat=0.019870
    • switch to inewtfac=0.306100
    • iterate 3 nrat=0.019028
    • iterate 4 nrat=0.013255
    • switch to inewtfac=0.377727
    • iterate 5 nrat=0.010855
    • iterate 6 nrat=0.007080
    • switch to inewtfac=0.466116
    • iterate 7 nrat=0.005142
    • iterate 8 nrat=0.002772
    • switch to inewtfac=0.575187
    • iterate 9 nrat=0.001922
    • iterate 10 nrat=0.000805
    • switching from linear interp
    • iterate 11 nrat=0.000343
    • iterate 12 nrat=0.000151
    • switch to inewtfac=0.678901
    • iterate 13 nrat=0.000076
  • – iteration converged
    '** ERROR: EDIT_full_copy: invalid dataset input ptr=0x6e92a00
    ** ERROR: Can’t copy nwarp dataset #0 ?!?
    ** FATAL ERROR: Warping failed for some reason :-(((
    ** Program compile date = Apr 12 2017

Output of ldd on redhat …just as a note for linux-vdso.so.1 this is also blank on the ubuntu machine that worked.
linux-vdso.so.1 => (0x00007ffc98ff8000)
libXt.so.6 => /usr/lib64/libXt.so.6 (0x00002b2e2c23d000)
libXft.so.2 => /usr/lib64/libXft.so.2 (0x00002b2e2c4a4000)
libXp.so.6 => /usr/lib64/libXp.so.6 (0x00002b2e2c6ba000)
libXpm.so.4 => /usr/lib64/libXpm.so.4 (0x00002b2e2c8c3000)
libXext.so.6 => /usr/lib64/libXext.so.6 (0x00002b2e2cad5000)
libXmu.so.6 => /usr/lib64/libXmu.so.6 (0x00002b2e2cce7000)
libSM.so.6 => /usr/lib64/libSM.so.6 (0x00002b2e2cf02000)
libICE.so.6 => /usr/lib64/libICE.so.6 (0x00002b2e2d10a000)
libX11.so.6 => /usr/lib64/libX11.so.6 (0x00002b2e2d326000)
libpng12.so.0 => /usr/lib64/libpng12.so.0 (0x00002b2e2d664000)
libz.so.1 => /usr/lib64/libz.so.1 (0x00002b2e2d88b000)
libm.so.6 => /usr/lib64/libm.so.6 (0x00002b2e2daa1000)
libdl.so.2 => /usr/lib64/libdl.so.2 (0x00002b2e2dda3000)
libc.so.6 => /usr/lib64/libc.so.6 (0x00002b2e2dfa7000)
libgomp.so.1 => /hpc/software/gcc/4.8.3/lib64/libgomp.so.1 (0x00002b2e2e368000)
libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00002b2e2e576000)
libfontconfig.so.1 => /usr/lib64/libfontconfig.so.1 (0x00002b2e2e792000)
libfreetype.so.6 => /usr/lib64/libfreetype.so.6 (0x00002b2e2e9cf000)
libXrender.so.1 => /usr/lib64/libXrender.so.1 (0x00002b2e2ec75000)
libXau.so.6 => /usr/lib64/libXau.so.6 (0x00002b2e2ee7f000)
libuuid.so.1 => /usr/lib64/libuuid.so.1 (0x00002b2e2f083000)
libxcb.so.1 => /usr/lib64/libxcb.so.1 (0x00002b2e2f288000)
/lib64/ld-linux-x86-64.so.2 (0x00002b2e2c01b000)
librt.so.1 => /usr/lib64/librt.so.1 (0x00002b2e2f4aa000)
libexpat.so.1 => /usr/lib64/libexpat.so.1 (0x00002b2e2f6b2000)

6

Hi Ajay,

Yes, the centos_7 package is compiled with OpenMP support.

What is the RAM comparison between those systems? There is a
reasonable chance that the RedHat system ran out of memory.

It is also worth noting that there was a small bug in the handling of
warp file names, however that seems very unlikely to be relevant.
But see this thread.

In any case, it might be worth trying that exact command with the
centos_7 binaries. To do so, go to the results directory and:

wget https://afni.nimh.nih.gov/pub/dist/bin/linux_centos_7_64/3dNwarpApply
chmod 755 3dNwarpApply
./3dNwarpApply -interp NN -nwarp ...

The 3dNwarpApply command should be the exact same one you
ran before, but with ./3dNwarpApply, to use the one in the current
directory.

Does that work any better? Probably not, but it is worth trying.
Also, try to monitor memory use while it runs, if possible.

  • rick