AFNI_21.0.12 Slow performance on Ubuntu 16.04 and 20.04

cameron · February 26, 2021, 9:49pm

Hello Colleagues,

I recently installed AFNI_21.0.12 on Ubuntu 20.04 (built Feb 23 or 24, 2021) and noticed that 3dROIstats was very slow. Taking 2.5 minutes for 3.5x3.5x4 data with 300 TRs. I creating a whole brain mean time course using a whole-brain mask (e.g. 1 inside the brain 0 outside). I verified this same behavior on Ubuntu 16.04. I get the same slow performance with and without PIGZ enabled using environment variables.

I built the latest code from the AFNI github repo using Makefile.linux_ubuntu_16_64 on Ubuntu 20.04 and get the same slow behavior.

AFNI_20.0.04:Linux_ubuntu_64 built Jan 29, 2020 does the same extraction takes 1 second on Ubuntu 16.04 and 20.04.

Any thoughts on how to fix this? In the mean time, is there a place where we can download an earlier build? We will be building a new container for CPAC soon and would like to avoid this problem.

Cheers,
cameron

ptaylor · February 26, 2021, 10:12pm

Hi, Cameron-

What is the full command you ran? I would like to try it out on my computer here, too.

Also, what is the output of each of these:


afni -no_detach -quiet -VAFNI_COMPRESSOR=
afni -no_detach -quiet -VAFNI_AUTOGZIP=
afni -no_detach -quiet -VAFNI_NOMMAP=

thanks,
pt

dglen · February 28, 2021, 6:05am

I’ll just add that I’m seeing still fast performance with “3dROIstats -mask” on an EPI dataset on my Ubuntu 18.04 laptop using the 16.04 Makefile and similar performance from the distributed binary. Is this a local file system, mounted network drive or USB?

cameron · March 1, 2021, 6:02pm

Here yo go …


++ Compile date = Feb 25 2021 {AFNI_21.0.12:linux_ubuntu_16_64}

afni -no_detach -quiet -VAFNI_COMPRESSOR=
<BLANK>

afni -no_detach -quiet -VAFNI_AUTOGZIP=
YES

afni -no_detach -quiet -VAFNI_NOMMAP=
<BLANK>

time 3dROIstats -mask mask2.nii.gz func.nii.gz
File	Sub-brick	Mean_1  
func.nii.gz	0[?]	9957.563410
func.nii.gz	1[?]	9947.920665
func.nii.gz	2[?]	9938.978018
func.nii.gz	3[?]	9942.901682
...
func.nii.gz	335[?]	10079.657652
func.nii.gz	336[?]	10037.459720
func.nii.gz	337[?]	10058.559225

real	2m28.275s
user	2m14.740s
sys	0m13.461s


++ Compile date = Jan 29 2020 {AFNI_20.0.04:linux_ubuntu_16_64}

~/abin_older/afni -no_detach -quiet -VAFNI_COMPRESSOR=
<BLANK>

~/abin_older/afni -no_detach -quiet -VAFNI_AUTOGZIP=
YES

~/abin_older/afni -no_detach -quiet -VAFNI_NOMMAP=
<BLANK>

time ~/abin_older/3dROIstats -mask mask2.nii.gz func.nii.gz 
File	Sub-brick	Mean_1  
func.nii.gz	0[?]	9957.563410
func.nii.gz	1[?]	9947.920665
func.nii.gz	2[?]	9938.978018
...
func.nii.gz	335[?]	10079.657652
func.nii.gz	336[?]	10037.459720
func.nii.gz	337[?]	10058.559225

real	0m0.531s
user	0m0.475s
sys	0m0.056s

cameron · March 1, 2021, 6:05pm

Hello Daniel,

The input data is on the local file system, which uses to PCIe 4.0 M.2 SSDs in a RAID 0 configuration with ext4. I also suspected the file system at first so reran from a ramdisk and saw the exact same performance.

ptaylor · March 2, 2021, 2:49pm

Hi, Cameron-

TL;DR of below is that I am not having the same performance issues on my Ubuntu 20.04, even with newest binaries and using all gzipped dsets. I think we do have some older binaries stored around, I will have to dig. But it would be better to solve this, so you don’t have to use outdated code…

I assume if you run


afni_system_check.py -check_all

nothing gets flagged?

I don’t see how this would be related, but I note that in July of 2020, the linux_ubuntu_16_64 binaries got bumped up to having shared object efficiency, in that libmri.so removed the degeneracy of dependencies and greatly reduced the file size of the distribution. That started at ver=20.2.03. I don’t see how that could relate to this, but that was a big change for these binaries between the version numbers you are citing.

If I could ask about two more env vars, what are your outputs of these?


afni -no_detach -q -VAFNI_COMPRESSOR=
afni -no_detach -q -VAFNI_DONT_USE_PIGZ=

(The results on my comp for each of these is .)

— my tests —

I tested the binaries using Bootcamp dsets. Just to reduce heterogeneity, could you try the 3dROIstats commands below on the same dsets (downloadable from here, if you don’t have it already:
https://afni.nimh.nih.gov/pub/dist/edu/data/CD.tgz
in AFNI_data6/afni/)

I just downloaded the latest binaries on my Ubuntu 20.04


Precompiled binary linux_ubuntu_16_64: Feb 25 2021 (Version AFNI_21.0.12 'Titus')

We have the same env variable output for those three:


$ afni -no_detach -quiet -VAFNI_COMPRESSOR=
<BLANK>

$ afni -no_detach -quiet -VAFNI_AUTOGZIP=
YES

$ afni -no_detach -quiet -VAFNI_NOMMAP=
<BLANK>

I ran this on the Bootcamp data (EPI has 152 vols and 2.75x2.75.3.00 mm**3 vox):


$ time ~/linux_ubuntu_16_64/3dROIstats -mask mask.auto.nii.gz epi_r1+orig.HEAD
File	Sub-brick	Mean_1  
epi_r1+orig.HEAD	0[#0]	1541.724452
epi_r1+orig.HEAD	1[#1]	1380.877077
epi_r1+orig.HEAD	2[#2]	1340.660904
epi_r1+orig.HEAD	3[#3]	1339.089622

epi_r1+orig.HEAD	148[#148]	1334.726368
epi_r1+orig.HEAD	149[#149]	1333.623429
epi_r1+orig.HEAD	150[#150]	1332.642166
epi_r1+orig.HEAD	151[#151]	1334.085370

real	0m0.103s
user	0m0.068s
sys	0m0.035s

getting good performance.

I also tried concatenating 4 of those time series together, to make a longer dset (and a gzipped one):


3dTcat -prefix LONG.nii.gz epi_r1+orig. epi_r1+orig. epi_r1+orig. epi_r1+orig.

… but even then the performance was still fine:


$ time ~/linux_ubuntu_16_64/3dROIstats -mask mask.auto.nii.gz LONG.nii.gz
File	Sub-brick	Mean_1  
LONG.nii.gz	0[#0]	1541.724452
LONG.nii.gz	1[#1]	1380.877077
LONG.nii.gz	2[#2]	1340.660904
LONG.nii.gz	3[#3]	1339.089622
.....
LONG.nii.gz	603[#147]	1334.791271
LONG.nii.gz	604[#148]	1334.726368
LONG.nii.gz	605[#149]	1333.623429
LONG.nii.gz	606[#150]	1332.642166
LONG.nii.gz	607[#151]	1334.085370

real	0m1.294s
user	0m1.154s
sys	0m0.140s

–pt

cameron · March 4, 2021, 3:03am

Here is everything that you requested. I do not see slow performance with the data from the bootcamp. You can download the data I am using from here: https://utexas.box.com/s/7penj3t5k5w3p4ah6ipfxxlfbdbp9lgx (it is from FCP-INDI/CoRR)


afni_system_check.py -check_all

-------------------------------- general ---------------------------------
architecture:         64bit ELF
system:               Linux
release:              5.4.0-66-generic
version:              #74-Ubuntu SMP Wed Jan 27 22:54:38 UTC 2021
distribution:         Ubuntu 20.04.2 LTS
number of CPUs:       24
apparent login shell: bash
shell RC file:        .bashrc (exists)

--------------------- AFNI and related program tests ---------------------
which afni           : /home/cameron/abin/afni
afni version         : Precompiled binary linux_ubuntu_16_64: Feb 25 2021 
                     : AFNI_21.0.12 'Titus'
AFNI_version.txt     : AFNI_21.0.12, linux_ubuntu_16_64, Feb 25 2021
which python         : /home/cameron/venvs/cpac/bin/python
python version       : 3.8.5
which R              : /usr/bin/R
R version            : R version 3.6.3 (2020-02-29) -- "Holding the Windsock"
which tcsh           : /usr/bin/tcsh

instances of various programs found in PATH:
    afni    : 1   (/home/cameron/abin/afni)
    R       : 1   (/usr/bin/R)
    python  : 1   (/usr/bin/python3.8)
    python2 : 1   (/usr/bin/python2.7)
    python3 : 1   (/usr/bin/python3.8)


testing ability to start various programs...
    afni                 : success
    suma                 : success
    3dSkullStrip         : success
    uber_subject.py      : success
    3dAllineate          : success
    3dRSFC               : success
    SurfMesh             : success
    3dClustSim           : success
    3dMVM                : success

checking for R packages...
    rPkgsInstall -pkgs ALL -check : FAILURE
        
        oo Warning: 
           These packages are not installed on the computer: afex!
        These packages are not installed on the computer: phia!
        These packages are not installed on the computer: snow!
        These packages are not installed on the computer: lmerTest!
        These packages are not installed on the computer: paran!
        These packages are not installed on the computer: brms!
        These packages are not installed on the computer: corrplot!
        These packages are not installed on the computer: metafor!
        

R RHOME : /usr/lib/R

checking for $HOME files...
    .afnirc                   : found
    .sumarc                   : found
    .afni/help/all_progs.COMP : found

------------------------------ python libs -------------------------------
** failed to load module PyQt4
-- PyQt4 is no longer needed for an AFNI bootcamp

++ module loaded: matplotlib.pyplot
   module file : /home/cameron/venvs/cpac/lib/python3.8/site-packages/matplotlib/pyplot.py

-------------------------------- env vars --------------------------------
PATH = /home/cameron/venvs/cpac/bin:/home/cameron/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/cameron/abin

PYTHONPATH = 
R_LIBS = 
LD_LIBRARY_PATH = 
DYLD_LIBRARY_PATH = 
DYLD_FALLBACK_LIBRARY_PATH = 

------------------------------ data checks -------------------------------
data dir : missing AFNI_data6
data dir : missing AFNI_demos
data dir : missing suma_demo
data dir : missing afni_handouts
atlas    : found TT_N27+tlrc  under /home/cameron/abin

------------------------------ OS specific -------------------------------
which apt-get        : /usr/bin/apt-get
apt-get version      : apt 2.0.4 (amd64)

have Ubuntu system: Ubuntu 20.04.2 LTS
have Ubuntu afni  : Precompiled binary linux_ubuntu_16_64: Feb 25 2021 

=========================  summary, please fix:  =========================
*  just be aware: login shell 'bash', but our code examples use 'tcsh'
*  missing R packages (see rPkgsInstall)
*  insufficient data for AFNI bootcamp


afni -no_detach -q -VAFNI_COMPRESSOR=
<BLANK>

afni -no_detach -q -VAFNI_DONT_USE_PIGZ=
<BLANK>

Still seeing the problem:


time ~/abin/3dROIstats -mask mask2.nii.gz func.nii.gz 
File	Sub-brick	Mean_1  
func.nii.gz	0[?]	9957.563410
func.nii.gz	1[?]	9947.920665
func.nii.gz	2[?]	9938.978018
func.nii.gz	3[?]	9942.901682
...
func.nii.gz	334[?]	10085.826948
func.nii.gz	335[?]	10079.657652
func.nii.gz	336[?]	10037.459720
func.nii.gz	337[?]	10058.559225

real	2m35.273s
user	2m21.209s
sys	0m13.998s

Now trying with bootcamp data …


++ Compile date = Feb 25 2021 {AFNI_21.0.12:linux_ubuntu_16_64}

time ~/abin/3dROIstats -mask mask.auto.nii.gz epi_r1+orig.HEAD
File	Sub-brick	Mean_1  
epi_r1+orig.HEAD	0[#0]	1541.724452
epi_r1+orig.HEAD	1[#1]	1380.877077
epi_r1+orig.HEAD	2[#2]	1340.660904
epi_r1+orig.HEAD	3[#3]	1339.089622
...
epi_r1+orig.HEAD	148[#148]	1334.726368
epi_r1+orig.HEAD	149[#149]	1333.623429
epi_r1+orig.HEAD	150[#150]	1332.642166
epi_r1+orig.HEAD	151[#151]	1334.085370

real	0m0.114s
user	0m0.065s
sys	0m0.049s


time ~/abin/3dROIstats -mask mask.auto.nii.gz LONG.nii.gz
File	Sub-brick	Mean_1  
LONG.nii.gz	0[#0]	1541.724452
LONG.nii.gz	1[#1]	1380.877077
LONG.nii.gz	2[#2]	1340.660904
LONG.nii.gz	3[#3]	1339.089622
...
LONG.nii.gz	605[#149]	1333.623429
LONG.nii.gz	606[#150]	1332.642166
LONG.nii.gz	607[#151]	1334.085370

real	0m1.466s
user	0m1.398s
sys	0m0.068s



3dinfo epi_r1+orig.HEAD 
++ 3dinfo: AFNI version=AFNI_21.0.12 (Feb 25 2021) [64-bit]

Dataset File:    epi_r1+orig
Identifier Code: AFN_1obeBrbaja6u2HfupnCFbA  Creation Date: Thu Mar 18 14:28:42 2010
Template Space:  ORIG
Dataset Type:    Echo Planar (-epan)
Byte Order:      LSB_FIRST [this CPU native = LSB_FIRST]
Storage Mode:    BRIK
Storage Space:   64,204,800 (64 million) bytes
Geometry String: "MATRIX(2.75,0,0,-112.6622,0,2.75,0,-122.7787,0,0,3,-17.65058):80,80,33"
Data Axes Tilt:  Plumb
Data Axes Orientation:
  first  (x) = Right-to-Left
  second (y) = Anterior-to-Posterior
  third  (z) = Inferior-to-Superior   [-orient RAI]
R-to-L extent:  -112.662 [R] -to-   104.588 [L] -step-     2.750 mm [ 80 voxels]
A-to-P extent:  -122.779 [A] -to-    94.471 [P] -step-     2.750 mm [ 80 voxels]
I-to-S extent:   -17.651 [I] -to-    78.349 [S] -step-     3.000 mm [ 33 voxels]
Number of time steps = 152  Time step = 2.00000s  Origin = 0.00000s  Number time-offset slices = 33  Thickness = 3.000
  -- At sub-brick #0 '#0' datum type is short:            0 to          3272
  -- At sub-brick #1 '#1' datum type is short:            0 to          2924
  -- At sub-brick #2 '#2' datum type is short:            0 to          2841
** For info on all 152 sub-bricks, use '3dinfo -verb' **

----- HISTORY -----
[rickr@manwe.nimh.nih.gov: Thu Mar 18 11:19:14 2010] 3dcopy epi_r1.nii FT_epi_r1
[rickr@manwe.nimh.nih.gov: Thu Mar 18 14:28:42 2010] 3dcopy FT_epi_r1+orig epi_r1


++ 3dinfo: AFNI version=AFNI_21.0.12 (Feb 25 2021) [64-bit]

Dataset File:    func.nii.gz
Identifier Code: NII_hDcnbsY6k2aQQvVLdGzh0w  Creation Date: Wed Mar  3 20:39:20 2021
Template Space:  ORIG
Dataset Type:    Echo Planar (-epan)
Byte Order:      LSB_FIRST {assumed} [this CPU native = LSB_FIRST]
Storage Mode:    NIFTI
Storage Space:   160,595,968 (161 million) bytes
Geometry String: "MATRIX(3.4375,0,0,-119.0361,0,-3.4375,0,65.85879,0,0,3.959766,-91.56892):64,64,29"
Data Axes Tilt:  Plumb
Data Axes Orientation:
  first  (x) = Right-to-Left
  second (y) = Posterior-to-Anterior
  third  (z) = Inferior-to-Superior   [-orient RPI]
R-to-L extent:  -119.036 [R] -to-    97.526 [L] -step-     3.438 mm [ 64 voxels]
A-to-P extent:  -150.704 [A] -to-    65.859 [P] -step-     3.437 mm [ 64 voxels]
I-to-S extent:   -91.569 [I] -to-    19.305 [S] -step-     3.960 mm [ 29 voxels]
Number of time steps = 338  Time step = 1.75000s  Origin = 0.00000s
  -- At sub-brick #0 '?' datum type is float
  -- At sub-brick #1 '?' datum type is float
  -- At sub-brick #2 '?' datum type is float
** For info on all 338 sub-bricks, use '3dinfo -verb' **

Other than voxel size and number of TRs the main different in datasets is the data type and the range of values, lets make epi_r1 match func.nii.gz for those and see if it changes anything:


3dcalc -prefix epi_r1_float.nii.gz -a epi_r1 -expr '10000*a/2962' -datum float

time 3dROIstats -mask mask.auto.nii.gz epi_r1_float.nii.gz 
File	Sub-brick	Mean_1  
epi_r1_float.nii.gz	0[#0]	5205.011654
epi_r1_float.nii.gz	1[#1]	4661.975278
epi_r1_float.nii.gz	2[#2]	4526.201565
epi_r1_float.nii.gz	3[#3]	4520.896766
...

epi_r1_float.nii.gz	148[#148]	4506.165995
epi_r1_float.nii.gz	149[#149]	4502.442365
epi_r1_float.nii.gz	150[#150]	4499.129527
epi_r1_float.nii.gz	151[#151]	4504.001923

real	0m0.481s
user	0m0.457s
sys	0m0.024s

When I try to convert func.nii.gz to float, I get:


3dcalc -prefix func_short.nii.gz -a func.nii.gz -exp 'a' -datum short
++ 3dcalc: AFNI version=AFNI_21.0.12 (Feb 25 2021) [64-bit]
++ Authored by: A cast of thousands
*+ WARNING: varying brick factors, writing NIfTI as float
++ Output dataset ./func_short.nii.gz

Could that be related to the problem

Thanks again for the help!

=cc

rickr · March 4, 2021, 3:10pm

Okay, wow. Even on Fedora I can compare those Ubuntu binaries to mine. The Unbuntu version runs almost instantly, whereas mine seems to make more of an arduous journey.

Thanks for providing this.

rick

rickr · March 4, 2021, 4:58pm

Hi Cameron,

This is fixed (or at least committed at github).

I did not quite find the source of the change. However, there was an unload/reload operation happening per time point, that had started making NIFTI float datasets get re-read from disk. That might have been an attempt to free memory as volumes were processed, being careful in case float data had a scale factor to apply (so no MMAP). Anyway, it is fast again.

Thanks for letting us know, and for the useful demo data!

rick

cameron · March 4, 2021, 7:08pm

Glad to hear! I will pull in the changes and let you know how it works.

Thanks!

Cameron