Clustsim Failed in 3dttest++

AFNI Message Board
1

Hi all,
I failed to use the -Clustsim option in an independent t-test on two samples (n1 = 22, n2 = 23).

My afni version:
Precompiled binary linux_ubuntu_16_64: Aug 22 2018 (Version AFNI_18.2.14)

my script:
datapath=“…/data”
3dttest++ -setA C2
s031 ${datapath}/stat_s031_MNI3mm_Blur6.nii.gz’[7]’
s045 ${datapath}/stat_s045_MNI3mm_Blur6.nii.gz’[7]’
s024 ${datapath}/stat_s024_MNI3mm_Blur6.nii.gz’[7]’
s034 ${datapath}/stat_s034_MNI3mm_Blur6.nii.gz’[7]’
s025 ${datapath}/stat_s025_MNI3mm_Blur6.nii.gz’[7]’
s040 ${datapath}/stat_s040_MNI3mm_Blur6.nii.gz’[7]’
s009 ${datapath}/stat_s009_MNI3mm_Blur6.nii.gz’[7]’
s019 ${datapath}/stat_s019_MNI3mm_Blur6.nii.gz’[7]’
s054 ${datapath}/stat_s054_MNI3mm_Blur6.nii.gz’[7]’
s023 ${datapath}/stat_s023_MNI3mm_Blur6.nii.gz’[7]’
s017 ${datapath}/stat_s017_MNI3mm_Blur6.nii.gz’[7]’
s042 ${datapath}/stat_s042_MNI3mm_Blur6.nii.gz’[7]’
s029 ${datapath}/stat_s029_MNI3mm_Blur6.nii.gz’[7]’
s044 ${datapath}/stat_s044_MNI3mm_Blur6.nii.gz’[7]’
s051 ${datapath}/stat_s051_MNI3mm_Blur6.nii.gz’[7]’
s053 ${datapath}/stat_s053_MNI3mm_Blur6.nii.gz’[7]’
s001 ${datapath}/stat_s001_MNI3mm_Blur6.nii.gz’[7]’
s049 ${datapath}/stat_s049_MNI3mm_Blur6.nii.gz’[7]’
s028 ${datapath}/stat_s028_MNI3mm_Blur6.nii.gz’[7]’
s047 ${datapath}/stat_s047_MNI3mm_Blur6.nii.gz’[7]’
s005 ${datapath}/stat_s005_MNI3mm_Blur6.nii.gz’[7]’
s027 ${datapath}/stat_s027_MNI3mm_Blur6.nii.gz’[7]’
-setB C1
s006 ${datapath}/stat_s006_MNI3mm_Blur6.nii.gz’[7]’
s055 ${datapath}/stat_s055_MNI3mm_Blur6.nii.gz’[7]’
s013 ${datapath}/stat_s013_MNI3mm_Blur6.nii.gz’[7]’
s018 ${datapath}/stat_s018_MNI3mm_Blur6.nii.gz’[7]’
s016 ${datapath}/stat_s016_MNI3mm_Blur6.nii.gz’[7]’
s020 ${datapath}/stat_s020_MNI3mm_Blur6.nii.gz’[7]’
s022 ${datapath}/stat_s022_MNI3mm_Blur6.nii.gz’[7]’
s048 ${datapath}/stat_s048_MNI3mm_Blur6.nii.gz’[7]’
s052 ${datapath}/stat_s052_MNI3mm_Blur6.nii.gz’[7]’
s030 ${datapath}/stat_s030_MNI3mm_Blur6.nii.gz’[7]’
s050 ${datapath}/stat_s050_MNI3mm_Blur6.nii.gz’[7]’
s056 ${datapath}/stat_s056_MNI3mm_Blur6.nii.gz’[7]’
s043 ${datapath}/stat_s043_MNI3mm_Blur6.nii.gz’[7]’
s035 ${datapath}/stat_s035_MNI3mm_Blur6.nii.gz’[7]’
s039 ${datapath}/stat_s039_MNI3mm_Blur6.nii.gz’[7]’
s033 ${datapath}/stat_s033_MNI3mm_Blur6.nii.gz’[7]’
s036 ${datapath}/stat_s036_MNI3mm_Blur6.nii.gz’[7]’
s038 ${datapath}/stat_s038_MNI3mm_Blur6.nii.gz’[7]’
s026 ${datapath}/stat_s026_MNI3mm_Blur6.nii.gz’[7]’
s032 ${datapath}/stat_s032_MNI3mm_Blur6.nii.gz’[7]’
s041 ${datapath}/stat_s041_MNI3mm_Blur6.nii.gz’[7]’
s012 ${datapath}/stat_s012_MNI3mm_Blur6.nii.gz’[7]’
s046 ${datapath}/stat_s046_MNI3mm_Blur6.nii.gz’[7]’
-prefix tomNetwork_C2-C1_withinBrain_beta.nii.gz
-toz
-mask meanBrainMask+tlrc.
-Clustsim

Output:
++ 3dttest++: AFNI version=AFNI_18.2.14 (Aug 22 2018) [64-bit]
++ Authored by: Zhark++
++ option -setA :: processing as LONG form (label label dset label dset …)
++ option -setB :: processing as LONG form (label label dset label dset …)
++ 74520 voxels in -mask dataset
++ Number of -Clustsim threads set to 8

  • Default clustsim prefix set to ‘tomNetwork_C2-C1_withinBrain_beta.nii.gz’
    ++ 2-sample test: ‘-AminusB’ option is assumed
    ++ 2-sample test: results will be C2 - C1
    ++ loading -setA datasets
    ++ loading -setB datasets
    ++ Memory usage now = 34,372,988 (34 million [mega])
    ++ t-testing:0123456789.0123456789.0123456789.0123456789.0123456789.!
    ++ ---------- End of analyses – freeing workspaces ----------
    ++ Memory usage now = 68,047,644 (68 million [mega])
    ++ Creating FDR curves in output dataset
    ++ Smallest FDR q [1 C2-C1_Zscr] = 0.0295759
    ++ Smallest FDR q [3 C2_Zscr] = 5.32057e-08
    ++ Smallest FDR q [5 C1_Zscr] = 5.27788e-09
  • Added 3 FDR curves to dataset
    ++ Output dataset ./tomNetwork_C2-C1_withinBrain_beta.nii.gz
    ++ Output dataset ./tomNetwork_C2-C1_withinBrain_beta.nii.gz.resid.nii
  • 2-sample test: results are C2 - C1
    ++ ================ Starting -Clustsim calculations ================
  • === temporary files will have prefix tomNetwork_C2-C1_withinBrain_beta.nii.gz ===
  • === running 8 -randomsign jobs (1250 iterations per job) ===
  • === creating 1,490,400,000 (1.5 billion [giga]) bytes of pseudo-data in .sdat files ===
  • — 3dClustSim reads .sdat files to compute cluster-threshold statistics —
  • — there is 16,720,445,440 (17 billion [giga]) bytes of memory on your system —
    ++ 3dttest++: AFNI version=AFNI_18.2.14 (Aug 22 2018) [64-bit]
    ++ Authored by: Zhark++
    ++ 74520 voxels in -mask dataset
    ++ option -setA :: processing as SHORT form (all values are datasets)
    ++ 3dttest++: AFNI version=AFNI_18.2.14 (Aug 22 2018) [64-bit]
    ++ Authored by: Zhark++
    ++ option -setB :: processing as SHORT form (all values are datasets)
    ++ random seeds are 255669912 569829177
    ++ 2-sample test: ‘-AminusB’ option is assumed
    ++ 2-sample test: results will be SetA - SetB
    ++ opened file ./tomNetwork_C2-C1_withinBrain_beta.sdat for output
    ++ loading -setA datasets
    ++ 3dttest++: AFNI version=AFNI_18.2.14 (Aug 22 2018) [64-bit]
    ++ Authored by: Zhark++
    ++ 74520 voxels in -mask dataset
    ++ option -setA :: processing as SHORT form (all values are datasets)
    ++ option -setB :: processing as SHORT form (all values are datasets)
    ++ random seeds are 256781063 570940328
    ++ 2-sample test: ‘-AminusB’ option is assumed
    ++ 2-sample test: results will be SetA - SetB
    ++ opened file ./tomNetwork_C2-C1_withinBrain_beta.sdat for output
    ++ loading -setA datasets
    ++ loading -setB datasets
    ++ 3dttest++: AFNI version=AFNI_18.2.14 (Aug 22 2018) [64-bit]
    ++ Authored by: Zhark++
    ++ 74520 voxels in -mask dataset
    ++ option -setA :: processing as SHORT form (all values are datasets)
    ++ option -setB :: processing as SHORT form (all values are datasets)
    ++ 3dttest++: AFNI version=AFNI_18.2.14 (Aug 22 2018) [64-bit]
    ++ Authored by: Zhark++
    ++ random seeds are 257892214 572051479
    ++ 2-sample test: ‘-AminusB’ option is assumed
    ++ 2-sample test: results will be SetA - SetB
    ++ opened file ./tomNetwork_C2-C1_withinBrain_beta.sdat for output
    ++ loading -setA datasets
    ++ 3dttest++: AFNI version=AFNI_18.2.14 (Aug 22 2018) [64-bit]
    ++ Authored by: Zhark++
    ++ loading -setB datasets
    ++ 3dttest++: AFNI version=AFNI_18.2.14 (Aug 22 2018) [64-bit]
    ++ Authored by: Zhark++
    ++ Memory usage now = 16,656,471 (17 million [mega])
    ++ t-test randomsign/permute:++ 74520 voxels in -mask dataset
    ++ option -setA :: processing as SHORT form (all values are datasets)
    ++ option -setB :: processing as SHORT form (all values are datasets)
    ++ random seeds are 259003365 573162630
    ++ 2-sample test: ‘-AminusB’ option is assumed
    ++ 2-sample test: results will be SetA - SetB
    ++ 3dttest++: AFNI version=AFNI_18.2.14 (Aug 22 2018) [64-bit]
    ++ Authored by: Zhark++
    ++ 74520 voxels in -mask dataset
    ++ opened file ./tomNetwork_C2-C1_withinBrain_beta.sdat for output
    ++ option -setA :: processing as SHORT form (all values are datasets)
    ++ option -setB :: processing as SHORT form (all values are datasets)
    ++ random seeds are 261225667 575384932
    ++ 2-sample test: ‘-AminusB’ option is assumed
    ++ 2-sample test: results will be SetA - SetB
    ++ loading -setB datasets
    ++ loading -setA datasets
    ++ opened file ./tomNetwork_C2-C1_withinBrain_beta.sdat for output
    ++ Memory usage now = 16,656,471 (17 million [mega])
    ++ t-test randomsign/permute:++ loading -setA datasets
    ++ 74520 voxels in -mask dataset
    ++ option -setA :: processing as SHORT form (all values are datasets)
    ++ option -setB :: processing as SHORT form (all values are datasets)
    ++ 74520 voxels in -mask dataset
    ++ option -setA :: processing as SHORT form (all values are datasets)
    ++ random seeds are 260114516 574273781
    ++ 2-sample test: ‘-AminusB’ option is assumed
    ++ 2-sample test: results will be SetA - SetB
    ++ option -setB :: processing as SHORT form (all values are datasets)
    ++ random seeds are 262336818 576496083
    ++ 2-sample test: ‘-AminusB’ option is assumed
    ++ 2-sample test: results will be SetA - SetB
    ++ opened file ./tomNetwork_C2-C1_withinBrain_beta.sdat for output
    ++ opened file ./tomNetwork_C2-C1_withinBrain_beta.sdat for output
    ++ 74520 voxels in -mask dataset
    ++ Memory usage now = 16,656,471 (17 million [mega])
    ++ t-test randomsign/permute:++ loading -setA datasets
    ++ option -setA :: processing as SHORT form (all values are datasets)
    ++ loading -setA datasets
    ++ option -setB :: processing as SHORT form (all values are datasets)
    ++ random seeds are 263447969 577607234
    ++ 2-sample test: ‘-AminusB’ option is assumed
    ++ 2-sample test: results will be SetA - SetB
    ++ opened file ./tomNetwork_C2-C1_withinBrain_beta.sdat for output
    ++ loading -setB datasets
    ++ loading -setA datasets
    ++ loading -setB datasets
    ++ loading -setB datasets
    ++ loading -setB datasets
    ++ loading -setB datasets
    ++ Memory usage now = 16,656,471 (17 million [mega])
    ++ t-test randomsign/permute:++ Memory usage now = 16,656,471 (17 million [mega])
    ++ t-test randomsign/permute:++ Memory usage now = 16,656,471 (17 million [mega])
    ++ t-test randomsign/permute:++ Memory usage now = 16,656,471 (17 million [mega])
    ++ t-test randomsign/permute:++ Memory usage now = 16,656,471 (17 million [mega])
    ++ t-test randomsign/permute:0000000011111111222222223333333344444444555555556666666677777777888888889.9.9.9.9.9.9.9.0000000011111111222222223333333344444444555555556666666677777777888888889.9.9.9.9.9.9.9.000000001111111122222222333333344344444555455556666666577777776888888879.9.9.9.9.9.9.800000009.11111110222222213333333244444443555555546666666577777776888888879.9.9.9.9.9.9.800000009.11111110222222213333333244444443555555546666666577777776888888879.9.9.9.9.9.9.!
    ++ saving main effect t-stat MIN/MAX values in ./tomNetwork_C2-C1_withinBrain_beta.minmax.1D
    ++ output short-ized file ./tomNetwork_C2-C1_withinBrain_beta.sdat
    !
    ++ saving main effect t-stat MIN/MAX values in ./tomNetwork_C2-C1_withinBrain_beta.minmax.1D
    ** ERROR: (FAILED) attempt to over-write file ./tomNetwork_C2-C1_withinBrain_beta.minmax.1D
    ++ output short-ized file ./tomNetwork_C2-C1_withinBrain_beta.sdat
    !
    ++ saving main effect t-stat MIN/MAX values in ./tomNetwork_C2-C1_withinBrain_beta.minmax.1D
    ** ERROR: (FAILED) attempt to over-write file ./tomNetwork_C2-C1_withinBrain_beta.minmax.1D
    ++ output short-ized file ./tomNetwork_C2-C1_withinBrain_beta.sdat
    !
    ++ saving main effect t-stat MIN/MAX values in ./tomNetwork_C2-C1_withinBrain_beta.minmax.1D
    ** ERROR: (FAILED) attempt to over-write file ./tomNetwork_C2-C1_withinBrain_beta.minmax.1D
    ++ output short-ized file ./tomNetwork_C2-C1_withinBrain_beta.sdat
    !
    ++ saving main effect t-stat MIN/MAX values in ./tomNetwork_C2-C1_withinBrain_beta.minmax.1D
    ** ERROR: (FAILED) attempt to over-write file ./tomNetwork_C2-C1_withinBrain_beta.minmax.1D
    ++ output short-ized file ./tomNetwork_C2-C1_withinBrain_beta.sdat
    !
    ++ saving main effect t-stat MIN/MAX values in ./tomNetwork_C2-C1_withinBrain_beta.minmax.1D
    ** ERROR: (FAILED) attempt to over-write file ./tomNetwork_C2-C1_withinBrain_beta.minmax.1D
    ++ output short-ized file ./tomNetwork_C2-C1_withinBrain_beta.sdat
  • 3dttest++ ===== simulation jobs have finished (448.3 s elapsed)
    *+ WARNING: Can’t read file ./tomNetwork_C2-C1_withinBrain_beta.nii.gz.0000.minmax.1D
    !
    ++ saving main effect t-stat MIN/MAX values in ./tomNetwork_C2-C1_withinBrain_beta.minmax.1D
    ** ERROR: (FAILED) attempt to over-write file ./tomNetwork_C2-C1_withinBrain_beta.minmax.1D
    ++ output short-ized file ./tomNetwork_C2-C1_withinBrain_beta.sdat
    *+ WARNING: Can’t read file ./tomNetwork_C2-C1_withinBrain_beta.nii.gz.0001.minmax.1D
    *+ WARNING: Can’t read file ./tomNetwork_C2-C1_withinBrain_beta.nii.gz.0002.minmax.1D
    *+ WARNING: Can’t read file ./tomNetwork_C2-C1_withinBrain_beta.nii.gz.0003.minmax.1D
    *+ WARNING: Can’t read file ./tomNetwork_C2-C1_withinBrain_beta.nii.gz.0004.minmax.1D
    *+ WARNING: Can’t read file ./tomNetwork_C2-C1_withinBrain_beta.nii.gz.0005.minmax.1D
    *+ WARNING: Can’t read file ./tomNetwork_C2-C1_withinBrain_beta.nii.gz.0006.minmax.1D
    *+ WARNING: Can’t read file ./tomNetwork_C2-C1_withinBrain_beta.nii.gz.0007.minmax.1D
    *+ WARNING: COULD NOT read any .minmax.1D files for unknown reasons!
  • ==> no global threshold .5percent.txt files are output :frowning:
  • … this failure does not affect any other Clustim/ETAC results!
  • 3dttest++ ===== starting 3dClustSim A: elapsed = 451.2 s
    ++ 3dClustSim: AFNI version=AFNI_18.2.14 (Aug 22 2018) [64-bit]
    ++ Authored by: RW Cox and BD Ward
    ++ Loading -insdat datasets
    ** FATAL ERROR: can’t find any data in file ‘./tomNetwork_C2-C1_withinBrain_beta.nii.gz.0000.sdat’
    ** Program compile date = Aug 22 2018
    ** FATAL ERROR: ===== 3dClustSim command failed :-((( =====
    ** Program compile date = Aug 22 2018
    ++ saving main effect t-stat MIN/MAX values in ./tomNetwork_C2-C1_withinBrain_beta.minmax.1D
    ** ERROR: (FAILED) attempt to over-write file ./tomNetwork_C2-C1_withinBrain_beta.minmax.1D
    ++ output short-ized file ./tomNetwork_C2-C1_withinBrain_beta.sdat

Thanks!
-LZ

2

Hi LZ,

Did you figure this out? I wonder whether it has to do with having
noclobber set on your system. It is not clear what shell this might
be using, so would you show the commands and output from:

tcsh -c ‘echo $noclobber’
bash -c ‘echo $noclobber’

Thanks,

  • rick
3

Hi Rick,
Thanks for your reply！
I have no idea to fix this problem since I posted this question.
This is the output using your command.

(BTW, I use ubuntu 16.04 subsystem within windows 10.)

lzyang@~$ tcsh -c ‘echo $noclobber’
noclobber: Undefined variable.
lzyang@~$ bash -c ‘echo $noclobber’

lzyang@~$

As you see, tcsh reported that this is an undefined variable. But the bash report nothing.

Thanks very much for your help!
-lz

4

So that isn’t it.

Try running it without the .nii.gz part of the prefix,
e.g. -prefix tomNetwork_C2-C1_withinBrain_beta

That nii.gz aspect might not be fully handled in the
case of -Clustsim.

  • rick
5

Hi Rick,

Thanks very much for your help!
it worked!

-lz