Cluster Explorer has arrived!!

Hi,

Cluster Explorer Intro Tutorial

There is a new set of scripts to help explore your data!
The scripts are included in the latest binaries.
The tutorial link (above) has more information and some basic steps to get started.
Post any problems, questions, or suggestions in this thread if you like.
The capabilities may be expanded in the future and it is rather new. So be kind.
Throw all of the data at it you can and see what breaks it.
As Bob says, “You are all beta testers for life.”

Thanks, Justin


Cluster Explorer is a set of scripts that will allow you to explore your data after you have run some group level analysis. This may hopefully save you some time in preparing your data for figures, publications, and sharing. It automates a few data wrangling steps that enable you to explore your subject data at the cluster level. The end result is a shiny app that you can view in your web browser linked to a afni instance that displays your thresholded statistics dataset and clusters. You can learn more about shiny apps here.

The basic functions performed are:

[ul]
[li] Collect all of the input datasets.
[/li][li] Threshold and clusterize the statistics dataset.
[/li][li] Look up atlas locations for each cluster.
[/li][li] For each cluster, extract the data from the all of the input datasets.
[/li][li] Extract the model information from the statistics dataset.
[/li][li] Label all clusters for all subjects and merge with the model information.
[/li][li] Create a shiny app that you can share to interactively explore and plot your data.
[/li][/ul]

The Cluster Explorer includes:
[ul]

[li] ClustExp_HistTable.py
[/li][li] @ClustExp_CatLab
[/li][li] ClustExp_StatParse.py
[/li][li] @ClustExp_run_shiny
[/li][/ul]

Updated a tried this yesterday!
For our lab we need to be able to render stats and images for “at a glance” viewing, and this looks like it will speed up the process.
This saves me from having to use two or more programs a multiple datasets just to show clinical data alongside the images.
Great directions and great project. I am excited to see where this goes!

Hi Justin,

I get this error at ClustExp_StatParse.py when I go through the tutorial with my data:


Resampling master to subject data.

-- will fix any bad char check errors in /media/z/data/results/loc/3dMVM/MyOutput_ClustExp_shiny/data/MyOutput_CleanSubjTab.1D
** non-Unix check output file '/media/z/data/results/loc/3dMVM/MyOutput_ClustExp_shiny/data/MyOutput_CleanSubjTab.1D' exists, will overwrite
-- will fix any non-Unix check errors in /media/z/data/results/loc/3dMVM/MyOutput_ClustExp_shiny/data/MyOutput_CleanSubjTab.1D
++ 3dmerge: AFNI version=AFNI_17.3.02 (Nov  1 2017) [64-bit]
-- Wrote edited dataset: /media/z/data/results/loc/3dMVM/MyOutput_ClustExp_shiny/data/MyOutput_p_uncor_0.005.nii.gz
++ 3dclust: AFNI version=AFNI_17.3.02 (Nov  1 2017) [64-bit]
++ Authored by: RW Cox et alii
++ Output dataset /media/z/data/results/loc/3dMVM/MyOutput_ClustExp_shiny/data/MyOutput_p_uncor_0.005_mask.nii.gz
/bin/sh: 1: Syntax error: "(" unexpected
Traceback (most recent call last):
  File "/home/z/abin/ClustExp_StatParse.py", line 495, in <module>
    subprocess.check_output(afni_cmd,shell=True)
  File "/usr/lib/python2.7/subprocess.py", line 574, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '3drefit -relabel_all_str (Intercept)_F_p_uncor_0.005 /media/z/data/results/loc/3dMVM/MyOutput_ClustExp_shiny/data/MyOutput_p_uncor_0.005.nii.gz' returned non-zero exit status 2


besides, I have run @afni_R_package_install -shiny, here is the system & afni info:


-------------------------------- general ---------------------------------
architecture:         64bit ELF
system:               Linux
release:              4.10.0-38-generic
version:              #42~16.04.1-Ubuntu SMP Tue Oct 10 16:32:20 UTC 2017
distribution:         Ubuntu 16.04 xenial
number of CPUs:       8
apparent login shell: tcsh
shell RC file:        .cshrc (exists)

--------------------- AFNI and related program tests ---------------------
which afni           : /home/z/abin/afni
afni version         : Precompiled binary linux_ubuntu_16_64: Nov  1 2017 
                     : AFNI_17.3.02
AFNI_version.txt     : AFNI_17.3.02, linux_ubuntu_16_64, Nov 01 2017
which python         : /usr/bin/python
python version       : 2.7.12
which R              : /usr/bin/R
R version            : R version 3.4.2 (2017-09-28) -- "Short Summer"
which tcsh           : /usr/bin/tcsh

instances of various programs found in PATH:
    afni    : 1   (/home/z/abin/afni)
    R       : 1   (/usr/bin/R)
    python  : 1   (/usr/bin/python2.7)
    python2 : 1   (/usr/bin/python2.7)
    python3 : 1   (/usr/bin/python3.5)


Thanks, 2086

hi,
Can you post the ClustExp_StatParse.py command you used?

The parentheses seems to be the problem here:
(Intercept)_F_p_uncor_0.005

Thanks, Justin

Hi Justin, what are the limitations of these scrips? Did anyone manage to bust them and with what?

Hi,
There are probably lots of ways to bust them…

The tutorial web page has some of the caveats:

https://afni.nimh.nih.gov/pub/dist/doc/htmldoc/tutorials/ClusterExplorer/Introduction.html

I hope that helps some.
If you need anything, let me know.

Thanks, Justin

Hi discoraj,

  1. There was an error when I ran ClustExp_StatParse.py. And I don’t know what the outputs mean:

...
Resampling master to subject data.
-- will fix any bad char check errors in /media/data/MyOutput_CleanSubjTab.1D
** non-Unix check output file '/media/data/MyOutput_CleanSubjTab.1D' exists, will overwrite
-- will fix any non-Unix check errors in /media/data/MyOutput_CleanSubjTab.1D
Traceback (most recent call last):
  File "/home/data/abin/ClustExp_StatParse.py", line 407, in <module>
    subj_dset_tab.append([fields[0],fields[2]])
IndexError: list index out of range

The code I used are:


ClustExp_StatParse.py \
-StatDSET box_ttest+tlrc \
-MeanBrik 0 \
-ThreshBrik 1 \
-SubjDSET All_My_Subj.nii.gz \
-SubjTable table.csv \
-master /media/data/TT_N27+tlrc

The table.csv:


sub001 sub001.results/stats.sub001_REML+tlrc[1] sub001.results/stats.sub001_REML+tlrc[1]
sub001 sub001.results/stats.sub001_REML+tlrc[4] sub001.results/stats.sub001_REML+tlrc[4]
sub002 sub002.results/stats.sub002_REML+tlrc[1] sub002.results/stats.sub002_REML+tlrc[1]
sub002 sub002.results/stats.sub002_REML+tlrc[4] sub002.results/stats.sub002_REML+tlrc[4]
sub003 sub003.results/stats.sub003_REML+tlrc[1] sub003.results/stats.sub003_REML+tlrc[1]
sub003 sub003.results/stats.sub003_REML+tlrc[4] sub003.results/stats.sub003_REML+tlrc[4]
sub004 sub004.results/stats.sub004_REML+tlrc[1] sub004.results/stats.sub004_REML+tlrc[1]
sub004 sub004.results/stats.sub004_REML+tlrc[4] sub004.results/stats.sub004_REML+tlrc[4]
sub005 sub005.results/stats.sub005_REML+tlrc[1] sub005.results/stats.sub005_REML+tlrc[1]
sub005 sub005.results/stats.sub005_REML+tlrc[4] sub005.results/stats.sub005_REML+tlrc[4]
sub006 sub006.results/stats.sub006_REML+tlrc[1] sub006.results/stats.sub006_REML+tlrc[1]
sub006 sub006.results/stats.sub006_REML+tlrc[4] sub006.results/stats.sub006_REML+tlrc[4]
sub007 sub007.results/stats.sub007_REML+tlrc[1] sub007.results/stats.sub007_REML+tlrc[1]
sub007 sub007.results/stats.sub007_REML+tlrc[4] sub007.results/stats.sub007_REML+tlrc[4]

And All_My_Subj.nii.gz:


3dinfo -verb All_My_Subj.nii.gz
++ 3dinfo: AFNI version=AFNI_18.0.02 (Jan 10 2018) [64-bit]

Dataset File:    All_My_Subj.nii.gz
Identifier Code: AFN_-S-e78_FXiL7GWL6dh4Gbg  Creation Date: Fri Jan 12 23:32:01 2018
Template Space:  TT_N27
Dataset Type:    Echo Planar (-epan)
Byte Order:      LSB_FIRST {assumed} [this CPU native = LSB_FIRST]
Storage Mode:    NIFTI
Storage Space:   9,676,800 (10 million [mega]) bytes
Geometry String: "MATRIX(3,0,0,-79.5,0,3,0,-79.5,0,0,3,-63.5):54,64,50"
Data Axes Tilt:  Plumb
Data Axes Orientation:
  first  (x) = Right-to-Left
  second (y) = Anterior-to-Posterior
  third  (z) = Inferior-to-Superior   [-orient RAI]
R-to-L extent:   -79.500 [R] -to-    79.500 [L] -step-     3.000 mm [ 54 voxels]
A-to-P extent:   -79.500 [A] -to-   109.500 [P] -step-     3.000 mm [ 64 voxels]
I-to-S extent:   -63.500 [I] -to-    83.500 [S] -step-     3.000 mm [ 50 voxels]
Number of time steps = 14  Time step = 1.00000s  Origin = 0.00000s
  -- At sub-brick #0 'sub001' datum type is float:     -11.9993 to       17.5647
  -- At sub-brick #1 'sub001' datum type is float:     -16.8132 to       18.0239
  -- At sub-brick #2 'sub002' datum type is float:     -13.7661 to       6.53791
  -- At sub-brick #3 'sub002' datum type is float:     -18.4548 to       11.5191
  -- At sub-brick #4 'sub003' datum type is float:     -19.1143 to       10.8273
  -- At sub-brick #5 'sub003' datum type is float:     -9.07329 to       5.96413
  -- At sub-brick #6 'sub004' datum type is float:     -6.55489 to       6.34929
  -- At sub-brick #7 'sub004' datum type is float:     -15.9532 to       5.78403
  -- At sub-brick #8 'sub005' datum type is float:     -7.85877 to        21.545
  -- At sub-brick #9 'sub005' datum type is float:     -11.0492 to       16.6702
  -- At sub-brick #10 'sub006' datum type is float:     -7.49449 to       7.94652
  -- At sub-brick #11 'sub006' datum type is float:     -10.1083 to       12.1575
  -- At sub-brick #12 'sub007' datum type is float:     -19.5165 to        33.166
  -- At sub-brick #13 'sub007' datum type is float:     -21.4625 to       28.2601

----- HISTORY -----
[data@data-Inspiron-7570: Fri Jan 12 23:32:01 2018] {AFNI_18.0.02:linux_ubuntu_16_64} 3dcopy delete.me00.nii.gz All_My_Subj.nii.gz

2)There was also something wrong when I ran @ClustExp_run_shiny

 ...
Attaching package: ‘shinydashboard’

The following object is masked from ‘package:graphics’:

    box

Error in library(plotly) : there is no package called ‘plotly’
Calls: <Anonymous> ... sourceUTF8 -> eval -> eval -> ..stacktraceon.. -> library
停止执行(stopping execution )

I think this error was caused by inappropriately installing some necessary R libraries. Because there were lots of warning messages when the command @afni_R_package_install - shiny was executed:

 ...
下载的程序包在
	‘/tmp/RtmpMEWPwg/downloaded_packages’里(Downloaded packages are in ‘/tmp/RtmpMEWPwg/downloaded_packages’)
Warning messages:
1: In install.packages(c("shiny", "shinydashboard", "plotly", "colourpicker",  :
  安装程序包‘openssl’时退出狀態的值不是0 (The value of the exit state when installing the package 'openssl' is not 0 )
2: In install.packages(c("shiny", "shinydashboard", "plotly", "colourpicker",  :
  安装程序包‘httr’时退出狀態的值不是0 (The value of the exit state when installing the package 'httr' is not 0 )
3: In install.packages(c("shiny", "shinydashboard", "plotly", "colourpicker",  :
  安装程序包‘plotly’时退出狀態的值不是0 (The value of the exit state when installing the package 'shiny' is not 0 )

But I don’t know how to fix these problems.
Could you help me?
Thank you in advance!

HI,
Sorry for the problems.

For the @afni_R_package_install -shiny
The default repository is https://cran.rstudio.com.
But if for some reason, you can’t reach that, you may have to install separately with another repository.
You can do that manually in R with:


install.packages(c('shiny','shinydashboard','plotly','colourpicker','data.table','gplots','RColorBrewer'))

It should give you an option to select a mirror, and choose one that is close to you.

It looks like “shiny” is the one giving you problems. It requires openssl and depending on your operating system, you may need to install it separately.
Some versions of linux need to install it with apt-get or yum or whatever your operating system uses.
If you are on linux, try installing openssl first then rerun


@afni_R_package_install -shiny

or do it manually as above.

For the ClustExp_StatParse.py error, can you email me your table.csv?
send to justin.rajendra@nih.gov
It looks to be some sort of formatting error, but I can’t tell from the message board post.

Thanks, Justin

Hello,

I’m having trouble running ClustExp_StatParse.py.

The below is my code.


ClustExp_StatParse.py -StatDSET stats.ttest+tlrc. -MeanBrik 0 -ThreshBrik 0 -SubjDSET All_clust+tlrc. -SubjTable header_was_the_problem.csv -master ~/abin/MNI_avg152T1+tlrc.

And this is the output.


Resampling master to subject data.

-- will fix any bad char check errors in /data2/open_data/ADNI/ADNI3/Group127/subject_analysis/left_pcc/group_results/test.002.3dttest++/test.results/MyOutput_ClustExp_shiny/data/MyOutput_CleanSubjTab.1D
** non-Unix check output file '/data2/open_data/ADNI/ADNI3/Group127/subject_analysis/left_pcc/group_results/test.002.3dttest++/test.results/MyOutput_ClustExp_shiny/data/MyOutput_CleanSubjTab.1D' exists, will overwrite
-- will fix any non-Unix check errors in /data2/open_data/ADNI/ADNI3/Group127/subject_analysis/left_pcc/group_results/test.002.3dttest++/test.results/MyOutput_ClustExp_shiny/data/MyOutput_CleanSubjTab.1D
Failed to find attribute BRICK_STATAUX with sub-brick selection.
Since BRICK_STATAUX exists in stats.ttest+tlrc. then it is an attribute that 
is not preserved with sub-brick selection.
Traceback (most recent call last):
  File "/home/bspl/abin/ClustExp_StatParse.py", line 457, in <module>
    dof = subprocess.check_output(dof_cmd,shell=True)
  File "/usr/lib/python2.7/subprocess.py", line 574, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '3dAttribute BRICK_STATAUX stats.ttest+tlrc.[0]' returned non-zero exit status 1

I’m struggling to understand this part “Failed to find attribute BRICK_STATAUX with sub-brick selection.”

My output from 3dttest++:


3dinfo stats.ttest+tlrc.
++ 3dinfo: AFNI version=AFNI_19.2.01 (Jul  3 2019) [64-bit]

Dataset File:    stats.ttest+tlrc
Identifier Code: AFN_7MQa8nMfzBhFl6bnXJEQ9g  Creation Date: Tue Jul 23 14:02:24 2019
Template Space:  MNI
Dataset Type:    Func-Bucket (-fbuc)
Byte Order:      LSB_FIRST [this CPU native = LSB_FIRST]
Storage Mode:    BRIK
Storage Space:   11,126,952 (11 million) bytes
Geometry String: "MATRIX(-2.5,0,0,90,0,-2.5,0,125.5,0,0,2.5,-72):73,87,73"
Data Axes Tilt:  Plumb
Data Axes Orientation:
  first  (x) = Left-to-Right
  second (y) = Posterior-to-Anterior
  third  (z) = Inferior-to-Superior   [-orient LPI]
R-to-L extent:   -90.000 [R] -to-    90.000 [L] -step-     2.500 mm [ 73 voxels]
A-to-P extent:   -89.500 [A] -to-   125.500 [P] -step-     2.500 mm [ 87 voxels]
I-to-S extent:   -72.000 [I] -to-   108.000 [S] -step-     2.500 mm [ 73 voxels]
Number of values stored at each pixel = 6
  -- At sub-brick #0 'control-MCI_mean' datum type is float:    -0.397327 to      0.550301
  -- At sub-brick #1 'control-MCI_Zscr' datum type is float:     -4.26816 to       4.46619
     statcode = fizt
  -- At sub-brick #2 'control_mean' datum type is float:    -0.273747 to       2.14475
** For info on all 6 sub-bricks, use '3dinfo -verb' **

Thank you.

Hi,

Try changing “-ThreshBrik 0” to “-ThreshBrik 1”
The program is looking for some statistic in the header.

I have not tested this with Zscr stats…
I am pretty sure it won’t work with Zscr as there is no degrees of freedom associated with it.
I can work on adding support for the z score and cleaning up the error message.
In the meantime, you could try running it 3dttest++ without the option you chose that makes it a z score rather than a t score.
I think those are the etac, clustsim, or toz options.

thanks, Justin

As you suspected, I received this output:


ERROR: only Tstats and FStats for now...Sorry!

I’ll experiment with tscores in the meantime.

Thanks for your time!

Dear Justin,
this morning we run Cluster Explorer with the results derived from one sample t-test executed with 3dttest++. It didn’t work, likely because the output was Zscr. We have tried to run it with an output based on t stat, but we obtained the following error message when executing @ClustExp_run_shiny:

Attaching package: ‘shinydashboard’
The following object is masked from ‘package:graphics’:
box

Loading required package: ggplot2
Attaching package: ‘plotly’
The following object is masked from ‘package:ggplot2’:
last_plot
The following object is masked from ‘package:stats’:
filter
The following object is masked from ‘package:graphics’:
layout
Attaching package: ‘psych’
The following objects are masked from ‘package:ggplot2’:
%+%, alpha
Loading required package: lme4
Loading required package: Matrix


Welcome to afex. For support visit: http://afex.singmann.science/

  • Functions for ANOVAs: aov_car(), aov_ez(), and aov_4()
  • Methods for calculating p-values with mixed(): ‘KR’, ‘S’, ‘LRT’, and ‘PB’
  • ‘afex_aov’ and ‘mixed’ objects can be passed to emmeans() for follow-up tests
  • NEWS: library(‘emmeans’) now needs to be called explicitly!
  • Get and set global package options with: afex_options()
  • Set orthogonal sum-to-zero contrasts globally: set_sum_contrasts()
  • For example analyses see: browseVignettes(“afex”)

Attaching package: ‘afex’
The following object is masked from ‘package:lme4’:
lmer
Error in factor(mean.df$Group, levels = c(lab.a, lab.b), ordered = TRUE) :
objeto ‘lab.a’ no encontrado
Calls: … sourceUTF8 → eval → eval → …stacktraceon… → factor
Ejecución interrumpida

Could you indicate me the potential source of my error?
Thanks a lot in advance.
Karel

Does it support 3dMEMA results?

Hi,
I am going to add the ability to check for and use zscores.
Hopefully soon.
For the 3dMEMA, I need to consult with Gang a bit so that may take more time.
thanks, Justin

Dear Justin,
this morning we run Cluster Explorer with the results derived from one sample t-test executed with 3dttest++. It didn’t work, likely because the output was Zscr. We have tried to run it with an output based on t stat, but we obtained the following error message when executing @ClustExp_run_shiny:

Attaching package: ‘afex’
The following object is masked from ‘package:lme4’:
lmer
Error in factor(mean.df$Group, levels = c(lab.a, lab.b), ordered = TRUE) :
objeto ‘lab.a’ no encontrado
Calls: … sourceUTF8 → eval → eval → …stacktraceon… → factor
Ejecución interrumpida

Could you indicate me the potential source of my error?
Thanks a lot in advance.
Karel

Hi,
I am going to split this thread into a new one for a cleaner topic.

Follow up here:
https://afni.nimh.nih.gov/afni/community/board/read.php?1,162101,162101#msg-162101

thanks, Justin