Pass dset names to 3dMEMA through a variable or text file

AFNI Message Board
1

Hello Afni gurus!

I want to pass my subject list and dset paths to 3dMEMA through a variable or text file so that it can be modified more easily if, say, my subject list changes.

This is what I'm trying right now:

foreach sid ($Parts)
    set subjPath = "${modeldir}/sub-${sid}/stats.sub-${sid}_X_${model}_REML+${view}.HEAD"

    echo "sub-${sid} $subjPath1'[${contrast}#${covnum}_Coef]' $subjPath2'[${contrast}#${covnum}_Tstat]'" >> ${outdir}/${model}_${contrast}_SubjList.txt
end

#Resorted to this two-step approach because building SubjList within the loop was leading to tcsh trying to expand paths and giving the "No match" error
set SubjList = `cat ${outdir}/${model}_${contrast}_SubjList.txt`
3dMEMA -prefix ${contrast}+${view}                                                          \
      -mask ${projectpath}/derivatives/AFNI_04_GroupLevel/MNI152_template_Resampled.nii     \
      -residual_Z                                                                           \
      -set ${contrast}                                                                      \
      "$SubjList"

But this gives a dset not found error:

** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-001/stats.sub-001_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-002/stats.sub-002_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-003/stats.sub-003_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-004/stats.sub-004_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-005/stats.sub-005_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-006/stats.sub-006_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-007/stats.sub-007_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-008/stats.sub-008_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-009/stats.sub-009_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-011/stats.sub-011_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-012/stats.sub-012_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-013/stats.sub-013_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-014/stats.sub-014_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-015/stats.sub-015_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-016/stats.sub-016_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-017/stats.sub-017_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-018/stats.sub-018_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-019/stats.sub-019_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-020/stats.sub-020_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-021/stats.sub-021_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-023/stats.sub-023_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-024/stats.sub-024_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-025/stats.sub-025_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-026/stats.sub-026_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-027/stats.sub-027_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-028/stats.sub-028_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-029/stats.sub-029_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-030/stats.sub-030_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-031/stats.sub-031_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-032/stats.sub-032_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-001/stats.sub-001_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-002/stats.sub-002_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-003/stats.sub-003_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-004/stats.sub-004_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-005/stats.sub-005_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-006/stats.sub-006_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-007/stats.sub-007_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-008/stats.sub-008_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-009/stats.sub-009_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-011/stats.sub-011_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-012/stats.sub-012_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-013/stats.sub-013_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-014/stats.sub-014_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-015/stats.sub-015_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-016/stats.sub-016_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-017/stats.sub-017_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-018/stats.sub-018_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-019/stats.sub-019_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-020/stats.sub-020_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-021/stats.sub-021_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-023/stats.sub-023_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-024/stats.sub-024_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-025/stats.sub-025_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-026/stats.sub-026_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-027/stats.sub-027_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-028/stats.sub-028_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-029/stats.sub-029_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-030/stats.sub-030_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-031/stats.sub-031_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-032/stats.sub-032_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded

But the dset exists, and the sub-brick name is correct:

3dinfo /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-032/stats.sub-032_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]'

++ 3dinfo: AFNI version=AFNI_23.1.00 (Apr  7 2023) [64-bit]

Dataset File:    stats.sub-032_X_Exp_UniCat_REML+tlrc
Identifier Code: XYZ_9qMtlX2z98kSSejWbDv59A  Creation Date: Mon Jan 22 16:15:21 2024
Template Space:  MNI
Dataset Type:    Func-Bucket (-fbuc)
Byte Order:      LSB_FIRST [this CPU native = LSB_FIRST]
Storage Mode:    BRIK
Storage Space:   4,202,496 (4.2 million) bytes
Geometry String: "MATRIX(-2,0,0,95,0,-2,0,131,0,0,2,-77):96,114,96"
Data Axes Tilt:  Plumb
Data Axes Orientation:
  first  (x) = Left-to-Right
  second (y) = Posterior-to-Anterior
  third  (z) = Inferior-to-Superior   [-orient LPI]
R-to-L extent:   -95.000 [R] -to-    95.000 [L] -step-     2.000 mm [ 96 voxels]
A-to-P extent:   -95.000 [A] -to-   131.000 [P] -step-     2.000 mm [114 voxels]
I-to-S extent:   -77.000 [I] -to-   113.000 [S] -step-     2.000 mm [ 96 voxels]
Number of values stored at each pixel = 1
  -- At sub-brick #0 'Face#0_Tstat' datum type is float:     -6.15902 to       15.9049
     statcode = fitt;  statpar = 261

Is it possible for 3dMEMA to get the subject and filenames from a variable? It seems like there's an easy solution to this. But I've tried all manner of quotes and backticks and have been getting different errors!

Thank you,
Mrinmayi

2

Hi Mrinmayi,

Note that the error message still has single quotes in the file names. You might try removing the double quotes around "$SubjList" in the 3dMEMA command.

But perhaps it might be more generally useful to explore gen_group_command.py, with which you can initialize the list using wildcards. If you don't want quite all of the subjects matched with a wildcard, consider something like example 7 to restrict the list.

  • rick
1 Like