Pass dset names to 3dMEMA through a variable or text file

Hello Afni gurus!

I want to pass my subject list and dset paths to 3dMEMA through a variable or text file so that it can be modified more easily if, say, my subject list changes.

This is what I'm trying right now:

foreach sid ($Parts)
    set subjPath = "${modeldir}/sub-${sid}/stats.sub-${sid}_X_${model}_REML+${view}.HEAD"

    echo "sub-${sid} $subjPath1'[${contrast}#${covnum}_Coef]' $subjPath2'[${contrast}#${covnum}_Tstat]'" >> ${outdir}/${model}_${contrast}_SubjList.txt
end

#Resorted to this two-step approach because building SubjList within the loop was leading to tcsh trying to expand paths and giving the "No match" error
set SubjList = `cat ${outdir}/${model}_${contrast}_SubjList.txt`
3dMEMA -prefix ${contrast}+${view}                                                          \
      -mask ${projectpath}/derivatives/AFNI_04_GroupLevel/MNI152_template_Resampled.nii     \
      -residual_Z                                                                           \
      -set ${contrast}                                                                      \
      "$SubjList"

But this gives a dset not found error:

** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-001/stats.sub-001_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-002/stats.sub-002_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-003/stats.sub-003_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-004/stats.sub-004_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-005/stats.sub-005_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-006/stats.sub-006_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-007/stats.sub-007_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-008/stats.sub-008_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-009/stats.sub-009_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-011/stats.sub-011_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-012/stats.sub-012_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-013/stats.sub-013_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-014/stats.sub-014_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-015/stats.sub-015_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-016/stats.sub-016_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-017/stats.sub-017_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-018/stats.sub-018_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-019/stats.sub-019_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-020/stats.sub-020_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-021/stats.sub-021_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-023/stats.sub-023_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-024/stats.sub-024_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-025/stats.sub-025_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-026/stats.sub-026_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-027/stats.sub-027_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-028/stats.sub-028_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-029/stats.sub-029_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-030/stats.sub-030_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-031/stats.sub-031_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-032/stats.sub-032_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Coef]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-001/stats.sub-001_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-002/stats.sub-002_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-003/stats.sub-003_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-004/stats.sub-004_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-005/stats.sub-005_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-006/stats.sub-006_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-007/stats.sub-007_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-008/stats.sub-008_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-009/stats.sub-009_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-011/stats.sub-011_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-012/stats.sub-012_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-013/stats.sub-013_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-014/stats.sub-014_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-015/stats.sub-015_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-016/stats.sub-016_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-017/stats.sub-017_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-018/stats.sub-018_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-019/stats.sub-019_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-020/stats.sub-020_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-021/stats.sub-021_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-023/stats.sub-023_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-024/stats.sub-024_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-025/stats.sub-025_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-026/stats.sub-026_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-027/stats.sub-027_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-028/stats.sub-028_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-029/stats.sub-029_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-030/stats.sub-030_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-031/stats.sub-031_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded
** ERROR: Dset /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-032/stats.sub-032_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]' could not be loaded

But the dset exists, and the sub-brick name is correct:

3dinfo /raid-09/LS/hannula/mrinmayi//Data/ConcealMemory//derivatives/AFNI_03_3dDeconvolve/Exp_UniCat/sub-032/stats.sub-032_X_Exp_UniCat_REML+tlrc.HEAD'[Face#0_Tstat]'

++ 3dinfo: AFNI version=AFNI_23.1.00 (Apr  7 2023) [64-bit]

Dataset File:    stats.sub-032_X_Exp_UniCat_REML+tlrc
Identifier Code: XYZ_9qMtlX2z98kSSejWbDv59A  Creation Date: Mon Jan 22 16:15:21 2024
Template Space:  MNI
Dataset Type:    Func-Bucket (-fbuc)
Byte Order:      LSB_FIRST [this CPU native = LSB_FIRST]
Storage Mode:    BRIK
Storage Space:   4,202,496 (4.2 million) bytes
Geometry String: "MATRIX(-2,0,0,95,0,-2,0,131,0,0,2,-77):96,114,96"
Data Axes Tilt:  Plumb
Data Axes Orientation:
  first  (x) = Left-to-Right
  second (y) = Posterior-to-Anterior
  third  (z) = Inferior-to-Superior   [-orient LPI]
R-to-L extent:   -95.000 [R] -to-    95.000 [L] -step-     2.000 mm [ 96 voxels]
A-to-P extent:   -95.000 [A] -to-   131.000 [P] -step-     2.000 mm [114 voxels]
I-to-S extent:   -77.000 [I] -to-   113.000 [S] -step-     2.000 mm [ 96 voxels]
Number of values stored at each pixel = 1
  -- At sub-brick #0 'Face#0_Tstat' datum type is float:     -6.15902 to       15.9049
     statcode = fitt;  statpar = 261

Is it possible for 3dMEMA to get the subject and filenames from a variable? It seems like there's an easy solution to this. But I've tried all manner of quotes and backticks and have been getting different errors!

Thank you,
Mrinmayi

Hi Mrinmayi,

Note that the error message still has single quotes in the file names. You might try removing the double quotes around "$SubjList" in the 3dMEMA command.

But perhaps it might be more generally useful to explore gen_group_command.py, with which you can initialize the list using wildcards. If you don't want quite all of the subjects matched with a wildcard, consider something like example 7 to restrict the list.

  • rick