foreach error in @stim_analyze script

Hello AFNI experts,

I am encountering an issue when attempting to run a script based on @stim_analyze.
More specifically, when I run the script I get the following message:

** removing output directory, stim_results …
++ creating output directory, stim_results …
iteration (of 100): 0000nums: Subscript out of range.

The directory stim_results gets created, but I think the issue has to do with the foreach loop, as when I try to run: iter(count -digits 4 1 $iterations)
I get the error: Badly placed ()'s

Any idea what I am doing wrong?

Thanks a lot for your help!

Laura

++++++++++++++AFNI version++++++++++++++++

Precompiled binary linux_ubuntu_16_64: Apr 18 2018 (Version AFNI_18.1.05)

++++++++++++++ script +++++++++++++++++++++

/bin/tcsh

set num_stim = 4
set num_runs = 4
set pre_rest = 4 # min rest before first stim (for magnet steady state)
set post_rest = 10 # min rest after last stim (for trailing BOLD response)
set min_rest = 0.5 # minimum rest after each stimulus
set tr = 2.0 # used in 3dDeconvolve, if not make_random_timing.py

set stim_durs = 10
set stim_reps = 40
set run_lengths = 54
set labels = “cond1 cond2 cond3 cond4”

---------------------------------------------------------------------------

execution parameters

set iterations = 100 # number of iterations to compare
set seed = 1234567 # initial random seed
set outdir = stim_results # directory that all results are under
set LCfile = NSD_sums # file to store norm. std. dev. sums in

set pattern = LC # search pattern for LC[0], say

set pattern = ‘norm. std.’ # search pattern for normalized stdev vals

===========================================================================

start the work

===========================================================================

------------------------------------------------------------

recreate $outdir each time

if ( -d $outdir ) then
echo “** removing output directory, $outdir …”
\rm -fr $outdir
endif

echo “++ creating output directory, $outdir …”
mkdir $outdir
if ( $status ) then
echo “failure, cannot create output directory, $outdir”
exit
endif

move into the output directory and begin work

cd $outdir

create empty LC file

echo -n “” > $LCfile

echo -n “iteration (of $iterations): 0000”

------------------------------------------------------------

run the test many times

foreach iter(count -digits 4 1 $iterations)

    # make some other random seed

    @ seed = $seed + 1


    # create randomly ordered stimulus timing files
    # (consider: -tr_locked -save_3dd_cmd tempfile)

    make_random_timing.py -num_stim $num_stim -stim_dur $stim_durs  \
            -num_runs $num_runs # -run_time $run_lengths              \
            -num_reps $stim_reps -prefix stimes.$iter               \
            -pre_stim_rest $pre_rest -post_stim_rest $post_rest     \
            -min_rest $min_rest                                     \
            -stim_labels $labels                                    \
            -seed $seed                                             \
            -tr $tr                                                 \
            -show_timing_stats                                      \
            -save_3dd_cmd cmd.3dd.$iter                             \
                    >& out.mrt.$iter

    # consider: sed 's/GAM/"TENT(0,15,7)"/' tempfile > cmd.3dd.$iter
    #           rm -f tempfile

    # now evaluate the stimulus timings

    tcsh cmd.3dd.$iter >& out.3dD.$iter

    # save the sum of the 3 LC values
    set nums = ( `awk -F= '/'"$pattern"'/ {print $2}' out.3dD.${iter}` )

    # make a quick ccalc command
    set sstr = $nums[1]
    foreach num ( $nums[2-] )
        set sstr = "$sstr + $num"
    end
    set num_sum = `ccalc -expr "$sstr"`

    echo -n "$num_sum = $sstr : " >> $LCfile
    echo    "iteration $iter, seed $seed"                  >> $LCfile

    echo -n "\b\b\b\b$iter"

end

echo “”
echo “done, results are in ‘$outdir’, LC sums are in ‘$LCfile’”
echo consider the command: “sort -n $outdir/$LCfile | head -1”

Hi Laura,

You are missing -run_time in the make_random_timing.py command.
See out.mrt.0001, which has the error message.

  • rick

Hi Rick,

Thank you very much for the very quick and helpful response! The script works like a charm now, but I still have a couple of questions.

A bit of background concerning the study:

  • four conditions NMA NMS HMA HMS for a total of 160 stimuli
  • 4 functional runs with 40 stimuli each + 56 null events of 10 secs (14 x run)
  • TR = 2
  • each stimulus comprises a video clip of 10 s + a response time (2 s)
  • each stimulus starts with a fixation cross (500 - 4000m, jittered)

I am trying to employ @stim_analyze to maximize the efficiency of my design, but I have some questions I have been unable to solve:

  1. What is the best way to model the response time (2s) after each video? Shall I add this as a separate condition? (at the moment I included this in the stimuli duration -12s total- but I really do not think this is optimal)

  2. At the moment I did not model explicitly the null events or the fixation cross, but I simply “added” their duration in the run length. Is this correct?

  3. I am a bit at loss in interpreting the outcome of 1dplot X.xmat.1D (attached); I understand that the first 4 rows are the 4 conditions, and then there are six rows for each run, but I do not understand what the x axis measures and why six rows per run?

  4. out of curiosity: what exactly is a seed (i.e., how is the sequence randomization built from it)?

Thanks in advance for your valuable input.

Laura

############## SCRIPT ###############################

/bin/tcsh

try to find reasonable random event related timing given the experimental

parameters

---------------------------------------------------------------------------

some experiment parameters (most can be inserted directly into the

make_random_timing.py command)

set num_stim = 4
set num_runs = 4
set pre_rest = 0 # min rest before first stim (for magnet steady state)
set post_rest = 10 # min rest after last stim (for trailing BOLD response)
set min_rest = 0.5 # minimum rest after each stimulus
set max_rest = 4 # max rest after each stimulus
set tr = 2.0 # used in 3dDeconvolve, if not make_random_timing.py

(options that take multiple values can also take just one if they are

all the same, such as with this example)

set stim_durs = “12 12 12 12”
set stim_reps = “10 10 10 10” # = 40 stimuli x condition
set run_lengths = “690 690 690 690”
set labels = “NMA NMS HMA HMS”

---------------------------------------------------------------------------

execution parameters

set iterations = 1000 # number of iterations to compare
set seed = 1234567 # initial random seed
set outdir = /home/lama/stim_results # directory that all results are under
set LCfile = NSD_sums # file to store norm. std. dev. sums in

set pattern = LC # search pattern for LC[0], say

set pattern = ‘norm. std.’ # search pattern for normalized stdev vals

===========================================================================

start the work

===========================================================================

------------------------------------------------------------

recreate $outdir each time

if ( -d $outdir ) then
echo “** removing output directory, $outdir …”
\rm -fr $outdir
endif

echo “++ creating output directory, $outdir …”
mkdir $outdir
if ( $status ) then
echo “failure, cannot create output directory, $outdir”
exit
endif

move into the output directory and begin work

cd $outdir

create empty LC file

echo -n “” > $LCfile

echo -n “iteration (of $iterations): 0000”

------------------------------------------------------------

run the test many times

foreach iter(count -digits 4 1 $iterations)

    # make some other random seed

    @ seed = $seed + 1

    # create randomly ordered stimulus timing files
    # (consider: -tr_locked -save_3dd_cmd tempfile)

    make_random_timing.py -num_stim $num_stim -stim_dur $stim_durs  \
            -num_runs $num_runs  -run_time $run_lengths              \
            -num_reps $stim_reps -prefix stimes.$iter               \
            -pre_stim_rest $pre_rest -post_stim_rest $post_rest     \
            -min_rest $min_rest                                     \
            -max_rest $max_rest                                     \
            -stim_labels $labels                                    \
            -seed $seed                                             \
            -tr $tr                                                 \
            -show_timing_stats                                      \
            -save_3dd_cmd cmd.3dd.$iter                             \
                    >& out.mrt.$iter

    # consider: sed 's/GAM/"TENT(0,15,7)"/' tempfile > cmd.3dd.$iter
    #           rm -f tempfile

    # now evaluate the stimulus timings

    tcsh cmd.3dd.$iter >& out.3dD.$iter

    # save the sum of the 3 LC values
    set nums = ( `awk -F= '/'"$pattern"'/ {print $2}' out.3dD.${iter}` )

    # make a quick ccalc command
    set sstr = $nums[1]
    foreach num ( $nums[2-] )
        set sstr = "$sstr + $num"
    end
    set num_sum = `ccalc -expr "$sstr"`

    echo -n "$num_sum = $sstr : " >> $LCfile
    echo    "iteration $iter, seed $seed"                  >> $LCfile

    echo -n "\b\b\b\b$iter"

end

echo “”
echo “done, results are in ‘$outdir’, LC sums are in ‘$LCfile’”
echo consider the command: “sort -n $outdir/$LCfile | head -1”

echo check summary stats with timing_tool.py

note that if iter 0594 seems to be the best, consider these commands:

timing_tool.py -multi_timing stimes.${iter}_0*
-run_len $run_lengths -multi_stim_dur $stim_durs
-multi_show_isi_stats

run 3d deconvolve

tcsh cmd.3dd.$iter

plot results

1dplot X.xmat.1D ‘[6…$]’
1dplot sum_ideal.1D

Hi Laura,

A basic question: assuming you start each run with
the fixation cross, would it be fair to say those
fixations come AFTER each stimulus (as an ISI),
rather than before? make_random_timing.py is set
up to have rest events attached to stimulus events,
where the rest comes after the stimulus.

It should not really matter, but that is the design
of the timing.

With the old form of make_random_timing.py (which
is used in the @stim_analyze example), you can set a
min and max limits on the ISIs, but the distribution
of such events is not fantastic. The newer usage does
well with that (perhaps making it more complicated to
use). See make_random_timing.py -help_advanced .

Getting to your questions:

  1. It is difficult to disentangle the video event
    from the 2s response, since there is no jitter
    between them. You can model them separately, but
    the response betas will be noisy, as they fight
    with the video events.

In areas of the brain that respond to only the video
or only the response, it might work out okay. But
especially if the response is not really a 2s event,
but the subject is given 2s to do something, that may
or may not really take that long, then modeling them
separately might not work well.

  1. Yes, though the distribution of ISI in the older
    version isn’t as smooth as that from the newer one
    (advanced).

  2. The 6 extra regressors per run are polort drift
    terms. Since each run is 690s, 5th polynomials are
    used to model the drift, which include 6 terms.

The x-axis is time points. 690s/2s(TR) = 345, +1
gives 346 time points per run.

  1. Random numbers on a computer are never quite
    random, as they are generated from some algorithm,
    usually based on very large numbers (for example,
    they might use 1024 bits (probably old now) or
    about 300 digits), which are then reduced to the
    range a user wants.

The seed is an initial number in the algorithm.
After that, the same sequence of pseudo-random
numbers should come out. This helps us to test
such software.

Anyway, the seeds are not so necessary here, but
with the seed and the MRT.py command, you can
reproduce the same output. It is more useful in
regression testing.

  • rick

Hi Laura,

I will try to rewrite your timing with the modern usage,
just for demo. However, I do not see in your example
where the NULL events are.

Each run is 690s, of which 480 are for task, 140 for
NULL trials, and 10 for post stim time. That accounts
for 630s, leaving only 60s for ISI. Attaching the ISI
to the 40 non-NULL events per run makes the average ISI
1.5s.

So consider such things in this MRT.py command:

set seed = 1234567
set iter = 001

make_random_timing.py -num_runs 4 -run_time 690 \
        -tr 2                                   \
        -pre_stim_rest 0 -post_stim_rest 10     \
        -rand_post_stim_rest no                 \
        -add_timing_class stim 12               \
        -add_timing_class tnull 10              \
        -add_timing_class rest 0.5 1.5 4 dist=decay_fixed \
        -add_stim_class NMA 10 stim rest        \
        -add_stim_class NMS 10 stim rest        \
        -add_stim_class HMA 10 stim rest        \
        -add_stim_class HMS 10 stim rest        \
        -add_stim_class NULL 14 tnull INSTANT   \
        -write_event_list events.$iter.txt      \
        -show_timing_stats                      \
        -seed $seed                             \
        -prefix threat_app.$iter                \
        -save_3dd_cmd cmd.3dd.$iter.txt

Note that the NULL events might have ISI time before
them, but not after, as the attached ISI is INSTANT
(meaning zero time).

View the text file events.001.txt for the list of events
across all runs.

I will put this in @stim_analyze_modern at the same
location, just as an example. I will probably updated
it to be more complicated later. Maybe.

  • rick

Dear Rick,

Thank you for this! It was very clear and helpful.
To answer you previous comment, yes it is ok to consider the rests as attached after the stimuli.

Kind regards,
Laura

Dear Rick,

I tried to implement the make_random_timing script within the @stim_analyze script (see below), but when I run it I get the following message.

** removing output directory, /home/xxx/stim_results …
++ creating output directory, /home/xxx/stim_results …
iteration (of 5): 0000nums
nums: Subscript out of range.

What am I missing?

Thanks a lot!


/bin/tcsh

try to find reasonable random event related timing given the experimental

parameters

---------------------------------------------------------------------------

some experiment

set num_stim = 4
set num_runs = 4
set pre_rest = 0 # min rest before first stim (for magnet steady state)
set post_rest = 10 # min rest after last stim (for trailing BOLD response)
set min_rest = 0.5 # minimum rest after each stimulus
set max_rest = 4 # max rest after each stimulus
set tr = 2.0 # used in 3dDeconvolve, if not make_random_timing.py

set stim_durs = “12 12 12 12”
set stim_reps = “10 10 10 10” # = 40 stimuli x condition
set run_lengths = “690 690 690 690”
set labels = “NMA NMS HMA HMS”

---------------------------------------------------------------------------

execution parameters

set iterations = 5 # number of iterations to compare
set seed = 1234567 # initial random seed
set outdir = /home/xxx/stim_results # directory that all results are under
set LCfile = NSD_sums # file to store norm. std. dev. sums in

set pattern = LC # search pattern for LC[0], say

set pattern = ‘norm. std.’ # search pattern for normalized stdev vals

===========================================================================

start the work

===========================================================================

------------------------------------------------------------

recreate $outdir each time

if ( -d $outdir ) then
echo “** removing output directory, $outdir …”
\rm -fr $outdir
endif

echo “++ creating output directory, $outdir …”
mkdir $outdir
if ( $status ) then
echo “failure, cannot create output directory, $outdir”
exit
endif

move into the output directory and begin work

cd $outdir

create empty LC file

echo -n “” > $LCfile

echo -n “iteration (of $iterations): 0000”

------------------------------------------------------------

run the test many times

foreach iter(count -digits 4 1 $iterations)

    # make some other random seed

    @ seed = $seed + 1

    # create randomly ordered stimulus timing files
    # (consider: -tr_locked -save_3dd_cmd tempfile)

    make_random_timing.py \
            -num_runs $num_runs \
            -run_time $run_lengths                            \
            -tr $tr                                           \
            -pre_stim_rest $pre_rest                          \
            -post_stim_rest $post_rest                        \
            -rand_post_stim_rest no                           \
            -add_timing_class stim 12                         \
            -add_timing_class tnull 10                        \
            -add_timing_class rest 0.5 1.5 4 dist=decay_fixed \
            -add_stim_class NMA 10 stim rest                  \
            -add_stim_class NMS 10 stim rest                  \
            -add_stim_class HMA 10 stim rest                  \
            -add_stim_class HMS 10 stim rest                  \
            -add_stim_class NULL 14 tnull INSTANT             \
            -max_consec 2 2 2 2 1 \
            -write_event_list events.$iter.txt                \
            -show_timing_stats                                \
            -seed $seed                                       \
            -prefix threat_app.$iter                          \
            -save_3dd_cmd cmd.3dd.$iter.txt                   \
              >& out.mrt.$iter \
    # consider: sed 's/GAM/"TENT(0,15,7)"/' tempfile > cmd.3dd.$iter
    #           rm -f tempfile

    # now evaluate the stimulus timings

    tcsh cmd.3dd.$iter >& out.3dD.$iter

    # save the sum of the 3 LC values
    set nums = ( `awk -F= '/'"$pattern"'/ {print $2}' out.3dD.${iter}` )

    # make a quick ccalc command
    set sstr = $nums[1]
    foreach num ( $nums[2-] )
        set sstr = "$sstr + $num"
    end
    set num_sum = `ccalc -expr "$sstr"`

    echo -n "$num_sum = $sstr : " >> $LCfile
    echo    "iteration $iter, seed $seed"                  >> $LCfile

    echo -n "\b\b\b\b$iter"

end

echo “”
echo “done, results are in ‘$outdir’, LC sums are in ‘$LCfile’”
echo consider the command: “sort -n $outdir/$LCfile | head -1”

echo check summary stats with timing_tool.py

Hi Laura,

What is the contents of each file: out.mrt.0000 and out.3dD.0000?

Thanks,

  • rick

Hi Rick,

Output of out.mrt.0001 is:

++ applying max_consec to: NMA, NMS, HMA, HMS, NULL
saving 3dD command to file ‘cmd.3dd.0001.txt’…

ISI statistics (5 elements) :

                    total      per run
                   ------      ------------------------------
total time         2760.0       690.0    690.0    690.0    690.0  
total time: stim   2480.0       620.0    620.0    620.0    620.0  
total time: rest    280.0        70.0     70.0     70.0     70.0  

rest: total isi     240.0        60.0     60.0     60.0     60.0  
rest: pre stim        0.0         0.0      0.0      0.0      0.0  
rest: post stim      40.0        10.0     10.0     10.0     10.0  

num stimuli         216          54       54       54       54  


                     min      mean     max     stdev
                   -------  -------  -------  -------
rest: pre-stim       0.000    0.000    0.000    0.000
rest: post-stim      9.990    9.998   10.000    0.005

rest: run #0 ISI     0.000    1.132    3.790    1.008
rest: run #1 ISI     0.000    1.132    3.790    1.008
rest: run #2 ISI     0.000    1.132    3.790    1.008
rest: run #3 ISI     0.000    1.132    3.770    0.981

all runs: ISI        0.000    1.132    3.790    0.994
all runs: stimuli   10.000   11.481   12.000    0.878

within-TR stimulus offset statistics :
per run
------------------------------
offset means 1.011 0.909 0.908 1.065
offset stdevs 0.666 0.557 0.575 0.621

overall:     mean = 0.973  maxoff = 2.000  stdev = 0.6056
fractional:  mean = 0.487  maxoff = 1.000  stdev = 0.3028

output of out.3dD.0001 is:

cmd.3dd.0001: No such file or directory.


Thank you for your help!

Best,
Laura

Hi Laura,

The make_random_timing.py command has option

-save_3dd_cmd cmd.3dd.$iter.txt

But the subsequent command to execute it is:

tcsh cmd.3dd.$iter >& out.3dD.$iter

That ‘tcsh’ command is missing the ‘.txt’ extension to
the 3dDeconvolve script, cmd.3dd.$iter.txt .

  • rick

Of course! Thank you so much for your help, have a great week end!
Best,
Laura