I’m running 3dDeconvolve on my dataset and it often got killed. The dataset is quite large. It has 32 runs of 10 min scanning, each run is ~400MB.
I run it on different computers and it seems not solely depend on the size of memory whether the process will be killed.
Here are my questions:
why is it that some computers have enough memory and hard drive but still got killed? For example, I ran this on two computers, both have 64GB memory and >500GB hard drive. One always ran successfully and another always got killed. What could be the reason, the age of computers?
Is there anything I can do to reduce the memory it uses so I can run it on computers with smaller memory?
Currently the process takes whole 5 days and I’m running a deadline, is there any way to speed it up?
Could 3) and 2) be solved using the parallel job option in the 3dDeconvolve? My computers usually have 4 cores, so I should use -jobs 4. But will this require each of the core have a large memory?
If the time series are being aligned to the anatomy or to a template, then they might occupy larger boxes, and therefor take more space. It might be good to verify the command that this fails with, along with the sizes of the datasets that are input to it.
Computers can have different amounts of swap space and different handling of it. A computer that allows for more swap space might not kill the process, but instead take a veeeeery long time to complete. Also, it is good to verify what else might be running on the computers, such as Microsoft office applications, web browsers, MATLAB, etc. Some of them can use a lot of RAM.
If it is 3dDeconvolve that fail, and you are using afni_proc.py for the analysis, add the afnI_proc.py option -regress_compute_fitts, which will save possibly 40% of RAM during the regression.
Anyway, can you verify the failing command, along with the sizes of inputs to it?
How much RAM do the computes have?
What operating systems are on those machines?
If I’m using afni_proc.py, it failed at the regress step. Sometimes it looks something like
voxel loop: 012
killed
If I’m using 3dDeconvolve, it’s the same situation.
The input to this step is usually around 32GB combining all the pb04* files. And the computers I’m running on are all iMac computers with MacOS. I try to run them on 64GB RAM machines, some succeeded and some failed. I don’t understand why it took 5 whole days to finish. I checked the activity monitor and it seems there is plenty of RAM free…And if possible, I really want to run them on 32GB machines too.
I’ll try the option -regress_compute_fitts with afni_procy.py. If I started with 3dDeconvolve, is there any way to reduce the RAM it used?
BTW, on the related note: every time I try to add some new contrasts to my analysis, I need to ran 3dDeconvolve with all the stim and glt labels. I don’t know if it’s the single stim labels or the glt contrasts that are taking so much time to fit. Is there any way to just add a new glt contrast quickly?
FYI, the input pb04* for the 3dDeconvolve step is ~32GB. It has 32 runs, each has a size of around 1GB.
I used the -mask option so the analysis should be performed within the brain. I’m considering using 3dZcutup to allow me to run them in smaller RAM computers…
The -regress_compute_fitts option should cut down RAM usage by about 40%, so definitely start with that.
If the input to 3dDeconvolve is 32 GB, consider that it stores the input, errts, fitts, betas and statistics together, and removing the fitts from that list can really help.
Yes, 3dZcutup can help, but that is much more work for you.
To repeat the regression quickly when adding GLTs, look into the -xsave and -xrestore options. It is not necessary to re-run the entire regression.
The reason it can be so slow when you run out of memory is due to the swapping issue that I mentioned. To use additional RAM, the OS writes out part of memory to swap space (on disk) and reads it back in when necessary. If this happens too much, it is called thrashing, when the application spends most of its time writing to and reading from disk, rather than actually performing computations.
Be sure to close any memory intensive application when this runs too, such as MATLAB, a web browser, or any Windows Office program.
Don’t just close the window, but formally Quit the process (right-click the App in the dock and choose Quit).
Thank you for your helpful advice! I’ll definitely try to use the options to speed up this process.
Just one more question: if I directly ran 3dDeconvolve without running afni_proc.py, what’s the way as equivalent to -regress_compute_fitts? Is it simply deleting the -fitts option?
Yes, you can just remove the -fitts option. But the proc script would still compute the fitts, but by using 3dcalc, something like:
3dcalc -a all_runs… -b errts… -expr a-b -prefix fitts
That would be run after 3dDeconvolve, and would produce the same result, subject to tiny truncation differences (or maybe not, since they are now float datasets?).
Got it. Thanks Rick. All of this is very helpful!!
The
National Institute of Mental Health (NIMH) is part of the National Institutes of
Health (NIH), a component of the U.S. Department of Health and Human
Services.