Dear AFNI team,
our Department finally received a nice server we can use for fMRI computations. The server has 120 CPUs. We tried to process a single subject using afni_proc.py with either 4 or 40 CPUs, but the total time needed to complete the script was very similar. Is there a way to exploit the full potential of the server to process multiple subjects in parallel or to speed up single subjects’ processing.
Dear AFNI team,
It’s true that throwing more processors at each subject with afni_proc.py has limited impacts on overall processing time. Some programs (3dREMLfit, 3dClustSim, 3dAllineate, 3dQwarp) do take advantage of OpenMP for using multiple processors. You can set how many threads that they use with the environmental variable OMP_NUM_THREADS, with the sweet spot (in my experience) being around 12 CPUs. However plenty of other parts of the pipeline are single threaded, which is what’s really slowing you down, in part due to all the disk read/write (IO).
The approach we take here is to batch process multiple subjects at the same time! We use a computer scheduler (e.g. swarm, sbatch) but almost any processor solution like this will work. If that’s a bit beyond what you’re comfortable with - I highly recommend GNU Parallel[/url]. Which allows you to submit jobs all at once or to submit them and then return to the command line. Which can allow you to do things like [url=https://blog.cogneurostats.com/2013/06/10/parallelizing-freesurfer/]Parellel Freesurfer.
Other sites I know use Sun Grid Engine (or the open source equivalents). Things like sbatch, SGE, etc have the advantage of showing you where all things are in processing at a given moment or how many cores/threads you’re making use of.
Finally, another way to speed things up is to copy your data from spinning slow hard drives (or network shares) to the server’s internal hard drive, or better yet SSD.