I was right! I'll leave my initial response for the record but I was able to fix this issue by setting export AFNI_DETACH=NO.
Thanks for the quick response.
I tried setting AFNI_ATLAS_PATH to the path to afni_atlases_dist (which I downloaded from 11.3. Other templates and atlases — AFNI, SUMA and FATCAT: v24.1.22), and this improves the system check slightly to include
atlas : found TT_N27+tlrc under /afni/afni_atlases_dist
(note that is the filepath inside the container). However this does not change afni from crashing when run via apptainer exec, whereas it still runs fine with apptainer shell. I still suspect it's related to afni detaching from the terminal and thus "forgetting" about the container's filesystem.
To test this, I watched ps aux every 0.1 seconds when running afni via apptainer shell vs apptainer exec and might have caught what I was suspecting? When running with apptainer exec, it starts off with something like
670 lexg 3664824 0.0 0.0 7256 3568 pts/72 S+ 11:52 0:00 /bin/bash /apps/afni/24.1.22/bin/afni
671 lexg 3664834 2.6 0.0 1253356 25684 pts/72 Sl+ 11:52 0:00 Apptainer runtime parent
672 lexg 3664861 1.3 0.0 85200 4892 pts/72 S+ 11:52 0:00 /opt/afni/install/afni
673 lexg 3664876 30.3 0.0 377236 14808 pts/72 Sl+ 11:52 0:00 /usr/libexec/apptainer/bin/squashfuse_ll -f -o uid=15673,gid=10025,offset=40960 /proc/self/fd/3 /var/apptainer/mnt/session/rootfs
674 lexg 3664881 9.0 0.0 6708 4696 pts/72 S+ 11:52 0:00 /usr/bin/fuse-overlayfs -f -o lowerdir=/var/apptainer/mnt/session/overlay-lowerdir:/var/apptainer/mnt/session/rootfs,noacl /var/a
674 pptainer/mnt/session/final
675 lexg 3664919 4.5 0.0 92160 8948 pts/72 S+ 11:52 0:00 /opt/afni/install/afni
Note the /usr/bin/fuse-overlayfs is running concurrently with afni. But just before it crashes, I see only
720 lexg 3664824 0.0 0.0 7256 3568 pts/72 S+ 11:52 0:00 /bin/bash /apps/afni/24.1.22/bin/afni
721 lexg 3664834 0.0 0.0 20892 5484 pts/72 S+ 11:52 0:00 Apptainer runtime parent
722 lexg 3664850 0.0 0.0 19184 3704 pts/74 R+ 11:52 0:00 ps aux
723 lexg 3664851 0.0 0.0 6412 2256 pts/74 S+ 11:52 0:00 grep lexg
724 lexg 3664853 0.0 0.0 1121700 20212 pts/72 Sl+ 11:52 0:00 Apptainer runtime parent
without /usr/bin/fuse-overlayfs. Comparing when running with apptainer shell, the very last ps aux before I quit the GUI after having it open for ~10 seconds is simply
lexg 3665751 1.0 0.0 1253100 24984 pts/72 Sl 11:55 0:00 Apptainer runtime parent
lexg 3665777 0.4 0.0 20588 4116 pts/72 S+ 11:55 0:00 /bin/bash --norc
lexg 3665790 12.0 0.0 302340 17520 pts/72 Sl 11:55 0:01 /usr/libexec/apptainer/bin/squashfuse_ll -f -o uid=15673,gid=10025,offset=40960 /proc/self/fd/3 /var/apptainer/mnt/session/rootfs
lexg 3665795 3.5 0.0 7048 4672 pts/72 S 11:55 0:00 /usr/bin/fuse-overlayfs -f -o lowerdir=/var/apptainer/mnt/session/overlay-lowerdir:/var/apptainer/mnt/session/rootfs,noacl /var/apptainer/mnt/session/final
lexg 3665807 0.7 0.0 7256 3908 pts/74 S+ 11:55 0:00 /bin/bash ./watch-procs.sh output-shell
lexg 3665853 4.6 0.0 104828 26356 pts/72 S 11:55 0:00 afni
so the /usr/bin/fuse-overlayfs was still running to the very end. I understand that if this really is the issue, then this is probably more of a question for apptainer experts at this point...
But nevermind I have found a solution! Originally I only saw a list of AFNI environment variables at 3.5. List of all AFNI environment variables — AFNI, SUMA and FATCAT: v22.3.03. Neither of those mentions detaching. But I searched the afni source code for "detach" and found afni/src/afni_startup_tips.h at bbcac392ad05617a409ed41505ea8a7efc924f58 · afni/afni · GitHub, which revealed there was an environment variable to control whether afni detaches from the terminal or not called AFNI_DETACH. Setting AFNI_DETACH=NO resolved my issue! I then saw that this was indeed documented at 3.6. List of all startup tips — AFNI, SUMA and FATCAT: v22.3.03.
I think the "5.3. List of all AFNI environment variables" page should be updated to include AFNI_DETACH, otherwise the "all" is a bit misleading.
Thanks anyway for your help dglen!