Body
To use all memory on an exclusive access node, please add the following to your Slurm job:
--exclusive --mem=0
In Slurm, --exclusive prevents other running jobs from sharing the allocated node. It allocates all CPUs and GPUs but it does not allocate all memory on the node. This is by design as stated in the Slurm documentation for --exclusive:
Slurm jobs using --exclusive are allocated all compute resources on all nodes in the allocation, but is only allocated as much memory as it requested. This is by design to support gang scheduling, because suspended jobs still reside in memory. To request all the memory on a node, use --mem=0.
https://slurm.schedmd.com/salloc.html#OPT_exclusive