August 7, 2026
Queuing system (Slurm)
Slurm (Simple Linux Universal Resource Manager) is a scheduler that allocates resources to the submitted job; therefore, all jobs on ERISXdl should be submitted through the SLURM scheduler system. For more information on using ERISXdl, see the Getting Started article.
Partitions
Slurm’s partitions are similar to ‘queues’ in other job schedulers like LSF on Scientific Computing (SciC) Linux Clusters. Each partition has its dedicated resources such as the number of nodes, run time, GPU, CPU, memory, etc.
To view the list of available partitions, execute the command:
$ sinfoA summary of the partitions
The Basic partition is a free tier for registered users only (cf. Application Procedure for the ERISXdl platform described here).
| Partition | GPU limit | Max time limit | Memory limit |
| Basic | 8 GPU | 10 min | 400G |
| Short | 8 GPU | 2 hour | 400G |
| Medium | 8 GPU | 1 day | 400G |
| Long | 8 GPU | 5 days | 400G |
| Mammoth | 8 GPU | 2 weeks | 400G |
* NOTE : i) Following further monitoring of job loads, a QOS (Quality of Service) will be introduced to implement suitable GPU and Memory Limits.
ii) For partitions which allow multinode jobs simply specify the total number of GPUs required, kubernetes/slurm will subsequently allocate the required number of nodes.
For additional info on a specific partition, execute command:
$ sinfo --long -p <partition_name>
There are several GPU nodes that accept jobs from all the partitions. The state of these 5 nodes dgx-[1-5] at any given time can be inspected with the following:
$ scontrol show nodes
Submitting jobs
Firstly, a bash script file is created containing #SBATCH values specifying resources for the Slurm batch job. Some typical settings are shown below:
| Job Name | #SBATCH --job-name=My-Job_Name |
| Wall time hours | #SBATCH --time=24:0:0 or -t[days-hh:min:sec] |
| Number of nodes | #SBATCH --nodes=1 |
| Number of proc per node | #SBATCH --ntasks-per-node=24 |
| Number of cores per task | #SBATCH --cpus-per-task=24 |
| Number of GPU | #SBATCH --gpus=3 |
| Send mail at end of the job | #SBATCH --mail-type=end |
| User's email address | #SBATCH --mail-user=@email |
| Working Directory | #SBATCH --workdir=dir-name |
| Job Restart | #SBATCH --requeue |
| Share Nodes | #SBATCH --shared |
| Dedicated nodes | #SBATCH --exclusive |
| Memory Size | #SBATCH --mem=[mem |M|G|T] or --mem-per-cpu |
| Account to Charge | #SBATCH --account=[account] (*Not required, unless you are associated with several accounts and need to specify one in particular*) |
| Partition | #SBATCH --partition=[name] |
| Quality of Service | #SBATCH --qos=[name] (*Not required and should be omitted*) |
| Job Arrays | #SBATCH --array=[array_spec] |
| Use specific resource | #SBATCH --constraint="XXX" |
Secondly, the Slurm batch job is submitted to the job scheduler with the following:
$ sbatch <path-to-script>
and where after submitting your jobs you can confirm the status of your job with
$ sacctor
$ squeue
View more verbose job status:
$ squeue -j <job_ID>
Check job in detail:
$ scontrol show job <job_ID>
Slurm Job status, code, and explanation
When you request status information of your job you can get one of the following:
|
COMPLETED |
CD |
The job has completed successfully. |
|
COMPLETING |
CG |
The job is finishing but some processes are still active. |
|
FAILED |
F |
The job terminated with a non-zero exit code and failed to execute. |
|
PENDING |
PD |
The job is waiting for resource allocation. It will eventually run. |
|
PREEMPTED |
PR |
The job was terminated because of preemption by another job. |
|
RUNNING |
R |
The job currently is allocated to a node and is running. |
|
SUSPENDED |
S |
A running job has been stopped with its cores released to other jobs. |
|
STOPPED |
ST |
A running job has been stopped with its cores retained. |
The job can be canceled or killed; execute the command:
$ scancel <jobID>
Common commands in Slurm vs. LSF
|
Slurm |
LSF |
Explanation |
|
sbatch |
bsub |
Submit job |
|
sinfo |
bqueues |
List queues |
|
spart <partition_name> |
bqueues -l <queue name> |
View queue in details |
|
squeue |
bjobs -u all |
List all jobs status |
|
scontrol show job <jobid> |
bjobs -l <jobID> |
Check job in details |
|
scancel |
bkill |
Cancel or kill job |
FairShare Job Scheduling
Jobs submitted to the Slurm scheduler will be queued according to a FairShare algorithm (with further insight described here). Users can examine their current FairShare score using
sshare -u <userID> -land for a comparative overview of all users
sshare -a -lExample SLURM Job Submissions
Examples of Slurm jobs can be found here and where we note the following in the context of ERISXdl:
- When running Apptainer sif images in slurm batch jobscripts the option "--nv" should be used to ensure the resulting container is aware of the gpus on the host compute node.
- GPU resources in slurm batch scripts should be requested in the following format:
#SBATCH --gpus=3