August 6, 2026
Getting Started with Slurm on the ERIS Cluster
Compute jobs on the ERIS cluster are submitted through the Slurm workload manager. There are two main ways to submit compute jobs to the cluster:
Interactive jobs allow you to interact directly with your job in real time. This is a good approach for developing or debugging batch jobs (see next bullet), for interactive analysis, or for other one-off computational needs. There are two ways to start an interactive job:
-
- salloc will start an interactive job on a single CPU in the default normal Slurm partition.
- srun -p interactive --mem=4G --pty /bin/bash will start an interactive job in the specified Slurm partition requesting 4GB of memory. srun allows you to specify suitable resources, such as memory, CPUs, etc for these types of jobs. Please note that eventually interactive jobs will be restricted to the interactive partition only.
NOTE: Please use the interactive partition for interactive jobs. In the future, all interactive jobs will be forced to run in the interactive partition, including ones started by salloc.
Batch jobs allow you to submit jobs to the Slurm workload manager, and Slurm will run the job as soon as the requested resources become available (i.e., jobs are submitted in batches, therefore the name). Batch jobs are described in a batch script, which constitutes a recipe for executing your compute job or analysis. Batch jobs are submitted to Slurm via:
-
- sbatch <batch script name>. We recommend that you choose a batch script name that uniquely identifies the job's purpose. Otherwise, it may be difficult to distinguish individual jobs amongst all the other jobs running on the cluster (yours and those of other users).
Example Slurm Batch Script
Below is an example Slurm batch script that you can customize to your specific needs
#!/bin/bash #SBATCH --job-name="My test job" #SBATCH --time=01:00:00 #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --cpus-per-task=1 #SBATCH --mem=8G #SBATCH --partition=normal #SBATCH -o %x_%j.out ### slurm output file, %x is jobname, %j is jobid #SBATCH -e %x_%j.err %j is jobid ### slurm error file, %x is jobname, # load any required modules # module load ... # cd into the appropriate directory # cd /data/.... # execute the job echo "running the job"
What Slurm Partition to Choose for Your Job
The ERIS Cluster was originally set up with the following Slurm partitions available, but for the latest information please refer to the sinfo utility.
| Partition Name | Max Job Duration | Memory/node | CPUs/node | Number of Nodes | Description |
| normal | 1day | 512GB | 96 | 6 | default partition recommended for most jobs |
| long | 7days | 384GB | 80 | 6 | recommended for compute jobs that require more than 1 day of execution |
| bigmem | 1TB+ | 1TB+ | 88-92 | 10 | reserved for large memory jobs that don't fit into the normal/long partitions |
| interactive | 512GB | 512GB | 96 | 1 | intended for interactive use |
| debug | 384GB | 384GB | 80 | 1 | partition for short jobs, debugging or testing |
Before submitting your job, please consider carefully which partition to choose. Since ERIS is a shared resource, it is important to select compute partitions that best suit your needs. E.g., do not submit a small analysis job to the bigmen partition because it wastes valuable large memory resources some of your colleagues may need. In addition to which the FairShare Scheduling policy (see later) would enforce a greater cost for such an inappropriately allocated job.
How to Right-Size Your Slurm Jobs
Among other factors, Slurm makes scheduling decisions for starting new jobs based on the job information you provide in your batch script. In general, providing Slurm with close estimates of the job's actual duration and required resources will allow Slurm to start your job more quickly. Let's assume your job takes 4h to complete and requires 4GB of memory. Now consider:
- You submit your job to the normal partition, but you do not specify a job duration. In this case, Slurm will assume that your job will take the maximum job duration for the normal partition, i.e., 1 day. Slurm has to wait for a 24h (or more) resource window to open up before it can start your job. Even if a smaller 8h time window is available earlier, which would have been sufficient to complete your 4h job, Slurm will not start your job since it assumes the job will take 24h. If instead you specify #SBATCH -- time=04:30:00 # job duration + 30min padding Slurm is aware of the actual duration of your job and will start it once a resource window with more than 4.5h becomes available.
- You are not quite sure how much memory your job needs, and you decide to request 24GB, just to be sure. Slurm now has to wait for 24GB (or roughly 5 CPUs worth) of resources to become available, even though in reality your 4GB job could have easily fit on a single CPU within the normal partition. Similarly, to the job duration case above, it will likely take Slurm much longer to provision 24GB of memory than to provision 4GB. If instead you specify #SBATCH --mem=4G Slurm will know that you only need 4GB of memory, and your job will likely start sooner.
To estimate resource requirements, we recommend that you use the
seff <jobid>
command available on the login nodes. seff will provide resource actuals for completed runs. Typically, you would conduct a few test runs with initially conservative resource estimates and then use seff information to subsequently constrain Slurm resource requests to what is actually needed (plus a small buffer). Of course, resource requirements and duration may depend on data set sizes, and you may have to repeat the process when data set sizes change or for different types of data sets.
Useful Slurm Commands
The following commands are useful for interacting with Slurm
| Slurm Command | Description |
|
sbatch <batch script> |
submit batch script to Slurm |
| srun |
run parallel jobs in Slurm. Can also be used to submit interactive jobs |
| salloc |
request a simple interactive job |
| scancel <jobID> |
cancel the job with jobID. WARNNING: jobs will be immediately canceled without confirmation, so please double-check that you're cancelling the correct job |
| sinfo |
show information about Slurm nodes and partitions |
| squeue |
show information about jobs managed by Slurm. squeue -u userid shows only jobs owned by a given user |
| sacct -u <userID> |
show slurm accounting data for user with userid |
Slurm Job status, code, and explanation
When you request status information on your job you can get one of the following:
|
COMPLETED |
CD |
The job has been completed successfully. |
|
COMPLETING |
CG |
The job is finishing but some processes are still active. |
|
FAILED |
F |
The job terminated with a non-zero exit code and failed to execute. |
|
PENDING |
PD |
The job is waiting for resource allocation. It will eventually run. |
|
PREEMPTED |
PR |
The job was terminated because of preemption by another job. |
|
RUNNING |
R |
The job currently is allocated to a node and is running. |
|
SUSPENDED |
S |
A running job has been stopped with its cores released to other jobs. |
|
STOPPED |
ST |
A running job has been stopped with its cores retained. |
Job Environment Variables
| Slurm | Description |
|---|---|
| $SLURM_JOBID | Job ID |
| $SLURM_SUBMIT_DIR | Submit directory |
| $SLURM_ARRAY_JOB_ID | Job Array Parent |
| $SLURM_ARRAY_TASK_ID | Job Array Index |
| $SLURM_SUBMIT_HOST | Submission Host |
| $SLURM_JOB_NODELIST | Allocated compute nodes |
| $SLURM_NTASKS (mpirun can automatically pick this up from Slurm, it does not need to be specified) |
Number of processors allocated |
| $SLURM_JOB_PARTITION | Queue |
Example SLURM Job Submission
EXAMPLE 1: Interactive JOB
An interactive session on one of the compute nodes can most conveniently be initiated by invoking the following:
srun --pty -p interactive /bin/bash
and where additional resources (e.g. memory, cpus etc) can be specified using the settings indicated in the table above. Alternatively, an interactive session can be initiated by using salloc.
EXAMPLE 2: SINGLE CPU JOB
In the example below, a simple Python program is submitted as a SLURM job to the ‘short’ partition using ‘sbatch <file_name>’ where 1 CPU and 1GB of memory are requested.
#!/bin/bash
#SBATCH --job-name=single_cpu_example
#SBATCH --partition=short
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH –-mem-per-cpu=1G
#SBATCH --output=log%J.out
#SBATCH --error=log%J.err
# Load conda with the module of miniforge3 distribution
module load miniforge3
# Run a demo python program on a single CPU
python hello.py
EXAMPLE 3: MULTIPLE CPU JOB
In the example below, a Python program is submitted as a SLURM job to the ‘normal’ partition using ‘sbatch <file_name>' where 4 CPUs and 1GB of memory per CPU are requested.
#!/bin/bash
#SBATCH --job-name=multi_cpu_example
#SBATCH --partition=normal
#SBATCH --ntasks=4
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=1
#SBATCH --output=log%J.out
#SBATCH --error=log%J.err
# Load conda with the module of miniforge3 distribution
module load miniforge3
# Run a python program on 4 CPUs that splits the iterations of the loop among each CPU
python square.py
EXAMPLE 4: SINGULARITY JOB
In the example below, a Python program is run upon the startup of a singularity container and is submitted as a SLURM job to the ‘short’ partition using ‘sbatch <file_name>’.
#!/bin/bash
#SBATCH --job-name=singularity_example
#SBATCH --partition=short
#SBATCH –-ntasks=1
#SBATCH --mem-per-cpu=2
#SBATCH --output=singularity-log%J.out
#SBATCH --error=singularity-log%J.err
# Check available versions of singularity with ‘module avail singularity’
module load singularity
# Run application using singularity
singularity exec conda-miniforge3.sif python3 pyTest.py
EXAMPLE 5: ARRAY JOB USING R
In the example below, an R script is submitted as a SLURM job to the ‘filemove’ partition using ‘sbatch <file_name>'.
#!/bin/bash
#SBATCH --job-name=hello-parallel-test
#SBATCH --partition=filemove
#SBATCH --array=1-5
#SBATCH –-ntasks=2
#SBATCH --mem-per-cpu=1
#SBATCH --output=hello-%j-%a.out
#SBATCH --error=hello-%j-%a.err
# Check available versions of R using ‘module avail R’
module load R/3.5.1-foss-2018b
# Run application using R passing in the array ID, corresponding to $SLURM_ARRAY_TASK_ID, as a command line argument
Rscript hello-parallel.R $SLURM_ARRAY_TASK_ID
The example below is the simple R script used in example 4.
hello-parallel.R
#!/usr/bin/env Rscript
args = commandArgs(trailingOnly=TRUE)
print(paste0('Hello! I am task number: ', args[1]))
vector = c(1, 10, 100, 1000, 10000)
multiply = function(x, y) {
return(x*y)
}
num = as.integer(args[1])
res = multiply(vector[num],num)
print(paste0('Result: ', res))