Getting Started with Slurm on the ERIS Cluster

Getting Started with Slurm on the ERIS Cluster

Compute jobs on the ERIS cluster are submitted through the Slurm workload manager. There are two main ways to submit compute jobs to the cluster:

Interactive jobs allow you to interact directly with your job in real time. This is a good approach for developing or debugging batch jobs (see next bullet), for interactive analysis, or for other one-off computational needs. There are two ways to start an interactive job:

    • salloc will start an interactive job on a single CPU in the default normal Slurm partition.
    • srun -p interactive --mem=4G --pty /bin/bash will start an interactive job in the specified Slurm partition requesting 4GB of memory. srun allows you to specify suitable resources, such as memory, CPUs, etc for these types of jobs. Please note that eventually interactive jobs will be restricted to the interactive partition only.

NOTE: Please use the interactive partition for interactive jobs. In the future, all interactive jobs will be forced to run in the interactive partition, including ones started by salloc.

Batch jobs allow you to submit jobs to the Slurm workload manager, and Slurm will run the job as soon as the requested resources become available (i.e., jobs are submitted in batches, therefore the name). Batch jobs are described in a batch script, which constitutes a recipe for executing your compute job or analysis. Batch jobs are submitted to Slurm via:

    • sbatch <batch script name>. We recommend that you choose a batch script name that uniquely identifies the job's purpose. Otherwise, it may be difficult to distinguish individual jobs amongst all the other jobs running on the cluster (yours and those of other users).

 

Example Slurm Batch Script

Below is an example Slurm batch script that you can customize to your specific needs

 

#!/bin/bash #SBATCH --job-name="My test job" #SBATCH --time=01:00:00 #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --cpus-per-task=1 #SBATCH --mem=8G #SBATCH --partition=normal #SBATCH -o %x_%j.out            ### slurm output file, %x is jobname, %j is jobid #SBATCH -e %x_%j.err %j is jobid    ### slurm error file, %x is jobname, # load any required modules # module load ... # cd into the appropriate directory # cd /data/.... # execute the job echo "running the job"

 

What Slurm Partition to Choose for Your Job

The ERIS Cluster was originally set up with the following Slurm partitions available, but for the latest information please refer to the sinfo utility.

Partition Name Max Job Duration Memory/node CPUs/node Number of Nodes Description
normal 1day 512GB 96 6 default partition recommended for most jobs
long 7days 384GB 80 6 recommended for compute jobs that require more than 1 day of execution
bigmem 1TB+ 1TB+ 88-92 10 reserved for large memory jobs that don't fit into the normal/long partitions
interactive 512GB 512GB 96 1 intended for interactive use
debug 384GB 384GB 80 1 partition for short jobs, debugging or testing

 

Before submitting your job, please consider carefully which partition to choose. Since ERIS is a shared resource, it is important to select compute partitions that best suit your needs. E.g., do not submit a small analysis job to the bigmen partition because it wastes valuable large memory resources some of your colleagues may need. In addition to which the FairShare Scheduling policy (see later) would enforce a greater cost for such an inappropriately allocated job.

How to Right-Size Your Slurm Jobs

Among other factors, Slurm makes scheduling decisions for starting new jobs based on the job information you provide in your batch script. In general, providing Slurm with close estimates of the job's actual duration and required resources will allow Slurm to start your job more quickly. Let's assume your job takes 4h to complete and requires 4GB of memory. Now consider:

  • You submit your job to the normal partition, but you do not specify a job duration. In this case, Slurm will assume that your job will take the maximum job duration for the normal partition, i.e., 1 day. Slurm has to wait for a 24h (or more) resource window to open up before it can start your job. Even if a smaller 8h time window is available earlier, which would have been sufficient to complete your 4h job, Slurm will not start your job since it assumes the job will take 24h. If instead you specify #SBATCH -- time=04:30:00 # job duration + 30min padding Slurm is aware of the actual duration of your job and will start it once a resource window with more than 4.5h becomes available.

 

  • You are not quite sure how much memory your job needs, and you decide to request 24GB, just to be sure. Slurm now has to wait for 24GB (or roughly 5 CPUs worth) of resources to become available, even though in reality your 4GB job could have easily fit on a single CPU within the normal partition. Similarly, to the job duration case above, it will likely take Slurm much longer to provision 24GB of memory than to provision 4GB. If instead you specify #SBATCH --mem=4G Slurm will know that you only need 4GB of memory, and your job will likely start sooner.

To estimate resource requirements, we recommend that you use the

seff <jobid>

command available on the login nodes. seff will provide resource actuals for completed runs. Typically, you would conduct a few test runs with initially conservative resource estimates and then use seff information to subsequently constrain Slurm resource requests to what is actually needed (plus a small buffer). Of course, resource requirements and duration may depend on data set sizes, and you may have to repeat the process when data set sizes change or for different types of data sets.

Useful Slurm Commands

The following commands are useful for interacting with Slurm

Slurm Command Description

sbatch <batch script>

submit batch script to Slurm

srun

run parallel jobs in Slurm. Can also be used to submit interactive jobs

salloc

request a simple interactive job

scancel <jobID>

cancel the job with jobID. WARNNING: jobs will be immediately canceled without

confirmation, so please double-check that you're cancelling the correct job

sinfo

show information about Slurm nodes and partitions

squeue

show information about jobs managed by Slurm. squeue -u userid shows only jobs owned by a given user

sacct -u <userID>

show slurm accounting data for user with userid

 

Slurm Job status, code, and explanation

When you request status information on your job you can get one of the following:

COMPLETED

CD

The job has been completed successfully.

COMPLETING

CG

The job is finishing but some processes are still active.

FAILED

F

The job terminated with a non-zero exit code and failed to execute.

PENDING

PD

The job is waiting for resource allocation. It will eventually run.

PREEMPTED

PR

The job was terminated because of preemption by another job.

RUNNING

R

The job currently is allocated to a node and is running.

SUSPENDED

S

A running job has been stopped with its cores released to other jobs.

STOPPED

ST

A running job has been stopped with its cores retained.

 

 

Job Environment Variables

Slurm Description
$SLURM_JOBID Job ID
$SLURM_SUBMIT_DIR Submit directory
$SLURM_ARRAY_JOB_ID Job Array Parent
$SLURM_ARRAY_TASK_ID Job Array Index
$SLURM_SUBMIT_HOST Submission Host
$SLURM_JOB_NODELIST Allocated compute nodes
$SLURM_NTASKS
(mpirun can automatically pick this up from Slurm, it does not need to be specified)
Number of processors allocated
$SLURM_JOB_PARTITION Queue

 

 

Example SLURM Job Submission

EXAMPLE 1: Interactive JOB

An interactive session on one of the compute nodes can most conveniently be initiated by invoking the following:

srun  --pty -p interactive /bin/bash

and where additional resources (e.g. memory, cpus etc) can be specified using the settings indicated in the table above. Alternatively,  an interactive session can be initiated by using salloc.

 

EXAMPLE 2: SINGLE CPU JOB 

 In the example below, a simple Python program is submitted as a SLURM job to the ‘short’ partition using ‘sbatch <file_name>’ where 1 CPU and 1GB of memory are requested.  

 

#!/bin/bash 

#SBATCH --job-name=single_cpu_example 

#SBATCH --partition=short 

#SBATCH --ntasks=1 

#SBATCH --cpus-per-task=1 

#SBATCH –-mem-per-cpu=1G 

#SBATCH --output=log%J.out 

#SBATCH --error=log%J.err 

 

# Load conda with the module of miniforge3 distribution 

module load miniforge3 

 

# Run a demo python program on a single CPU 

python hello.py 

 

EXAMPLE 3: MULTIPLE CPU JOB 

In the example below, a Python program is submitted as a SLURM job to the ‘normal’ partition using ‘sbatch <file_name>' where 4 CPUs and 1GB of memory per CPU are requested.  

 

#!/bin/bash 

#SBATCH --job-name=multi_cpu_example 

#SBATCH --partition=normal 

#SBATCH --ntasks=4 

#SBATCH --cpus-per-task=1 

#SBATCH --mem-per-cpu=1 

#SBATCH --output=log%J.out 

#SBATCH --error=log%J.err 
 

# Load conda with the module of miniforge3 distribution 

module load miniforge3 

 

# Run a python program on 4 CPUs that splits the iterations of the loop among each CPU 

python square.py 

 

EXAMPLE 4: SINGULARITY JOB 

 In the example below, a Python program is run upon the startup of a singularity container and is submitted as a SLURM job to the ‘short’ partition using ‘sbatch <file_name>’.  

 

#!/bin/bash 

#SBATCH --job-name=singularity_example 

#SBATCH --partition=short 

#SBATCH –-ntasks=1 

#SBATCH --mem-per-cpu=2 

#SBATCH --output=singularity-log%J.out 

#SBATCH --error=singularity-log%J.err 

 

# Check available versions of singularity with ‘module avail singularity’ 

module load singularity 

 

# Run application using singularity 

singularity exec conda-miniforge3.sif python3 pyTest.py 

 

More details about Singularity on the ERIS cluster can be found here.

 

EXAMPLE 5: ARRAY JOB USING R 

 In the example below, an R script is submitted as a SLURM job to the ‘filemove’ partition using ‘sbatch <file_name>'.  

 

#!/bin/bash 

#SBATCH --job-name=hello-parallel-test 

#SBATCH --partition=filemove 

#SBATCH --array=1-5 

#SBATCH –-ntasks=2 

#SBATCH --mem-per-cpu=1 

#SBATCH --output=hello-%j-%a.out 

#SBATCH --error=hello-%j-%a.err 
 

# Check available versions of R using ‘module avail R’ 

module load R/3.5.1-foss-2018b 

 

# Run application using R passing in the array ID, corresponding to $SLURM_ARRAY_TASK_ID, as a command line argument 

Rscript hello-parallel.R $SLURM_ARRAY_TASK_ID 

 

The example below is the simple R script used in example 4.

hello-parallel.R 

#!/usr/bin/env Rscript 

args = commandArgs(trailingOnly=TRUE) 

print(paste0('Hello! I am task number: ', args[1])) 

vector = c(1, 10, 100, 1000, 10000) 

multiply = function(x, y) { 

return(x*y) 

} 

num = as.integer(args[1]) 

res = multiply(vector[num],num) 

print(paste0('Result: ', res)) 

Go to KB0039912 in the IS Service Desk