Using Apptainer (formerly known as Singularity) on the ERIS Cluster

What is Apptainer?

Apptainer is a container platform similar to Docker in which each container runs its own OS and contains software appropriate for that OS.

Crucially, the container OS can differ from that of the host machine running Apptainer. Consequently, users can deploy a Apptainer container (derived from a Apptainer or Docker image) with an OS version that is compatible with the software they wish to use. This feature should be particularly convenient for users who wish to continue to use their ERISOne legacy workflows on ERISTwo for example.

While Docker and Apptainer share many key features, the key difference is that running Apptainer containers does not require sudo privileges. Consequently, users can run computations using Apptainer containers as they would other types of jobs. Furthermore and for convenience, users should be able to pull a suitable Docker image from one of the many Docker repositories available on the web. Customization and creation of Docker images using Dockerfiles for example will be outlined here.

Below can be found some examples starting with how to import a Docker image (containing a desired collection of software) into Apptainer and then subsequently how to deploy the generated Apptainer-image in Slurm jobs. 

EXAMPLE 1. Pulling an external Docker Image into Apptainer

In this example we pull a Docker image containing a python installation from the Docker repository located here, and where particular choices for the Docker image OS can be found under the Tags menu item. Here are the suggested steps:

1. login to ERISTwo

2. Invoke an interactive Slurm session using the interactive partition

 $ srun --pty -p interactive /bin/bash

so that sufficient resources are available to run Apptainer. This step is necessary since the login nodes are resource-limited as described here and where the enforced limits can be examined with 

 $ ulimit -a

Note, particularly, that the user is limited to 30 processes in a login session.

3. Use this command to check which versions of Apptainer are available on the cluster

$ module avail singularity

4. Load the desired version of Apptainer

$ module load singularity/<version>

5. Pull and import the Docker image into Apptainer, and where in this case the output will be the image file miniconda3.sif located in the current directory

$ singularity pull miniconda3.sif docker://continuumio/miniconda3:main

for more details about Apptainer pull please refer to this link.

6. Examine the generated sif image file

$ ls -la miniconda3.sif

$ -rwxr-xr-x. 1 userABC groupXYZ 214M May  2 17:32 miniconda3.sif

from which it is clear that sif images can be quite large. In a separate article (to be published later) we will describe how we can instead build these images from much smaller (text) Docker files and so save on local storage.

Finally, we note that most community-supported Docker images can be found at the Docker repository at https://hub.docker.com.

EXAMPLE 2. Running Apptainer interactively

For initial testing purposes we can run the sif Apptainer image interactively with the following steps:

1. login to ERISTwo

2. Invoke an interactive Slurm session using the interactive partition

 $ srun --pty -p interactive /bin/bash

so that sufficient resources are available to run Apptainer.

3. Load the Apptainer module 

$ module load singularity

4. Create an interactive bash shell from the sif image

$ singularity shell miniconda3.sif -s /bin/bash  

5. Let us interrogate the shell and examine the contents of the filesystem

Apptainer> whoami
rk398

Apptainer> echo $SHELL
/bin/bash

Apptainer> ls -al /
total 11
drwxr-xr-x.   1 rk398 scicomp   60 May  6 15:04 .
drwxr-xr-x.   1 rk398 scicomp   60 May  6 15:04 ..
lrwxrwxrwx.   1 root  root      27 May  2 17:31 .exec -> .Apptainer.d/actions/exec
lrwxrwxrwx.   1 root  root      26 May  2 17:31 .run -> .Apptainer.d/actions/run
lrwxrwxrwx.   1 root  root      28 May  2 17:31 .shell -> .Apptainer.d/actions/shell
drwxr-xr-x.   5 root  root     127 May  2 17:31 .Apptainer.d
lrwxrwxrwx.   1 root  root      27 May  2 17:31 .test -> .Apptainer.d/actions/test
drwxr-xr-x.   3 rk398 scicomp   60 May  6 15:04 PHShome
drwxr-xr-x.   2 root  root    1115 Apr 25 08:57 bin
drwxr-xr-x.   2 root  root       3 Jan 28 16:20 boot
drwxr-xr-x.  21 root  root    3740 Feb 10 09:53 dev
lrwxrwxrwx.   1 root  root      36 May  2 17:31 environment -> .Apptainer.d/env/90-environment.sh
drwxr-xr-x.  41 root  root    1458 May  2 17:31 etc
drwxr-xr-x.   2 root  root       3 Jan 28 16:20 home
drwxr-xr-x.   8 root  root     117 Apr 23 11:00 lib
drwxr-xr-x.   2 root  root      43 Apr 23 11:00 lib64
drwxr-xr-x.   2 root  root       3 Apr 23 11:00 media
drwxr-xr-x.   2 root  root       3 Apr 23 11:00 mnt
drwxr-xr-x.   3 root  root      28 Apr 25 08:57 opt
dr-xr-xr-x. 749 root  root       0 Sep 13  2022 proc
drwx------.   3 root  root      60 Apr 25 08:57 root
drwxr-xr-x.   3 root  root      39 Apr 23 11:00 run
drwxr-xr-x.   2 root  root    1033 Apr 25 08:57 sbin
lrwxrwxrwx.   1 root  root      24 May  2 17:31 Apptainer -> .Apptainer.d/runscript
drwxr-xr-x.   2 root  root       3 Apr 23 11:00 srv
dr-xr-xr-x.  13 root  root       0 Sep 13  2022 sys
drwxrwxrwt.  10 root  root     287 May  6 14:45 tmp
drwxr-xr-x.  11 root  root     165 Apr 23 11:00 usr
drwxr-xr-x.  11 root  root     160 Apr 23 11:00 var

Apptainer> ls -al /PHShome/rk398

total 20908848
drwx------. 100 rk398 rk398         28672 May  6 15:02 .
drwxr-xr-x.   3 rk398 scicomp          60 May  6 15:04 ..
-rw-rw----.   1 rk398 rk398         10244 Feb 14 16:24 .DS_Store
-rw-------.   1 rk398 rk398         34158 Apr 15 16:44 .ICEauthority
-rw-r--r--.   1 rk398 scicomp           0 Apr 26 09:34 .Rhistory
-rw-------.   1 rk398 scicomp         431 Apr 15 16:45 .Xauthority
..

and where we notice that by default the home directory is mounted inside the container at /PHShome/<partnersID>. We can exit the container either with the command exit or by applying CTL-D.

Apptainer> exit

If it is more convenient, the user's home directory (/PHShome/<partnersID>) and briefcase storage (i.e. /data/<groupBriefcase) can be mounted under /home and /data respectively in the following way for example:

$ singularity shell -H /PHShome/<partnersID>:/home -B /data/<briefcaseName>:/data miniconda3.sif -s /bin/bash  

where the option -H indicates the user's home directory setting and -B the bind path specification for the briefcase. This command can be made less cumbersome by making use of the following bash environment variables:

$ export SINGULARITY_BIND="<srcdir1>:<container_directory1>,<srcdir2>:<container_directory2>"

$ export SINGULARITY_HOME="<homedir>:<container_homedir>"

so that these directories are automatically mounted at the specified locations when the following is invoked:

$ singularity shell miniconda3.sif -s /bin/bash

More information about Apptainer environment variables can be found here.

EXAMPLE 3. Running a Apptainer sif image in a Slurm Job

With the Apptainer sif image obtained previously we will subsequently show how to deploy it in a Slurm batch job. In the following job script slurmJobExample2.sh the command Apptainer exec is used to invoke the miniconda3 version of python on miniconda3.sif to process the program primeNumbers.py:

#!/bin/bash
#SBATCH --partition=short
#SBATCH --job-name=example2ApptainerJob
#SBATCH --ntasks=1
#SBATCH --time=00:50:00
#SBATCH --mem-per-cpu=1G
#SBATCH --output=log.%j
#SBATCH --error=logErrors.%j

# Load Apptainer Module

module load singularity

# Specify container mount paths for the user's home directory and briefcase using environment variables

export SINGULARITY_BIND="<srcdir1>:<container_directory1>,<srcdir2>:<container_directory2>"
# Use default
# export SINGULARITY_HOME="/PHShome/<partnersID>:/PHShome/<partnersID>"


# Invoke container/python to run a python program

srun singularity exec miniconda3.sif  python  primeNumbers.py

and where the contents of the file primeNumbers.py (located in the current directory) are the following:

# Python program to print all prime number in a specified interval

def prime(x, y):
    prime_list = []
    for i in range(x, y):
        if i == 0 or i == 1:
            continue
        else:
            for j in range(2, int(i/2)+1):
                if i % j == 0:
                    break
            else:
                prime_list.append(i)
    return prime_list

starting_range = 1000
ending_range = 2000
lst = prime(starting_range, ending_range)
print("The prime numbers in this range are: ", lst)

Subsequently, the job is submitted to the Slurm job scheduler from an eristwo login node with

$ sbatch slurmJobExample2.sh

More generally, in the slurm jobscript we can invoke a series of container commands with exec using a bash script file myScriptFile.sh in the following manner: 

srun singularity exec miniconda3.sif $SLURM_SUBMIT_DIR/myScriptFile.sh

and where myScriptFile.sh is located in the job submission folder in this case.

Finally, more details about Apptainer exec can be found here while further information about Slurm batch script settings is provided at this link.