November 5, 2024
What is Apptainer?
Apptainer is a container platform similar to Docker in which each container runs its own OS and contains software appropriate for that OS.
Crucially, the container OS can differ from that of the host machine running Apptainer. Consequently, users can deploy an Apptainer container (derived from a Apptainer or Docker image) with an OS version that is compatible with the software they wish to use. This feature should be particularly convenient for users who wish to continue to use their ERISTwo legacy workflows on ERISTwo for example.
While Docker and Apptainer share many key features, the key difference is that running Apptainer containers does not require sudo privileges. Consequently, users can run computations using Apptainer containers as they would other types of jobs. Furthermore and for convenience, users should be able to pull a suitable Docker image from one of the many Docker repositories available on the web. Customization and creation of Docker images using Dockerfiles for example will be outlined here.
Below can be found some examples starting with how to import a Docker image (containing a desired collection of software) into Apptainer and then subsequently how to deploy the generated Apptainer-image in Slurm jobs.
EXAMPLE 1. Pulling an external Docker Image into Apptainer
In this example we pull a Docker image containing a python installation from the Docker repository located here, and where particular choices for the Docker image OS can be found under the Tags menu item. Here are the suggested steps:
1. login to ERISTwo
2. Invoke an interactive Slurm session using the interactive partition
$ srun --pty -p interactive /bin/bashso that sufficient resources are available to run Apptainer. This step is necessary since the login nodes are resource-limited as described here and where the enforced limits can be examined with
$ ulimit -aNote, particularly, that the user is limited to 30 processes in a login session.
3. Use this command to check which versions of Apptainer are available on the cluster
$ module avail singularity4. Load the desired version of Apptainer
$ module load singularity/<version>5. Pull and import the Docker image into Apptainer, and where in this case the output will be the image file miniconda3.sif located in the current directory
$ singularity pull miniconda3.sif docker://continuumio/miniconda3:mainfor more details about Apptainer pull please refer to this link. If this operation should fail with an authentication error then it will be necessary to first register for a free account at Docker hub.
6. Examine the generated sif image file
$ ls -la miniconda3.sif$ -rwxr-xr-x. 1 userABC groupXYZ 214M May 2 17:32 miniconda3.siffrom which it is clear that sif images can be quite large. In a separate article (to be published later) we will describe how we can instead build these images from much smaller (text) Docker files and so save on local storage.
Finally, we note that most community-supported Docker images can be found at the Docker repository at https://hub.docker.com.
EXAMPLE 2. Running Apptainer interactively
For initial testing purposes we can run the sif Apptainer image interactively with the following steps:
1. login to ERISTwo
2. Invoke an interactive Slurm session using the interactive partition
$ srun --pty -p interactive /bin/bashso that sufficient resources are available to run Apptainer.
3. Load the Apptainer module
$ module load singularity4. Create an interactive bash shell from the sif image
$ singularity shell miniconda3.sif -s /bin/bash 5. Let us interrogate the shell and examine the contents of the filesystem
Apptainer> whoami
rk398
Apptainer> echo $SHELL
/bin/bash
Apptainer> ls -al /
total 11
drwxr-xr-x. 1 rk398 scicomp 60 May 6 15:04 .
drwxr-xr-x. 1 rk398 scicomp 60 May 6 15:04 ..
lrwxrwxrwx. 1 root root 27 May 2 17:31 .exec -> .Apptainer.d/actions/exec
lrwxrwxrwx. 1 root root 26 May 2 17:31 .run -> .Apptainer.d/actions/run
lrwxrwxrwx. 1 root root 28 May 2 17:31 .shell -> .Apptainer.d/actions/shell
drwxr-xr-x. 5 root root 127 May 2 17:31 .Apptainer.d
lrwxrwxrwx. 1 root root 27 May 2 17:31 .test -> .Apptainer.d/actions/test
drwxr-xr-x. 3 rk398 scicomp 60 May 6 15:04 PHShome
drwxr-xr-x. 2 root root 1115 Apr 25 08:57 bin
drwxr-xr-x. 2 root root 3 Jan 28 16:20 boot
drwxr-xr-x. 21 root root 3740 Feb 10 09:53 dev
lrwxrwxrwx. 1 root root 36 May 2 17:31 environment -> .Apptainer.d/env/90-environment.sh
drwxr-xr-x. 41 root root 1458 May 2 17:31 etc
drwxr-xr-x. 2 root root 3 Jan 28 16:20 home
drwxr-xr-x. 8 root root 117 Apr 23 11:00 lib
drwxr-xr-x. 2 root root 43 Apr 23 11:00 lib64
drwxr-xr-x. 2 root root 3 Apr 23 11:00 media
drwxr-xr-x. 2 root root 3 Apr 23 11:00 mnt
drwxr-xr-x. 3 root root 28 Apr 25 08:57 opt
dr-xr-xr-x. 749 root root 0 Sep 13 2022 proc
drwx------. 3 root root 60 Apr 25 08:57 root
drwxr-xr-x. 3 root root 39 Apr 23 11:00 run
drwxr-xr-x. 2 root root 1033 Apr 25 08:57 sbin
lrwxrwxrwx. 1 root root 24 May 2 17:31 Apptainer -> .Apptainer.d/runscript
drwxr-xr-x. 2 root root 3 Apr 23 11:00 srv
dr-xr-xr-x. 13 root root 0 Sep 13 2022 sys
drwxrwxrwt. 10 root root 287 May 6 14:45 tmp
drwxr-xr-x. 11 root root 165 Apr 23 11:00 usr
drwxr-xr-x. 11 root root 160 Apr 23 11:00 var
Apptainer> ls -al /PHShome/rk398
total 20908848
drwx------. 100 rk398 rk398 28672 May 6 15:02 .
drwxr-xr-x. 3 rk398 scicomp 60 May 6 15:04 ..
-rw-rw----. 1 rk398 rk398 10244 Feb 14 16:24 .DS_Store
-rw-------. 1 rk398 rk398 34158 Apr 15 16:44 .ICEauthority
-rw-r--r--. 1 rk398 scicomp 0 Apr 26 09:34 .Rhistory
-rw-------. 1 rk398 scicomp 431 Apr 15 16:45 .Xauthority
..
and where we notice that by default the home directory is mounted inside the container at /PHShome/<partnersID>. We can exit the container either with the command exit or by applying CTL-D.
Apptainer> exitIf it is more convenient, the user's home directory (/PHShome/<partnersID>) and briefcase storage (i.e. /data/<groupBriefcase) can be mounted under /home and /data respectively in the following way for example:
$ singularity shell -H /PHShome/<partnersID>:/home -B /data/<briefcaseName>:/data miniconda3.sif -s /bin/bash where the option -H indicates the user's home directory setting and -B the bind path specification for the briefcase. This command can be made less cumbersome by making use of the following bash environment variables:
$ export SINGULARITY_BIND="<srcdir1>:<container_directory1>,<srcdir2>:<container_directory2>"
$ export SINGULARITY_HOME="<homedir>:<container_homedir>"so that these directories are automatically mounted at the specified locations when the following is invoked:
$ singularity shell miniconda3.sif -s /bin/bashMore information about Apptainer environment variables can be found here.
EXAMPLE 3. Running a Apptainer sif image in a Slurm Job
With the Apptainer sif image obtained previously we will subsequently show how to deploy it in a Slurm batch job. In the following job script slurmJobExample2.sh the command Apptainer exec is used to invoke the miniconda3 version of python on miniconda3.sif to process the program primeNumbers.py:
#!/bin/bash
#SBATCH --partition=short
#SBATCH --job-name=example2ApptainerJob
#SBATCH --ntasks=1
#SBATCH --time=00:50:00
#SBATCH --mem-per-cpu=1G
#SBATCH --output=log.%j
#SBATCH --error=logErrors.%j
# Load Apptainer Module
module load singularity
# Specify container mount paths for the user's home directory and briefcase using environment variables
export SINGULARITY_BIND="<srcdir1>:<container_directory1>,<srcdir2>:<container_directory2>"
# Use default
# export SINGULARITY_HOME="/PHShome/<partnersID>:/PHShome/<partnersID>"
# Invoke container/python to run a python program
srun singularity exec miniconda3.sif python primeNumbers.py
and where the contents of the file primeNumbers.py (located in the current directory) are the following:
# Python program to print all prime number in a specified interval
def prime(x, y):
prime_list = []
for i in range(x, y):
if i == 0 or i == 1:
continue
else:
for j in range(2, int(i/2)+1):
if i % j == 0:
break
else:
prime_list.append(i)
return prime_list
starting_range = 1000
ending_range = 2000
lst = prime(starting_range, ending_range)
print("The prime numbers in this range are: ", lst)
Subsequently, the job is submitted to the Slurm job scheduler from an eristwo login node with
$ sbatch slurmJobExample2.sh
More generally, in the slurm jobscript we can invoke a series of container commands with exec using a bash script file myScriptFile.sh in the following manner:
srun singularity exec miniconda3.sif $SLURM_SUBMIT_DIR/myScriptFile.shand where myScriptFile.sh is located in the job submission folder in this case.
Finally, more details about Apptainer exec can be found here while further information about Slurm batch script settings is provided at this link.