Installing Applications as Modules for Lab Groups

Loading Available Modules

Software provided on the Scientific Computing (SciC) Linux Clusters through the Modules system (aka environment modules) are installed in one place but available on all servers and compute nodes.  Loading one of these modules sets the correct paths and environment variables for your account to use software, and they can be "unloaded" at a later time.  There are many advantages to using environment modules for providing access to software, over manually setting paths and variables.

Create New Modules

If a lab requires a change to a module, a new module should be created under "modulefiles/lab/their_lab". Essential changes to modules under "modulefiles/lab" can only be made during the Sunday maintenance window following notification to SciC Linux Clusters Support. The new version is created under "modulefiles/test" and moved to "modulefiles/base" during maintenance.

Where a package may be available, either on the compute node or through environment modules, use the module version if you require version stability or a specific release version.

Module file organization

Module files are located under "/apps/modulefiles", there are several repositories described below. Within each repository, the structure is (all in lower case)

package_name/package_version

eg. the module for "SCons" version 2.3.0 is in a text file named "2.3.0" in a folder named "scons" and appears when "module avail" is run as

scons/2.3.0

Modules can have a default version, for researchers who wish to always have the current version and do not require or desire notification when the current version changes. A default version is created with a symbolic link pointing to the current version. This can be changed without affecting running jobs

cd scons/
ln -s 2.3.0 default

/apps/modulefiles/base

Available to all users by default. Finalized modules go here after being tested and accepted (under "/apps/modulefiles/test") by SciC Linux Clusters and the requesting researcher

/apps/modulefiles/test

Modules in development go here until finalized. Ask the requesting researcher to issue this command to make the module available for testing:

module use /apps/modulefiles/test

Modules in this location may be unfinished or broken

/apps/modulefiles/licensed

Modules for software requiring a proprietary license tied to a specific group goes here, to prevent confusion if users without access to the license see it in the main repository and attempt to run the software.

/apps/modulefiles/lab

Lab groups wishing to keep software centrally for high-availability will be given a modules folder under "/apps/modulefiles/lab". This is also the location for individuals or lab groups requiring the same software as under "base" but with different compile options. By issuing

module use /apps/modulefiles/lab/mylab

the custom version or lab-specific software will be loaded in preference over "base" modules

Module template

Refer to the Documentation

All modules should create an environment variable with their name followed by "VERSION" containing the version number

setenv SCONSVERSION "2.3.0"

Modules depending on other modules should use the "prereq" statement (eg. to depend on any version of python)

prereq "python"

Python Interpreter Environment Module Template

In addition to the standard environment module template, a python interpreter module requires this variable set

setenv PYTHONLIBVERSION "python2.7"

Python Egg/Application Environment Module Template

Python "eggs" are compatible with the major version of python they are installed for. The additions to the standard environment module template are:

Check that a python interpreter is loaded

prereq python

Read the PYTHONLIBVERSION set by the python interpreter environment module and use this to set the correct path to the module source

set curMod module-info name
set module_root "/apps/source/python-lib/$::env(PYTHONLIBVERSION)/$curMod"

Check that the module is available for the version of Python currently loaded

if

Unknown macro: { ! [file exists "$module_root"] }

Unknown macro: { puts stderr "$curMod is not installed for Python version $}

Source (software installations) organization

Source files are located under /apps/source/ The structure is (all in lower case)

package_name/package_version

eg. the source for "tophat" version 2.3.0 is in a folder named "tophat/2.3.0".

Within the "tophat/2.3.0" folder, a folder "source" will contain the downloaded installation files (either source code or binary archive). Any compilation will be done under "tophat/2.3.0/source" - the install location (prefix) will be "tophat/2.3.0"

Python (eggs or software) organization

/source/python-lib/$

Unknown macro: {PYTHONLIBVERSION}

/packagename/packageversion (target for installation)

/source/python-lib/$

/packagename/packageversion/source (downloaded installer)

PYTHONLIBVERSION is "python2.7" etc... which is also set as an environment variable when a python interpreter environment module is loaded

Go to KB0027986 in the IS Service Desk

Related articles