Using Python on Scientific Computing Linux Clusters with Anaconda

Scientific Computing (SciC) Linux Clusters is the Linux compute cluster for research.
 
Anaconda is a tool providing Python and many associated scientific libraries. 

Loading and Initializing Anaconda

Note: the loading and initializing of conda only needs to happen once every time for version 4.6 and up until you change your ~/.bashrc file or use a different version of anaconda.

To use Python on SciC Linux Clusters, load in your preferred Anaconda module. Type the command below at the $ prompt on the SciC Linux Clusters command line

   $ module load anaconda/<version> 

To see all the version we have available, you can run:

$ module avail anaconda
Check that you have the correct version:
     $ conda --version 

To create and activate virtual conda environments, you will need to initialize conda. For versions 4.6 and greater:

$ conda init bash

The conda init command modifies your ~/.bashrc file that contains configuration information to be applied whenever you login. After running conda init, you will then need to logout and log back in, or run $ /bin/bash to start a new subshell. Once you run conda init for the first time, conda will automatically the default base environment for every subsequent login. You can tell the conda is successfully initialized and activated because your terminal prompt should now have the name of the environment current activated before it:

(base) $ <command to be run in environment>

For versions lower than 4.6, no initialization is needed, but you will need to module load anaconda every session you would like to use a conda environment.

Creating and activating environments

To create a new environment, you can use the conda create command :

$ conda create -n <env-name> [packages to be installed]

This will create a new environment with the given name in the default location, which is $HOME/.conda/envs. For example, to create a new environment called 'test-python that will have python 3.8 installed:

$ conda create -n test-python-3.8 python=3.8

If you would like to create an environment in a location other than the default, use the -p flag instead of the -n flag to specify a path:

$ conda create -p path/to/envs/<env-name> [packages to be installed]

To view the environments you have created:

$ conda env list

Finally, to activate your conda environment, for versions >= 4.4 :

$ conda activate <env-name>

For versions < 4.4 :

$ source activate <env-name>

Once activated, you can now start installing using all the python packages included with Anaconda.

Adding additional packages

In your activated environment, iff you still need a package that is not installed just use:

   (env-name) $ pip install --user package_name

Using the "--user" option ensures that the package is installed in your home directory. Without this option you would get an error message as permissions will not allow pip to install in the SciC Linux Clusters software repository. 

Additionally, you can also use the conda install tool to get python packages. When using the conda install tool, it is helpful to include which channel you would like the package installed from. Different anaconda channels offer different builds of each package - the most common channel is conda-forge, but others include the defaults channel that pulls directly from anaconda and the bioconda channel. More information on using channels can be found here.

(env-name) $ conda install -c <channel-name> package_name

Packages installed by pip are available for use right away.  They are installed under a hidden folder ".local" in your home directory and in a separate folder for each Python version. For example, to look at what packages are installed in your home folder for python 2.7, do

     $ ls $⁠{HOME}/.local/lib/python2.7/site-packages/

Creating virtual environments

Refer to this FAQ and the online documentation for more information on setting up virtual environments

 

Go to KB0028044 in the IS Service Desk

Related articles