Adding Your Own Conda Virtual Environment to Scientific Computing Linux Clusters Jupyterhub

When using the Jupyterhub server (https://jupyterhub.partners.org or, alternatively, https://jupyterhub2.partners.org), users are able to create and launch their own conda virtual environments. This workflow will advise users on how to create their own conda virtual environment and add it to their Jupyterhub.

Creating A Conda Virtual Environment

To begin, a conda virtual environment needs to be created in your Scientific Computing (SciC) Linux Clusters account. If you do not already have a virtual environment, refer to the Using Python with Conda article to create one. You can use miniforge3 module or your licensed miniconda distribution to manage conda environments.

Once created, install packages in your custom environment, using “conda install”. Be sure to include the “jupyter” conda package, which is necessary for ipykernel. Create a Jupyter kernel based on your environment.

$ module load miniforge3
$ conda activate my_env
$ python -m ipykernel install --user --name <myenv> --display-name "Python (<myenv>)"

Adding The Virtual Environment To Jupyterhub

At this point your virtual environment will be visible as a kernel in Jupyterhub, but will not function as expected. The Python modules path from your virtual environment need to be added to the kernel file that was created in the step above.

Go to your custom kernels folder and locate the newly created kernel.json file

$ cd .local/share/jupyter/kernels/
$ find . -name kernel.json

Backup the file and edit it to add an “ENV” section, that sets “PYTHONPATH” to point to “site-pacakges” in your conda virtual environment. To find the path to your conda virtual environment, run the following command in your terminal and find the path corresponding to your environment:

​$ conda env list | grep <myenv>

<myenv>                     /PHShome/<usr_name>/.conda/envs/<myenv>

Once you have the path, you will want to set "PYTHONPATH" and "LD_LIBRARY_PATH" to the appropriate locations in the "env" section you created in kernel.json:

• "PYTHONPATH" : "<path-to-env>/lib/<python-version>/site-packages/"
• "LD_LIBRARY_PATH": "<path-to-env>/lib"

If you are unsure of the python version in your kernel, you can look at the folders within the "lib" directory:

$ ls <path-to-env>/lib | grep python

You can also check which python version you are using within the activated conda environment.

(myenv) $ python --version

Your kernel.json will look something like this (your additions to the file are bolded in this example). (Be sure to replace “abc123” with your username!)

{
 "argv": [
 "/PHShome/abc123/.conda/envs/myenv/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],


 "env": {
 "PYTHONPATH": "/PHShome/abc123/.conda/envs/myenv/lib/python3.6/site-packages/",
 "LD_LIBRARY_PATH": "/PHShome/abc123/.conda/envs/myenv/lib"
 },
 "display_name": "Python (myenv)",
 "language": "python"
} 

Now, you should be able to work in your custom environment within Jupyterhub! To switch to your environment, select "Kernel" > "Change kernel" on the top toolbar, and select your environment kernel. You should see the kernel name displayed in the upper right-hand corner of the toolbar.