Software Deployment with Conda

Intro

Conda is an open source package management system and environment management system for installing multiple versions of software packages and their dependencies and switching easily between them. See the conda documentation for more.

Environments

Activating the root environments

Every beamline workstation has some standard conda environments. All users can access them, but they can only be installed or updated with root privileges. You can list the names and locations of the available environments like so:

$ conda env list
# conda environments:
#
analysis-17Q2.0          /opt/conda_envs/analysis-17Q2.0
collection-17Q2.0        /opt/conda_envs/collection-17Q2.0
root                  *  /opt/conda

To use an environment, active it.

$ source activate analysis-17Q2.0
discarding /opt/conda/bin from PATH
prepending /opt/conda_envs/analysis-17Q2.0/bin to PATH

You can inspect the packages installed in the current environment using the command

$ conda list
# packages in environment at /opt/conda_envs/analysis-17Q2.0:
#
amostra                   0.2                      py36_0
anaconda-client           1.6.2                    py36_0
analysis                  17Q2.0                   py36_4
attrs                     16.3.0                   py36_1
bleach                    1.5.0                    py36_0
bluesky                   0.9.0                    py36_0
<snip>

Creating custom user environments

If the standard system conda environments are missing some packges that you want to use, you can create your own.

You can copy one of the standard system environments and add some packages.

$ conda create -n myenv --clone analysis some_package another_package

Or, you can create a new environment from scratch. (If you plan to use the environment with Jupyter, you must include the ipykernel package along with any others you want.)

$ conda create -n myenv ipykernel some_package another_package

Activate the environment.

$ source activate myenv

If the package is available from pip but not conda, you can use pip. Make sure you have activated the environment first, as we did just above.

$ pip install some_other_package

Access custom user environments in NSLS-II’s JupyterHub

Users can connect to their own user-created conda environments through JupyterHub.

1. Activate your custom environment.

   $ source activate myenv
   

2. Create a new Jupyter kernel for the environment.

   $ python -m ipykernel install --user --name myenv --display-name "Python (myenv)"
   Installed kernelspec myenv in /home/dallan/.local/share/jupyter/kernels/myenv
   

   The --name value is used by Jupyter internally. These commands will overwrite any existing kernel with the same name. --display-name is what you see in the notebook menus. For details see this section of the IPython documentation

3. Specify a host in the kernel file.

   The kernel requires one simple customization to work on NSLS-II’s JupyterHub deployment. We need to specify which host to run the kernel on.

   You can check the hostname of the current machine like so:

   $ hostname
   xf23id1-srv1
   

   Open the kernel file that we generated above. You can get its location from the output displayed by our kernel creation command in the previous step. In our example, recall you that it showed

   Installed kernelspec myenv in /home/dallan/.local/share/jupyter/kernels/myenv
   

   so we will open the file kernel.json in that directory.

   {
    "argv": [
    "/home/dallan/conda_envs/myenv/bin/python",
    "-m",
    "ipykernel_launcher",
    "-f",
    "{connection_file}"
    ],
    "display_name": "Python (myenv)",
    "language": "python"
   }
   

   Add a new item, "host" mapped to the hostname of the machine to run this kernel on.

   {
    "argv": [
    "/home/dallan/conda_envs/myenv/bin/python",
    "-m",
    "ipykernel_launcher",
    "-f",
    "{connection_file}"
    ],
    "display_name": "Python (myenv)",
    "language": "python",
    "host": "xf23id1-srv1"
   }
   

   Notice that we added a comma on the second-to-last line, after "python". There is no comma after the last entry. JSON files are strict about this. (This often trips up Python users, because Python tolerates trailing commas in lists.)

Now, when you create a new notebook in JupyterHub, your new kernel should appear in the menu. (You may need to refresh the browser, but you should not need to log out of JupyterHub or restart your server.)

Internal Anaconda Server

We run an internal anaconda server at https://conda.nsls2.bnl.gov.

The anaconda channel contains the “official” packages distributed by Continuum Analytics, including widely-used packages like numpy. When Continuum releases new version of packages, we can vet them before we make them available to users on this channel. The defaults channel is not vetted by us; it should never be used internally.

The nsls2-tag channel is where we put the latest tagged stable versions of every package not officially distributed by Continuum. This includes in-house packages like ophyd and bluesky and other dependencies that don’t happen to be packaged by Continuum yet. The nsls2-dev channel is where we put bleeding-edge development versions of these packages.