General

The Anaconda distribution is a collection of software for scientific purposes. It includes a Python installation, an R installation, and the Condapackage manager, which can be used to install Anaconda packages. Since the full Anaconda distribution, with all its packages, takes up a great deal of storage space, there is also a versioncalled Miniconda, which contains only Python, Conda, and a few basic packages. Both versions are completely free and open-source.

Miniconda is available on the OMNI cluster. It is based on Python 3.8.5 and Conda 4.9.2. To use Miniconda, you must load theminiconda3module:

module load miniconda3

You can find the Condadocumentation here. A complete overview of available packages in the Anaconda distribution is availablehere. Note that Conda can also install third-party packages and manage multiple sources (called “channels” in Conda). The most important community channel isconda-forge.

Note for TensorFlow and PyTorch users: Software with GPU support is already preinstalled on the cluster as part of ourGPU stack.

Setting Up Conda

You primarily use Conda with the `conda` command. For example, you can use:

conda help

to display an overview of the available Conda commands. A complete list can be found in the Conda documentationhere.

If you’re using Conda for the first time, you must initialize it with the`conda init`command, then close and reopen the shell (for example, by terminating the SSH connection to the cluster and reconnecting). Here is the command with an example output:

[demo_user@hpc-login01 ~]$ conda init
no change     /cm/shared/omni/apps/miniconda3/condabin/conda
no change     /cm/shared/omni/apps/miniconda3/bin/conda
no change     /cm/shared/omni/apps/miniconda3/bin/conda-env
no change     /cm/shared/omni/apps/miniconda3/bin/activate
no change     /cm/shared/omni/apps/miniconda3/bin/deactivate
no change     /cm/shared/omni/apps/miniconda3/etc/profile.d/conda.sh
no change     /cm/shared/omni/apps/miniconda3/etc/fish/conf.d/conda.fish
no change     /cm/shared/omni/apps/miniconda3/shell/condabin/Conda.psm1
no change     /cm/shared/omni/apps/miniconda3/shell/condabin/conda-hook.ps1
no change     /cm/shared/omni/apps/miniconda3/lib/python3.8/site-packages/xontrib/conda.xsh
no change     /cm/shared/omni/apps/miniconda3/etc/profile.d/conda.csh
modified      /home/js056352/.bashrc

==> For changes to take effect, close and reopen your current shell. <==

(After logging out and logging back in...)

(base) [demo_user@hpc-login01 ~]$

Two things are noticeable: First, `conda init`has modified the user’s`.bashrc`file(if you look in the file, you can see that the `PATH` variable has been modified). Second, the command line now displays(base)on the left. This is the active Conda environment. Conda environments are explained in the next section.

Warning: There is arisk of unintended side effects if you load or unload theminiconda3module, especially after running `conda init`, since this changes the order of directories in the PATH variable. If you are using Miniconda, we recommend keeping the`miniconda3`modulepermanently active, for example, by adding `module load miniconda3`before `conda init` in your`.bashrc` file(see also Linux Basics).

Please also note that the`python`and`python3`commands point to Conda’s own Python installation once Conda is active. This Python installation differs from the regular system Python on the cluster. In addition, a separate Python instance is created for each Conda environment you create (see the next section).

There are additional options for configuring Conda, which are explainedhere.

Conda Environments

Conda environments, similar to Python’s virtual environments, are used to isolate software installations for different projects from one another. However, Conda environments have the advantage of not being internal to Python. Note: Youmust set up at least one separate Conda environment, as you cannot install packages in the default (base) environment. You can list the available environments using the `conda env list` command:

$ conda env list

# conda environments:
#
base                  *  /cm/shared/omni/apps/miniconda3
test                     /home/demo_user/.conda/envs/test
                         /home/demo_user/miniconda3

As you can see from this sample output, the `base` environment is located in the installation directory, where regular users do not have write permissions, while environments you create yourself are located in your home directory.

Creating an Environment

You can create your own environment using the following command:

conda create --name 

You can also install packages at this point. In particular, if you want to use a specific Python version (Conda treats Python like any other package), you can specify it directly:

conda create --name  python=3.8

By the way, this syntax (=) works for all Conda packages if you need a specific version. However, we recommend specifying a specific version only when absolutely necessary.

Using an Environment

You must activate an environment before you can use it. It’s also recommended to deactivate the environment after use. The commands for this are:

conda activate 

and

conda deactivate

The active environment is displayed in parentheses on the left side of the command line. If you want to deactivate all environments, you can enter `conda deactivate` again.

For more information on Conda environments, see the Conda documentationhere.

Installing Packages

You install Conda packages using the command:

conda install 

As mentioned earlier, Conda packages are installed in a subfolder of your home directory. If you receive an error message about missing write permissions, check to make sure you aren’t accidentally inthe base environment.

Note: If you want to install Python packages, you should install them using `conda install` rather than `pip install` whenever possible, as the latter can cause conflicts.

It should also be noted that packages can be installed from sources other than the official Conda repository. Like most package managers, Conda allows you to add additionalchannels. One important unofficial channel, for example, isconda-forge, which contains community packages. Installing unofficial packagesis at your own risk and should be avoided whenever possible.

Detailed information on Conda packages can be found in theConda documentation here; more information on Conda channels can be foundhere. A complete overview ofavailable packages in the Anaconda distribution is availablehere.

Conda and Slurm Jobs

Some of our users have reported issues arising from the complex interplay of the following factors:

• Loading the Conda module
• Activating the Conda environment
• Executing the commands in the user’s.bashrc file
• The configuration options in Slurm scripts (#SBATCH ...) regarding which environment variables are passed to the job.

There is no general solution to this problem because much depends on the modules used and the order of the steps listed above.

A job script in the following form has proven to be a solution for several issues:

#!/bin/bash
#SBATCH (...)

source ~/.bashrc
conda deactivate
conda activate