General

The Anaconda distribution is a collection of software for scientific purposes. It contains a Python installation, an R installation and the package manager Conda, which can be used to install Anaconda packages. As the complete Anaconda distribution with all packages takes up a lot of disk space, there is also the Miniconda version, which only contains Python, Conda and a few basic packages. Both versions are completely free and open source.

Miniconda is available on the OMNI cluster. It is based on Python 3.8.5 and Conda 4.9.2. To use Miniconda, the miniconda3 module must be loaded:

module load miniconda3

The Conda documentation can be found here. A complete overview of available packages of the Anaconda distribution can be found here. Note that Conda can also install third-party packages and manage multiple sources (called "channels" in Conda). The most important community channel is conda-forge.

Note for TensorFlow and PyTorch users: There is already pre-installed software with GPU support on the cluster in our GPU stack.

Setting up Conda

You essentially use Conda with the conda command. For example, you can use:

conda help

to display an overview of the possible Conda commands. You can find a complete list in the Conda documentation here.

If you are using Conda for the first time, you need to initialize it with the conda init command, and then close and reopen the shell (for example, by closing the SSH connection to the cluster and reconnecting). Here is the command with a sample output:

[demo_user@hpc-login01 ~]$ conda init
no change /cm/shared/omni/apps/miniconda3/condabin/conda
no change /cm/shared/omni/apps/miniconda3/bin/conda
no change /cm/shared/omni/apps/miniconda3/bin/conda-env
no change /cm/shared/omni/apps/miniconda3/bin/activate
no change /cm/shared/omni/apps/miniconda3/bin/deactivate
no change /cm/shared/omni/apps/miniconda3/etc/profile.d/conda.sh
no change /cm/shared/omni/apps/miniconda3/etc/fish/conf.d/conda.fish
no change /cm/shared/omni/apps/miniconda3/shell/condabin/Conda.psm1
no change /cm/shared/omni/apps/miniconda3/shell/condabin/conda-hook.ps1
no change /cm/shared/omni/apps/miniconda3/lib/python3.8/site-packages/xontrib/conda.xsh
no change /cm/shared/omni/apps/miniconda3/etc/profile.d/conda.csh
modified /home/js056352/.bashrc

==> For changes to take effect, close and re-open your current shell. ==

(After logging out and logging in again...)

(base) [demo_user@hpc-login01 ~]$

There are two things to recognize: First, conda init has modified the .bashrc file
file of the user (if you look in the file, you can see that the PATH variable has been modified). Secondly, the command line now shows left (base). This is the active Conda environment. Conda environments are explained in the next section.

Attention: There is a risk of unintended side effects when you load or unload the miniconda3 module, especially after a conda init, as the order of the directories in the PATH variable is then changed. If you use Miniconda, we recommend that you have the miniconda3 module permanently active, e.g. by adding module load miniconda3 toyour .bashrc file before the conda init(see also Linux basics).

Please also note that the python and python3 commands point to a conda-specific Python as soon as Conda is active. This Python installation is different from the regular system Python on the cluster. In addition, a separate Python instance is created for each Conda environment that you create (see next section).

You have further options for configuring Conda, which are explained here
are explained here.

Conda environments

Conda environments, like Python's virtual environments, are used to separate software installations for different projects. However, Conda environments have the advantage that they are not internal to Python. Caution: You must set up at least one Conda environment of your own, as you cannot install packages in the standard environment(base). You can use the conda env list command to list the available environments:

$ conda env list

# conda environments:
#
base * /cm/shared/omni/apps/miniconda3
test /home/demo_user/.conda/envs/test
                         /home/demo_user/miniconda3

As you can see from this example output, the base environment is located in the installation directory, in which normal users do not have write access, while environments that you set up yourself are located in your home directory.

Creating an environment

You create your own environment with the command:

conda create --name 

You can also install packages at this point. In particular, if you want to use a specific Python version (Python is treated by Conda like any other package), you can specify this directly:

conda create --name  python=3.8

This syntax (= ) works for all Conda packages if you need a specific version. However, we recommend that you only specify a specific version if it is absolutely necessary.

Using an environment

You must activate an environment before you can use it. It is also advisable to deactivate the environment again after use. The commands for this are

conda activate 

and

conda deactivate

The active environment is displayed in brackets on the left-hand side of the command line. If you do not wish to have an environment activated at all, you can enter conda deactivate again.

You can find more information on Conda environments in the Conda documentation here.

Installing packages

You install Conda packages with the command:

conda install 

As already mentioned, Conda packages are installed in a subfolder of your home directory. If you get an error message about missing write permissions, check that you are not in the base environment by mistake.

Attention: If you want to install Python packages, you should install them with conda install and not with pip install, as the latter can lead to conflicts.

It should also be noted that not only packages from the official Conda repository can be installed. Conda, like most package managers, has the option of adding additional channels. An important non-official channel is, for example, conda-forge, which contains community packages. The installation of unofficial packages is at your own risk and should be avoided as far as possible.

Detailed information on Conda packages can be found in the Conda documentation here, more information on Conda channels can be found here. A complete overview of available packages of the Anaconda distribution can be found here.

Conda and Slurm jobs

Some of our users have reported problems resulting from the complex interaction of the following factors:

• Loading the Conda module
• Activating the Conda environment
• The execution of the commands in the user's .bashrc file
• The setting options of Slurm scripts(#SBATCH ...) regarding the question of which environment variables are passed to the job.

There is no general solution to this problem because much depends on the modules used and the sequence of the above steps.

A job script of the following form has proved to be a solution for several problems:

#!/bin/bash
#SBATCH (...)

source ~/.bashrc
conda deactivate
conda activate