Both BLAS and LAPACK are available in several forms on the OMNI cluster.

OpenBLAS is a widely used library for calculating vector and matrix operations that implements both the BLAS interface (BLAS = Basic Linear Algebra Subprograms) and the LAPACK interface (LAPACK = Linear Algebra PACKage). OpenBLAS version 0.3.7 (compiled with GCC 9.3.0) is installed on the OMNI cluster as part of OpenHPC.

In addition, SCALAPACK version 2.1.0 is installed on the OMNI cluster. SCALAPACK contains some of the LAPACK routines in parallelized form.

Intel MKL is also installed on the cluster, which also implements the BLAS and LAPACK interfaces.

Environment modules

OpenBLAS environment module

To use OpenBLAS, the corresponding module must be loaded:

module load openblas

Note: Due to the module structure of OpenHPC, not all modules are always visible with module avail. OpenBLAS is only displayed with a module avail if the corresponding GCC module(module load gnu9) is already loaded. The module gnu9 is loaded on the OMNI by default, but it could happen that you have unloaded the module intentionally or accidentally. In this case, the information on the OpenBLAS module can be retrieved with

module spider openblas

can be displayed.

SCALAPACK environment module

The SCALAPACK module can be loaded with

module load scalapack

Note: As SCALAPACK, like OpenBLAS, also comes from OpenHPC, it is not necessarily visible with module avail, but only if a compiler module and an MPI module are already loaded. The modules gnu9 and openmpi4 are loaded by default.

Analogous to OpenBLAS, SCALAPACK can always be loaded with

module spider scalapack

can always be displayed. Please note that SCALAPACK is dependent on OpenBLAS. This means that the OpenBLAS module is also loaded when SCALAPACK is loaded.

In addition to the standard combination of GCC 9.3.0 and OpenMPI 4, other variants of SCALAPACK are available which are linked against other compilers and MPI libraries. You can also display the complete list with module spider scalapack.

$ module spider scalapack
(...)
    Description:
      A subset of LAPACK routines redesigned for heterogeneous computing
      
    You will need to load all module(s) on any one of the lines below before the "scalapack/2.1.0" module is available to load.

      gnu9/9.3.0 impi/2019.5
      gnu9/9.3.0 openmpi4/4.0.4
   (...)
      intel/19.1.2.100006_cm9.0_ae8d786e90  impi/2020.2.100006_cm9.0_ae8d786e90
      intel/19.1.2.100006_cm9.0_ae8d786e90 openmpi4/4.0.4

As can be seen in this abbreviated list, there are a number of combinations of compiler (both Intel and GCC) and MPI library with which SCALAPACK works. As mentioned in the text, all modules must be loaded on one line (e.g. module load gnu9/9.3.0 impi/2019.5) in order to use SCALAPACK.

Compiling and linking

Compiling with OpenBLAS libraries

When loading the OpenBLAS module, the following environment variables are set:

• OPENBLAS_DIR: Installation directory of OpenBLAS
• OPENBLAS_LIB: Library directory of OpenBLAS
• OPENBLAS_INC: Include directory of OpenBLAS

If you want to compile a program that uses OpenBLAS, you must specify the corresponding directories when compiling or linking.

In addition to this, certain OpenBLAS directories are also added to the environment variables PATH, LD_LIBRARY_PATH and others when the module is loaded. You can use module show openblas to display in detail which settings are made.

Attention: OpenBLAS uses dynamically loaded libraries (shared objects). If you link your program against OpenBLAS, the OpenBLAS library directory must also be in LD_LIBRARY_PATH during execution, i.e. you must also load the openblas module during execution.

Compiling with SCALAPACK

Similar to OpenBLAS, SCALAPACK also sets certain environment variables that can be used during compilation.

• SCALAPACK_DIR: the SCALAPACK installation directory
• SCALAPACK_LIB: the SCALAPACK library directory

You can display the complete list with module show scalapack. Like OpenBLAS, SCALAPACK also requires that the module is loaded at runtime (or at least that the shared objects are available in LD_LIBRARY_PATH ).