Overview

Jupyter is a suite of tools for interactive work with data and source code. It is used primarily in data science, but has many different applications.

Jupyter is accessible only on campus—either via VPN or on-site.

You can find the Jupyter documentationhere.

Overview of Jupyter

Jupyter consists of a number of components. The most important of these are:

• Jupyter Notebooks: afile format(with the.ipynb extension) consisting of cells that can contain code in various languages, text in Markdown format, and other elements. Jupyter Notebook is also the name of the standard user interface for editing files of this type. The Jupyter Notebook interface is entirelyweb-based, meaning it runs in a browser. Individualcells in the notebook can be executed, and the user’s input (e.g., Python source code) in the respective cell is then executed. The output appears below the cell and may vary depending on the cell type.
• Jupyter kernels: the programs that execute the code in code cells of a Jupyter notebook. Normally, exactly one compute kernel is assigned to each notebook. Depending on which language(s) and other features are required in the notebook, an appropriate kernel must be selected. ThePython kernel, for example, which enables Python code in code cells, is widely used. Jupyter kernels can run on the same machine as the notebook interface or on a different machine (such as a compute node in a cluster). In the latter case, a JupyterHub server must also be running.
• JupyterLab: an Integrated Development Environment (IDE) that is also used to edit, launch, and manage Jupyter notebooks. JupyterLab includes more features than the notebook interface but is also entirely web-based. On the OMNI cluster, JupyterLab will be set as the default interface.
• JupyterHub: the server component of Jupyter. It allows multiple users to launch notebooks on the same machine.
• Jupyter Enterprise Gateway: While the notebook files are edited on the frontends (i.e., on the login nodes in the case of clusters), the execution of the kernels—and thus the actual computations—typically takes place on a different machine (in the case of clusters, on a compute node within aSlurm job). Jupyter Enterprise Gateway provides the connection between the two, thereby enabling interactive work on clusters.

All of these components will be available on the OMNI cluster.

Working with Jupyter on the OMNI cluster

You can access the Jupyter portal from the university network or via the university VPN by entering the corresponding address in your browser (you’ll find the address in your welcome email).

If you already haveaccess to our systems, you can log in as usual with your username and password.

This will take you to the JupyterHub Control Panel. You’ll usually need to start a new JupyterHub server by clicking the “Start My Server” button. Once your server is running, you’ll be taken to the JupyterLab interface, where you can edit notebooks.

On the JupyterLab interface, you’ll see your files on the left (by default, those in your home directory on the cluster) and a Linux console on the right. This is essentially the same console you see when you connect to the cluster via SSH as usual. Here, for example, you can create workspaces if necessary.

Notebooks and Kernels on the Cluster

If you want to work with notebooks, you must either open an existing notebook by double-clicking a notebook file in the list on the left or create a new notebook via “File” → “New” → “Notebook.”

When you create a new notebook, you must select a kernel. For an existing notebook, you can see which kernel it uses in the top-right corner.

Note: Only kernels with “via SLURM” in their names will run on the compute nodes! Whenever possible, you shouldstart kernelson the compute nodes so that you do not slow down the frontend for all other users. We reserve the right to stop processes that place an excessive load on the frontend without prior warning.

When you start a kernel that uses SLURM, a SLURM job is automatically started for you. In the worst-case scenario, this may result in wait times. You can view and manage your jobs, just like any other jobs, using the appropriateSLURM commands(e.g., `squeue`).

Note: The SLURM jobs in which the Jupyter kernels run have atime limit, just like any other job. When the time limit for your job expires, the kernel will stop. However, the notebook remains open, and you can continue working in it. You can simply restart the kernel using the corresponding button. By default, the “Python 3.7 via SLURM” kernel is started in theshort queue, which means the time limit is one hour.