Jupyter Kernels

Introduction¶

Jupyter kernels execute the code that you write.

Many packages are preinstalled in our default Python and R environments and these can be extended further as described on the Python and R support pages.

Adding a custom Python kernel¶

You can configure custom Python kernels for running your Jupyter notebooks. This could be necessary and/or recommended in some situations, including:

if you wish to load a different combination of environment modules than those we load in our default kernels
if you would like to activate a virtual environment or conda environment before launching the kernel

The following example will create a custom kernel based on the Miniconda3 environment module (but applies to other environment modules too).

In a terminal run the following commands to load a Miniconda environment module:

module purge
module load Miniconda3/4.8.2

Now create a conda environment named "my-conda-env" using Python 3.6. The ipykernel Python package is required but you can change the names of the environment, version of Python and install other Python packages as required.

conda create --name my-conda-env python=3.6
source $(conda info --base)/etc/profile.d/conda.sh
conda activate my-conda-env
conda install ipykernel
# you can pip/conda install other packages here too

Now create a Jupyter kernel based on your new conda environment:

python -m ipykernel install --user --name my-conda-env --display-name="My Conda Env"

We must now edit the kernel to load the required NeSI environment modules before the kernel is launched. Change to the directory the kernelspec was installed to ~/.local/share/jupyter/kernels/my-conda-env, (assuming you kept --name my-conda-env in the above command):

cd ~/.local/share/jupyter/kernels/my-conda-env

Now create a wrapper script, called wrapper.sh, with the following contents:

#!/usr/bin/env bash

# load required modules here
module purge
module load Miniconda3/4.8.2

# activate conda environment
source $(conda info --base)/etc/profile.d/conda.sh 
conda deactivate  # workaround for https://github.com/conda/conda/issues/9392
conda activate my-conda-env

# run the kernel
exec python $@

Make the wrapper script executable:

chmod +x wrapper.sh

Next edit the kernel.json to change the first element of the argv list to point to the wrapper script we just created. The file should look like this (change <username> to your NeSI username):

{
 "argv": [
 "/home/<username>/.local/share/jupyter/kernels/my-conda-env/wrapper.sh",
 "-m",
 "ipykernel_launcher",
 "-f",
 "{connection_file}"
 ],
 "display_name": "My Conda Env",
 "language": "python"
}

After refreshing JupyterLab your new kernel should show up in the Launcher as "My Conda Env".

You can also configure a shared Python kernel that others with access to the same NeSI project will be able to load. If this kernel is based on a Python virtual environment, Conda environment or similar, you must make sure it also exists in a shared location (other users cannot see your home directory).

The example below shows creating a shared Python kernel based on the Python/3.8.2-gimkl-2020a module and also loads the ETE/3.1.1-gimkl-2020a-Python-3.8.2 module.

In a terminal run the following commands to load the Python and ETE environment modules:

module purge
module load Python/3.8.2-gimkl-2020a
module load ETE/3.1.1-gimkl-2020a-Python-3.8.2

Now create a Jupyter kernel within your project directory, based on your new virtual environment:

python -m ipykernel install --prefix=/nesi/project/<project_code>/.jupyter --name shared-ete-env --display-name="Shared ETE Env"

Next change to the kernel directory, which for the above command would be:

cd /nesi/project/<project_code>/.jupyter/share/jupyter/kernels/shared-ete-env

Create a wrapper script, wrapper.sh, with the following contents:

#!/usr/bin/env bash

# load necessary modules here
module purge
module load Python/3.8.2-gimkl-2020a
module load ETE/3.1.1-gimkl-2020a-Python-3.8.2

# run the kernel
exec python $@

Note we also load the ETE module so that we can use that from our kernel.

Make the wrapper script executable:

chmod +x wrapper.sh

Next, edit the kernel.json to change the first element of the argv list to point to the wrapper script we just created. The file should look like this (change <project_code> to your NeSI project code):

{
 "argv": [
 "/nesi/project/<project_code>/.jupyter/share/jupyter/kernels/shared-ete-env/wrapper.sh",
 "-m",
 "ipykernel_launcher",
 "-f",
 "{connection_file}"
 ],
 "display_name": "Shared Conda Env",
 "language": "python"
}

After refreshing JupyterLab your new kernel should show up in the Launcher as "Shared Virtual Env".

Custom kernel in a Singularity container¶

An example showing setting up a custom kernel running in a Singularity container can be found on our Lambda Stack support page.

Adding a custom R kernel¶

You can configure custom R kernels for running your Jupyter notebooks. The following example will create a custom kernel based on the R/4.4.1-foss-2023a environment module.

In a terminal run the following commands to load the required environment modules:

module purge
module load JupyterLab/2024.05.0-foss-2023a-4.2.0
module load IRkernel/1.3.2-foss-2023a-R-4.4.1
module load Python/3.11.6-foss-2023a

The IRkernel module loads the R module as a dependency and provides the R kernel for Jupyter. Python is required to install the kernel (since Jupyter is written in Python).

To create the kernel:

R -e "IRkernel::installspec(name='r4.4.1', displayname = 'R/4.4.1', user = TRUE)"

We must now to edit the kernel to load the required NeSI environment modules when the kernel is launched. Change to the directory the kernelspec was installed to (~/.local/share/jupyter/kernels/r4.4.1, assuming you kept --name r4.4.1 in the above command):

cd ~/.local/share/jupyter/kernels/r4.4.1

Now create a wrapper script in that directory, called wrapper.sh, with the following contents:

#!/usr/bin/env bash

# load required modules here
module purge
module load R/4.4.1-foss-2023a

# run the kernel
exec R $@

Make the wrapper script executable:

chmod +x wrapper.sh

Next edit the kernel.json to change the first element of the argv list to point to the wrapper script we just created. The file should look something like this (change <username> to your NeSI username):

{
 "argv": [
 "/home/agresearch.co.nz/<username>/.local/share/jupyter/kernels/r4.4.1/wrapper.sh",
 "--slave",
 "-e",
 "IRkernel::main()",
 "--args",
 "{connection_file}"
 ],
 "display_name": "R/4.4.1",
 "language": "R"
}

After refreshing JupyterLab your new R kernel should show up in the Launcher as "R/4.4.1".

Jupyter Kernels

Introduction¶

Adding a custom Python kernel¶

Sharing a Python kernel with your project team members¶

Custom kernel in a Singularity container¶

Adding a custom R kernel¶