Deploy JupyterHub and JupyterLab on Ubuntu Server

8 min readMay 25, 2023

If you are a team-leader or a teacher working in the area of machine learning like me, you might love having a Jupyter server where all your team members or students can concurrently work on their own machine learning experiments or lessons. In this guide, I demonstrate how you can easily spin up a JupyterHub server from the ground-up on any decent machine on your local network running an Ubuntu Server. I also show how you can serve read-only or user-editable files to all your users and enable them to install their own Conda environments and packages. Common areas of confusion and troubleshooting are covered. Finally, I go over some additional tasks that’ll make your server even more robust and friendly, such as integrating git, managing user groups and permissions, and distribute your custom python packages to all users.

This guide was compiled from documentations from JupyterHub [1], JupyterLab [2], Conda, Ubuntu and other sources such as StackOverflow.

Prerequisites

Ubuntu Server with SSH access to shell
Python (modern Ubuntu servers come bundled with Python)

Goals

JupyterHub + JypyterLab will be installed as a webservice in /opt using Python.
Conda will be installed globally. Conda is intended to be the preferred Python environment to serve the actual computing tasks of the users.
A shared conda environment will be created for all users. They can use it, but not modify it.
Demonstrate how users can create their own private conda environments.

Steps

Create python virtual environment for JupyterHub

Create a Python virtual environment in /opt for jupyterhub.
sudo python3 -m venv /opt/jupyterhub/
Tip: You may need to install python3-venv first to be able to create a virtual environment.

Install JupyterHub and JupyterLab

Install jupyterhub, jupyterlab, and UI related packages in the jupyterhub virtual environment we just created.
sudo /opt/jupyterhub/bin/python3 -m pip install wheel
sudo /opt/jupyterhub/bin/python3 -m pip install jupyterhub jupyterlab
sudo /opt/jupyterhub/bin/python3 -m pip install ipywidgets
Tip: You can specify the version of the package like this: pip install jupyterhub==3.1 but that may create incompatibility issues and problems with starting of the JupyterHub service successfully. I installed the latest versions at this step, but went back and downgraded jupyterlab according to my specific needs.
Install nodejs, npm and then, configurable-http-proxy systemwide. These are required by jupyterlab.
sudo apt install nodejs npm
sudo npm install -g configurable-http-proxy

Configure JupyterHub

To keep things together, create any configurations under the virtualenv, i.e. under /opt/jupyterhub/etc/.

Create a folder for jupyturhub configurations and navigate to it.
sudo mkdir -p /opt/jupyterhub/etc/jupyterhub
cd /opt/jupyterhub/etc/jupyterhub
Generate the default configuration file.
sudo /opt/jupyterhub/bin/jupyterhub --generate-config
Edit the configuration file to make JupyterLab interface start by default, rather than Jupyter Notebook.
sudo nano jupyterhub_config.py
And update the following parameter to:
c.Spawner.default_url = '/lab'

Set up JupyterHub as a system service

Create a folder for systemd service file.
sudo mkdir -p /opt/jupyterhub/etc/systemd
Create a service file.
sudo nano /opt/jupyterhub/etc/systemd/jupyterhub.service
Paste the following into the service file:

[Unit]
Description=JupyterHub
After=syslog.target network.target

[Service]
User=root
Environment="PATH=/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/jupyterhub/bin"
ExecStart=/opt/jupyterhub/bin/jupyterhub -f /opt/jupyterhub/etc/jupyterhub/jupyterhub_config.py

[Install]
WantedBy=multi-user.target

Create a symlink to this sevice file from the system-wide systemd folder.
sudo ln -s /opt/jupyterhub/etc/systemd/jupyterhub.service /etc/systemd/system/jupyterhub.service
Tell systemd to reload its configuration files, and then enable and start the service.
sudo systemctl daemon-reload
sudo systemctl enable jupyterhub.service
sudo systemctl start jupyterhub.service
sudo systemctl status jupyterhub.service # check status of service
You can now test the service by navigating to http://<your-server-ip>:8000 in your browser. You should see the JupyterLab interface.

Install Conda

Install conda systemwide. This involves a series of simple commands, more details of which can be found here [3].
curl https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc | gpg --dearmor > conda.gpg
sudo install -o root -g root -m 644 conda.gpg /etc/apt/trusted.gpg.d/
echo "deb [arch=amd64] https://repo.anaconda.com/pkgs/misc/debrepo/conda stable main" | sudo tee /etc/apt/sources.list.d/conda.list
sudo apt update
sudo apt install conda
sudo ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh
This installs conda in /opt/conda. If you wish to use the conda command in terminal now, without restarting the shell/computer, it is available from /opt/conda/bin/conda.

Configure Conda

Install a default conda environment for all users. Note that this is different from the base environment that is included when Conda is first installed. It's a good practice to not work in the base environment.

Create a folder for new conda environments.
sudo mkdir /opt/conda/envs
Tip: At this time, also give write permissions to the user/group intended to install packages to this environment. Otherwise conda install will fail with the EnvironmentNotWritableError error.
sudo chown -R root:<groupname> /opt/conda/envs
sudo chmod -R 775 /opt/conda/envs
Create a new conda environment in this folder. Here we will install tensorflow (which includes Python).
sudo /opt/conda/bin/conda create --prefix /opt/conda/envs/cpuenv tensorflow
We need to install ipykernel in this environment so that it can be used with Jupyter. Be careful not to install ipykernel in the default base environment.
sudo /opt/conda/bin/conda install --prefix /opt/conda/envs/cpuenv ipykernel
Make this environment visible to Jupyter by installing the kernel spec system-wide by putting it into /usr/local. This step makes the environment visible on the JupyterHub UI. Read more here [4].
sudo /opt/conda/envs/cpuenv/bin/python -m ipykernel install --prefix /usr/local/ --name 'cpuenv' --display-name "CPU Environment"
A folder /usr/local/share/jupyter/kernels/cpuenv will be created.

Per-user Conda environments

To let users set up their own environments, they should login into shell and run conda init or /opt/conda/bin/conda and thereafter use conda to create their own environment. I believe they should create the environment in their home directory (to which they have write permissions to) instead of the /opt/conda/envs directory. They will also need to install ipykernel (required by Jupyter) in the new environment. Finally, they can enable the kernel to appear in JupyterHub UI by running the following command:
/path/to/env/bin/python -m ipykernel install --user --name 'user-env' --display-name "User Environment"

Additional Tasks

Install Git on the Ubuntu server

sudo apt install git-all

Create users and user groups on Ubuntu

Before a user can use JupyterLab, they need to be an Ubuntu user first. For convenience, we will also add them to a group called jupyterhub-user.
sudo adduser <username>
sudo addgroup jupyterhub-user
sudo adduser <username> jupyterhub-user

Set a user as admin on JupyterHub

Navigate to /opt/jupyterhub/etc/jupyterhub/jupyterhub_config.py and add the following line:
c.Authenticator.admin_users = {'<username>'}
Tip: make sure to use {} instead of set()
Restart the service:
sudo systemctl restart jupyterhub.service

Disable conda from automatically activating its base environment

The following command will add a .condarc file in the home directory of the user.
conda config --set auto_activate_base false

Create a read only shared folder for all users

Create a folder in /srv. You can add data to this folder however you like.
sudo mkdir /srv/data
Note that the permission of this folder are drwxr-xr-x by default.
Next, optionally, create a symlink to this folder in each user's home directory. To automate this, you can add this link in /etc/skel; anything placed in this folder shows up in the home directory of any new user created.
sudo ln -s /srv/data /etc/skel/data
Note 1: Make sure to do this for existing users' home dirs manually.
Note 2: You will want to grant at least one user (besides root) the ability to write (upload) to the /srv/data folder. This user can then add data from the JuoyterLab UI, and remaining users can read from it.
sudo chown -R <username>:<groupname> /srv/data

Create a folder for users to share data with each other

Each uploader will have write access to their files (only), while everyone else will have read access to those files.
sudo mkdir /srv/shared
sudo chown root:<jupyterhub-users-group> /srv/shared
Using chmod g+s tells new files to use the default permissions for the group jupyterhub-users-group which is rw-r--r--.
sudo chmod 777 /srv/shared
sudo chmod g+s /srv/shared
Finally, add a symlink to this folder in /etc/skel/ for future users and in each existing user's home directory.

Let users change their password

<your_server_ip>/hub/auth/change-password

Install git extension for JupyterLab

Make sure you have git >= 2.x installed in Ubuntu and JupyterLab >= 3.0 installed in the python3 virtual environment that houses JupyterLab.
Note that jupyterlab-git is not compatible with jupyterlab 4 yet.
sudo /opt/jupyterhub/bin/pip install --upgrade jupyterlab jupyterlab-git
Downgrade jupyterlab if needed.
sudo /opt/jupyterhub/bin/pip install jupyterlab==3.6.3
Restart the JupyterHub service.
sudo systemctl restart jupyterhub.service
The JupyterLab UI should show the git button and the menu item. Additionally, it should also show the git extension already enabled/installed in the left sidebar.
You can now clone a git repository from the JupyterLab UI. When asked for a password, make sure to provide a fine-grained (preferred) or a personal access token (not preferred) instead of your account password. GitHub no longer supports third-party authentication using passwords. You can create a token from your GitHub account settings.

Make your custom python package available to all users

I prefer the strategy of adding our package’s folder path to a conda environment installation instead of adding the path to users’ home directories or add the package itself to the site-packages directory of the environment [5].
First, create a folder and set ownership for the desired user so they can upload their package to this folder.
sudo mkdir /srv/_packages
sudo chown -R <username>:<groupname> /srv/_packages
Upload files to this folder however you like. I cloned my package from GitHub so that my folder looks like:
/srv/_packages/my_package_1
To make this package available to other users (note that they have read access to the _packages folder by default, which is perfect), we will add the path to this folder to conda's site-packages directory, for every environment that is created. Just adding the path to the base environment's site-packages, did not add it to other environments for me [5].
Create a *.pth file in the site-packages directory of the base environment.
sudo nano /opt/conda/lib/python3.10/site-packages/my_packages.pth
Add the path to the package folder in the file. Save and exit.
/srv/_packages
Create a symlink to this file in the site-packages directory of every new environment that is created. (I am not sure if this is intended or I am doing something unnecessary, but I was able to import packages from cpuenv environment only after I created this symlink.)
sudo ln -s /opt/conda/lib/python3.10/site-packages/my_packages.pth /opt/conda/envs/cpuenv/lib/python3.11/site-packages/my_packages.pth

Things I am still figuring out

How to remove the default python kernel showing up in JupyterLab UI for the current version of JupyterLab.
Successfully set up a static ip address for the server in your local network using netplan. Once a static ip address is obtained, you can also use LetsEncrypt to get a free SSL certificate for your server. This will allow you to access your server using https instead of http and also get rid of the Not Secure warning in the browser.

References

Install JupyterHub and JupyterLab from the ground-up. Link
The Littlest JupyterHub How-To Guides. Link
RPM and Debian Repositories for Miniconda. Link
Installing the IPython kernel. Link
Stackoverflow: Anaconda: Permanently include external packages (like in PYTHONPATH). Link