Jupyter Notebook Culling in Kubeflow

By default, when Jupyter Notebooks are launched, the notebook server hangs on forever even if there is no activity. Clearly, this is a waste of resources, and even more so when GPUs are provisioned for the notebook.

Thankfully, there is notebook culling logic built into the Kubeflow Jupyter Notebook controller. However, it is a) not documented and b) not enabled by default.

How does it look like? Imagine you have two notebooks running:

Then assume that cullings has not been accessed for a while and is ripe for culling:

That’s pretty much what it does! The culling logic stops the Jupyter Notebook server, and you are free to hit the play button to start it back up again and connect to it again.

Setting Up Jupyter Notebook Culling

% kubectl edit deploy -n kubeflow notebook-controller-deployment

First, make sure that the deployment has Istio side-car injection enabled:

labels:
  app: notebook-controller
  kustomize.component: notebook-controller
  sidecar.istio.io/inject: "true" # <-- Add this