Making Jupyter Kernels persistent in AWS Sagemaker
So our AI team has been using AWS Sagemaker for a while at Decathlon, and I have to say… it has been an interesting journey so far.
Yes, Sagemaker is an amazing tools suite that literally crushes the time-to-market, from setting up our infrastructure to deliver results. It is uncanny. The Sagemaker API is amazing and the serving options are wide (from End-points to Batch Transform).
We have high expectations regarding industrializing AI products in Decathlon to serve our users : from forecasting future sales (pretty much a standard use-case in Retail) to image recognition to help users improving their movements or gestures in their sport practise.
Use cases are numerous, and Sagemaker helps us from prototyping and exploring to scale & serve productive cases.
There are however a few caveats, as deploying an industrial application might require a deep knowledge of the Sagemaker architecture. The most recent we experienced would be keeping heavy & custom-made Jupyter kernels remanent after an instance reboot.
Sagemaker does provide a few custom kernels including (but not limited to) MXNet, Tensorflow, etc…
You can of course install your own conda environment and install it as a Jupyter kernel :
conda create --name myenv python=3.7 somepackagespython -m ipykernel install --user --name myenv --display-name "myenv"
Note : if you create your Conda environment without using ipykernel to install it as a Jupyter kernel, you will not see it displayed in the list of available kernels.
However, let’s say we need an entirely different set of packages and these packages are sufficiently heavy so reinstalling them at each instance start can become really annoying… Some packages can take a long time installing, due to dependencies (especially in complex environments). For example, installing fbprophet from conda-forge might take a while (10–15min every time in our case). And yes, we’ve been following Anaconda’s recommendations to improve conda’s performances.
It wouldn’t have been an issue if after stopping & re-starting our Sagemaker instance, if our Jupyter kernel and conda environment didn’t have the most annoying habit of disappearing.
Therefore, I’ve been looking for a way to make these kernels remanent in Sagemaker.
Needless to say, AWS Sagemaker is built on top of famous service EC2. As some of you might know, EC2 is not to be considered as a persistent storage. I guess there is an EBS (Elastic Block Storage) behind the Sagemaker notebook instance ; the only folder that will stay intact after our instance restart will be /home/ec2-user/SageMaker/ or ~/SageMaker/. Anything outside this folder will be reset to a “default” state defined by AWS Sagemaker machines definitions.
…including all conda configurations and data.
Conda environments are installed in the default path ~/anaconda3/envs/ (it might differ depending on your Anaconda version/Miniconda version/OS or other parameters, but this is the default path for Sagemaker notebook instances). When you execute conda env list, conda will actually iterate on all subfolders in ~/anaconda3/envs/ and return each subfolder’s name as an environment.
Still with me ?
Taking all of this into account, if you :
- install your environment in the persistent storage in your Sagemaker instance,
- indicate to conda that it should look for an environment in this folder,
… you should be able to install this conda environment into Jupyter as a kernel.
The AWS documentation does mention the possibility of installing a conda environment and installing it in Jupyter as a kernel at the Sagemaker notebook instance creation, using a Lifecycle Configuration Script, but they do not address the situation when the aforementioned environment installation takes forever.
A Lifecycle Configuration Script allows the user to automate the execution of a few bash commands at the notebook instance creation and/or start. There is a 5-minute hard-limit that will put the instance creation/start in a failed status if reached. There, installing the environment/kernel will have to be done manually once, but we can automate the part where we make Jupyter recognise a conda environment that we’ve installed into the ~/SageMaker/ folder. This can be done using a Lifecycle Configuration Script.
Initialising your environment/kernel
The following part has to be executed once (and every time you want to update your environment/install new packages) in a Sagemaker instance terminal (or in a %%sh cell in your notebook).
envName=myEnv# Creating env in remanent folder (Sagemaker folder in the EBS volume)
conda create --prefix ~/SageMaker/kernels/$envName python=3.7 otherpackages# Creating symlink so that Conda can see the env in a custom path
ln -s ~/SageMaker/kernels/$envName ~/anaconda3/envs/$envName# Exporting conda env to Jupyter Notebook kernel
python -m ipykernel install --user --name $envName --display-name $envName
Notes :
- the
--prefixoption forconda createallows you to define a custom environment installation - your kernel should already appear in your notebook (if not, shutdown it and start it again)
Making your kernel persistent
In “Lifecycle Configurations” in the Sagemaker console, click on “Create lifecycle configuration”. Name your configuration (in our example, we’ll use conda-kernels), and paste the following script into the “Start Notebook” section (in order for it to be executed at each instance start)
sudo -u ec2-user -i <<'EOF'# To be executed at each start-up to ensure replication of conda envs in Conda & in Jupyter
if [ -d "/home/ec2-user/SageMaker/kernels" ]; then
for env in ~/SageMaker/kernels/*; do# Creating symlink so that Conda can see the env in a custom path
ln -s $env ~/anaconda3/envs/$(basename "$env")# Exporting conda env to Jupyter Notebook kernel
python -m ipykernel install --user --name $(basename "$env") --display-name "$(basename "$env")"done
fiEOF
Notes :
- This script will iterate on every subfolders in
~/SageMaker/kernels/, create a symlink for Conda to know where to look for environments, and install in Jupyter these environments as kernels. - Your
~/Sagemaker/kernelsfolder should ONLY contain conda environments folders (you’ll have weird errors otherwise) - If the folder is empty or does not exists, the script will be ignored.
You then have to add this script in the “Lifecycle Configuration” section of your notebook settings. This can be defined at the notebook creation, or by editing it.
Conclusion
I don’t know why AWS Sagemaker team didn’t implement a way to easily install and keep Jupyter kernels (We can’t be the only ones in that case, right?), but it is still an amazing tool and gives a tremendous boost in the set-up of a data science development architecture, without worrying (too much) about technical considerations.
Although it has been intended for a standard use and is delivered with multiple basis configurations, we do have a few opportunities to tweak it to our advantage and specific case…. I guess as much as our sport practise can differ from the standard, mainstream, way of training ! :)
We’ve been able to :
- install a Conda environment in the persistence EBS of a Sagemaker instance
- automate the installation of the Conda environments as Jupyter kernels.
Hope this helps !!
—
👩💻 Looking for a tech job and an opportunity to make sports more accessible ?check out our careers section on https://developers.decathlon.com/careers
💌 Subscribe to our newsletter on http://developers.decathlon.com
