From Jupyter Notebook to Azure Web App in 5 Easy Steps
You have been exploring data in Jupyter Notebook and would like to share the insights. Some of your plots have filters, interactivity. You could send the plots as separate .html files. However, you would lose the ability to tell the story. Also, if the plots need to be updated weekly, you would clutter the client with emails or dumps of .html files in the shared storage. That is a scenario for which makes sense to create a dashboard using PowerBI, Tableau or similar tool. To build a dashboard usually involves additional effort and time. However, this can be minimised by the help of Voilà library . In this post I will show you how to convert your Jupyter Notebook to a dashboard and share it as a standalone app with minimal effort.
For this example, let’s imagine that you have collected a sample of data from multiple sources which could be useful in solving your problem. Usually subject matter experts (SMEs) have not seen all these data sources brought together into one place. Thus, it is a good idea to visualise this data and share with the SMEs to validate it. Also, it would be useful to have an easy way to gather feedback or labels for dataset. All these steps will be covered in this tutorial which at the end will produce the web app as seen below.
Let’s begin.
Roadmap and resources needed
Assumptions
- You work with Python and Jupyter Notebooks
- You have Azure Subscription
Step 1 - Create Jupyter Notebook
Let’s use Azure Machine Learning for development. The same process could be done on a local computer. However, if you want to move your work to the cloud, it is one of the best online development environments for data science. Machine Learning resource can be created from Azure Portal.
A few things to bring your attention to in the overview page of your new Azure Machine Learning resource:
- Each Azure Machine Learning resource comes with a designated Azure Storage space. The easiest way to reach it is via the link next to the Storage title on the overview page (see screenshot below)
- config.json file will have all the details you need to log into this workspace.
Storage
Most of the data related projects start by acquiring data or a sample of it. For this example, let’s use a dataset about air pollution in Seoul from Kaggle.com. Let’s go to your Azure Blob storage attached to this Machine Learning resource and upload the example dataset. If you click on the link next to storage in the overview section in Machine Learning space, you should be able to navigate to the Storage Explorer. It allows you to upload data/ files using GUI (graphical user interface) as you can see below.
You will mount this Blob storage to your web app. This way, all the data stored in here will be available to your web app. More about this is in Step 4.
Compute
Now you need a resource to run Python code to analyse the dataset. For this, let’s use Azure Machine Learning Compute Instance. Go to the Compute subsection under Assets section and create a new compute instance.
After the resource is up and running, you will be presented with 4 options/ links to interact with it:
- JupyterLab
- Jupyter
- RStudio
- SSH
JupyterLab is my personal favourite interface.
Compute resource is a Ubuntu virtual machine with tools mentioned previously preinstalled. If you open a new terminal and type hostnamectl, you will see which version of Ubuntu is installed. Also, if you type pip list or conda list, you will see which python libraries are already installed. Pip or conda install any missing python packages using terminal.
- If you need any additional software like ODBC connector, or blob fuse in your favourite search engine, look for how to install X on Ubuntu. As mentioned earlier, for additional Python libraries it is usually pip or conda install followed by the library/ package name.
- Also, git comes pre installed. Thus, if you need to clone your code from your code repository, in the terminal type git clone followed by the http to your repository. If you want to follow along, then type in git clone https://github.com/mrjurbonas/medium.git to get the code I used to create the example web app.
Jupyter Notebooks
This example has two Notebooks (they can be seen here). One for processing data and one for visualising the results. There are couple advantages for doing this.
- keep the Notebooks short and concise.
- if data is updated infrequently, you could rerun the Notebook manually when required. If data is updated often, you could incorporate data processing Notebook into ML Pipeline, which would run based on time schedule or other triggers.
- visualisation/ dashboard Notebook will be converted to web app. Thus, you want to keep only the required code to improve the performance of the app.
- similarly, if possible, you want to pre-process your data and store it in Blob storage. This way, when you open your web app, you do not need to wait for data to be processed. This will improve user experience.
Let’s run process_data.ipynb notebook. This will load data from Blob storage, process it, save a local copy as a Pickle file and upload it to your Azure Blob storage. When you run dashboard.ipynb, processed data that you saved locally will be used. However, once dashboard.ipynb notebook is converted to web app, it will use the Pickle file that was uploaded to your Blob storage.
Preserving data types (like datetime) and minimising the size of the dataset are few of the advantages of saving data as a Pickle file.
👉 Note: If you get an error when saving Pandas Dataframe as a Pickle file, update pandas version by typing pip install --upgrade pandas in the terminal.
The last cell in the dashboard Notebook is embedded Microsoft Form. Once you have created Microsoft Form with desired questions, copy the embedding html from share section (as shown in the screenshot below) and paste it in your Notebook. This way the user will have an easy way to provide feedback or ability for you to collect labels for your data.
dashboard.ipynb notebook will be converted to a stand alone web app using Voilà library. You do not need to do any modifications to your notebook. Voilà library will run the code, hide it and expose only the markdown and output cells to the user.
👉 Tip: You could use Jupyter-flex library to get more control on location and positioning of your plots when they are displayed using Voilà.
Step 2 - Create Dockerfile
Dockerfile is a set of instructions on how to build the environment and run your app. You can see Dockerfile which will be used for your web app below.
Python dependencies are installed using requirements.txt file. This can be obtained by typing pip freeze > requirements.txt in the terminal. However, to keep Docker image small, I would recommend to delete Python packages which are not necessary to run your Jupyter Notebook that is converted to the app.
👉 Note: You need to add voila==xx in the requirements list. Also, I like using nb-black library to automatically format my Python code in Jupyter Notebooks and JupyterLab. That is why I have included nb-black in the requirements.
Step 3 - Build, tag and push docker image
In this step you will package your Jupyter Notebook as a Docker image and upload it to online repository.
You are going to use Azure Container Registry to store your Docker images as it integrates seamlessly into the rest of Azure stack. Go to the Azure Portal, choose Create a recourse, then type in Container Registry and click create. Do not forget to enable Admin user (see screenshot below) as this will be required for the Web App to access the stored Docker images.
Now you can create and upload Docker image to your Azure Container Registry. Open a terminal in JupyterLab and navigate to your working directory/ folder. Then type in the commands below.
Step 4 - Create Azure Web App
It is relatively easy to create a web app using Azure Web App resource. Go to the Azure Portal, choose Create a recourse, then type in Web App and click create. You need to chose to create your app from Docker Container and run it on Linux (see screenshot below).
In the next (Docker) section of setting up the web app, you need to choose Azure Container Registry for Image Source and select your Docker image that you pushed to your Azure Container Registry.
Configure Web App
Once the app is up and running you need to go to the Configuration section of your app and finish setting it up. First, you need to add New Application Setting. These are environmental variables which are passed to the Docker image. Add WEBSITE_PORT with a value of 8866. This will expose your dashboard which will be running using Voilà library. Also, these variables are accessible to your code inside the Docker container as environmental variables.
Next step is to mount/ attach Blob storage from your Azure Machine Learning resource. You can do this by going to Configuration, Path mappings and adding a new Azure Storage Mount. You should be able to find the Blob storage from Machine Learning resource using drop down menus (see screenshot below).
The code inside the container will be able to access the files on your Blob storage by going to /ml_blob_storage folder or any path that you choose as Mount path. To make sure that dashboard.ipynb notebook can access files both when running on Azure Machine Learning Services and as Web App include the code below (cell 2 from dashboard.ipynb notebook).
Now, if you go to the URL of your app, you should see the Jupyter Notebook converted to dashboard as shown at the beginning of this article. However, anyone can access your app using that URL. Let’s use Azure Active Directory to control (give permissions) the access to your web app. This will be explained in the final step.
Step 5 - Register App
For the final step, you need to configure authentication to your web app using Azure Active Directory (AAD). Similar to the previous steps, go to the Azure Portal, choose Create a recourse, then type in Azure Active Directory and click create.
The new AAD will be added to your existing directories. You can see the list of your all subscriptions and directories as shown in the screenshot below.
App registration
Navigate to your Azure Active Directory, go to App registrations and register your new app. Do not forget to add azurewebsites.net/.auth/login/aad/callback to the end of your app’s URL.
We need to copy and save Application ID and Directory ID from this registration which will be needed to complete future steps.
After this, go to Authentication section and enable ID tokens as shown in the screenshot below. Remember to always save your changes.
Enabling Authentication via Azure Active Directory
We need to go back to your app to Authentication section under Settings and turn on App Service Authentication, choose Log in with Azure Active Directory and then choose Azure Active Directory from Authentication Providers (see screenshot below).
After this, you will be presented with the window to set Azure Active Directory Authentication. Choose Advanced option and put in Application ID that you saved previously to Client ID field and add Directory ID to the end of https://login.microsoft.com/ in Issuer Url field (see screenshot below).
Adding users to your Azure Active Directory
The final steps are to add users to your Azure Active Directory and then assign them to your app. There are two options when adding new users:
- Create User
- Invite User (more convenient and easier option)
Before you start adding users to your app, please remember to go to Properties section of your app in Enterprise Application and turn on User assignment required (as shown in the screenshot below).
Voilà, now you have a fully functioning web app with controlled access.
Thank you for reading. There are many ways to create web apps on Azure platform. This is only one of the possibilities how different Azure resources can be combined together to make a web app. I hope this inspires you to share the insights from Jupyter Notebooks using Azure Web Apps.
