HOW TO GET AI/ML DEVELOPMENT, TRAINING & INFERENCE USING PYTHON & JUPYTER KIT ON Azure
Introduction
Welcome to our guide on how to get started with AI/ML development, training, and inference using Python and Jupyter Kit on Azure! Azure, Microsoft’s cloud computing platform, provides a robust and scalable environment for building and deploying AI/ML models. Python, coupled with the powerful Jupyter Notebook, offers a versatile framework for developing and experimenting with AI/ML algorithms.
Artificial Intelligence (AI) and Machine Learning (ML) are transforming industries across the globe, enabling businesses to extract valuable insights from vast amounts of data. Python has become the language of choice for AI/ML due to its simplicity, readability, and extensive library ecosystem. Libraries such as TensorFlow, PyTorch, scikit-learn, and Keras provide a wide range of tools and models for tackling various AI/ML tasks, from image recognition to natural language processing.
Jupyter Notebook, on the other hand, offers an interactive and collaborative environment where developers and data scientists can write, execute, and visualize code in a single document. With its support for rich media, including images, plots, and equations, Jupyter Notebook becomes an ideal platform for documenting and sharing AI/ML workflows.
In this guide, we will walk you through the process of setting up an AI/ML development environment on Azure using Python and Jupyter Kit. We will cover essential steps such as creating an Azure account, setting up a virtual machine (VM) with the necessary resources, installing Python and Jupyter, and configuring the environment for AI/ML development.
Once the setup is complete, we will explore various aspects of AI/ML, including data preprocessing, model building, training, and evaluation. You will learn how to leverage Azure’s powerful infrastructure to efficiently process large datasets, train complex models, and optimize performance.
Furthermore, we will delve into the realm of model deployment and inference, exploring different options for deploying your trained models on Azure. Whether it’s deploying as a web service, using Azure Machine Learning services, or leveraging Azure’s serverless capabilities, we will guide you through the process of making your AI/ML models accessible and scalable.
By the end of this guide, you will have a solid foundation for AI/ML development on Azure, equipped with the knowledge and tools necessary to build, train, and deploy your own AI/ML models using Python and Jupyter Kit. So, let’s embark on this exciting journey and unlock the power of AI/ML on Azure!
Step by Step Guide to Setup AI/ML DEVELOPMENT, TRAINING & INFERENCE USING PYTHON & JUPYTER KIT On Azure
Step 1
Open AI/ML development, training & inference using Python & Jupyter Kit Listed on Azure Marketplace.
Step 2
Click on the Create Button.
Select a Resource group for your virtual machine.
Select a Region where you want to launch the VM(such as East US).
Optionally change the number of cores and amount of memory. (This defaults to 2 vCPUs and 8 GB ram.)
Select the Authentication type as Password and enter Username as ubuntu and Password of your choice for the VM.
Optionally change the OS disk type.
Optionally change the network name and subnetwork names. Be sure that whichever network you specify has ports 22 (for SSH), 3389 (for RDP) and 80 (for http) are exposed.
Optionally go to the Management, Advanced and Tags tabs for any advance settings you want for the VM.
Click on Review + create and then click on Create when you are done. AI/ML Kit will be start deploying.
Step 3
A summary page displays when the virtual machine is successfully created. Click on Go to resource link to go to the resource page.
Step 4
In the left navigation pane, select Run command , select RunShellScript and enter following command to change the password of the VM.
sudo echo ubuntu:yourpassword | chpasswd
Step 5
Now the password for ubuntu user is set, you can connect to the VM’s desktop environment from any local windows machine using RDP protocol or local linux machine using Remmina.
Step 6
To connect using RDP via Windows Machine, first note the public IP address of the VM from VM details page as highlighted below.
Step 7
Then From you local windows machine, go to “start” menu , in the search box type and select “Remote desktop connection”
Step 8
In the “Remote desktop connection” wizard, copy the public IP address and click connect
Step 9
This will connect you to the VM’s desktop environment. Provide the username (e.g “ubuntu”) and the password set in the above “Reset password” step to authenticate. Click on the OK Button.
Step 10
Now you are connected to the out of box AI/ML environment.
Step 11
You can use the remote desktop you connected in above step for using the VM, however, more convenient and better method is to use the Jupyter/Ipython notebook which comes with the VM .
Step 12
The Notebook is available on the same public IP you used for remote desktop and accessible via any browser. Just open the browser and type the public IP address and you will get below screen for login .
Step 13
The Jupyter Notebook is configured with the ubuntu as an admin user. Login with ubuntu as username and use a strong password & note it down somewhere, since this will be the password for the admin user account from now on.
Note : Make sure you use “http” and not “https” in the URL.
Step 14
This VM comes with the default ubuntu as an admin user. So to access the Web UI and to install additional packages, Login with ubuntu user and the password you set during the first login to the Jypyter Notebook.
Step 15
Open a Terminal in your Jupyter Notebook and enter the below command to install the there package using pip.
sudo -E pip install thereNote: Don’t forget to use sudo in the above command.
Step 16
Conda lets you install new languages (such as new versions of python, node, R, etc) as well as packages in those languages. For lots of scientific software, installing with Conda is often simpler & easier than installing with pip — especially if it links to C / Fortran code.
Install a package using Conda with below command.
sudo -E conda install -c conda-forge seaborn
Step 17
The packages seaborn and there are now available to all users in JupyterHub. If a user already had a python notebook running, they have to restart their notebook’s kernel to make the new libraries available.
Step 18
Alternatively, you can install the packages from jupyter notebook itself. Open a new jupyter notebook by click new dropdown and selecting Python 3 (ipykernel)from top right corner.. Run below pip installation as
!sudo pip install matplotlib
Step 19
Install conda package!
!sudo conda install -c conda-forge gdal
The user environment is a conda environment set up in /opt/tljh/user, with a python3 kernel as the default. It is readable by all users, but writeable only by users who have root access. This makes it possible for JupyterHub admins (who have root access with sudo) to install software in the user environment easily.
Conclusion
In this guide, we have explored the process of setting up an AI/ML development environment on Azure using Python and Jupyter Kit. We started by creating a virtual machine on Azure, configuring the necessary resources, and connecting to the VM using RDP or Remmina. Once connected, we accessed the out-of-the-box AI/ML environment provided by the virtual machine.
One of the highlights of this environment is the Jupyter Notebook, which allows for interactive and collaborative AI/ML development. We accessed the Jupyter Notebook by opening a web browser and entering the public IP address of the VM. From there, we logged in as the admin user (ubuntu) and set a strong password for future access.
Using the Jupyter Notebook, we have a powerful platform for developing, training, and deploying AI/ML models. We can leverage popular Python libraries like TensorFlow, PyTorch, scikit-learn, and Keras to build and experiment with various AI/ML algorithms. The rich media capabilities of Jupyter Notebook enable us to document and share our workflows effectively.
Throughout the guide, we emphasized the scalability and efficiency of Azure’s infrastructure for AI/ML tasks. Azure provides robust resources to handle large datasets, train complex models, and optimize performance. We can take advantage of Azure Machine Learning services, serverless capabilities, and other deployment options to make our trained models accessible and scalable.
By following the step-by-step instructions in this guide, you should now have a solid foundation for AI/ML development on Azure using Python and Jupyter Kit. Remember to continue exploring the vast possibilities of AI/ML and stay updated with the latest advancements in the field. Happy coding and building innovative AI/ML solutions on Azure!
