AutoML + GPT-3: a Match made in Heaven for Data Science Success

Welcome to my blog about how Automated Machine Learning (AutoML) can be enhanced by integrating it with OpenAI, or more specifically GPT-3. Using Azure Cloud as the platform for discussing and demonstrating examples of the integration, I will delve into ways in which OpenAI can augment AutoML. Let’s dive in!

Azure Automated Machine Learning and OpenAI GPT-3 empowerment

Table of contents

1. Understanding the concepts
2. Unlocking AutoML’s full potential with GPT-3
3. Putting it into practice: A real-world example of the integration

1. Understanding the concepts

First things first, time to clarify some concepts:

• AutoML, or Automated Machine Learning, is an ultimate game-changer in the data science world by automating time-consuming, iterative tasks. It simplifies machine learning model development, allowing organizations to quickly and easily unlock insights from data and freeing up resources for more complex tasks. However, there is more. AutoML democratizes access to machine learning. For instance, making it more accessible to non-data scientists. Teams without extensive machine learning experience are now also able to build advanced models, pulling businesses to the forefront of innovation.
• GPT-3, or Generative Pre-trained Transformer 3, is a language processing model developed by OpenAI. With its 175 billion parameters and backed by a massive amount of training data, it can perform a wide range of tasks, such as generating human-like text, translation, summarization, etc.

Both of these concepts are suitable for a separate blog post, but for the remainder of this blog, I will be focusing on why AutoML and GPT-3 form a perfect partnership.

2. Unlocking AutoML’s full potential with GPT-3

Let’s unleash the opportunities that arise from combining AutoML with GPT-3.

The modeling stage of the data science lifecycle consists of multiple parts. As a short summary, AutoML experiments follow these steps:

1/ Feature Engineering

During the process of transforming your raw data into useful features that can be fed into the machine learning model, a wide range of techniques are applied in AutoML experiments (e.g. normalization, handling missing data, converting text to numeric…).

2/ Model Training

AutoML creates multiple pipelines that experiment with training various algorithms and parameters. Hyperparameter tuning forms a big part of training in which validation data is used to tweak the parameters of a machine learning model to find the optimal set and achieve optimal performance.

3/ Model Evaluation

At the end of the AutoML process, the model is built and you determine its performance. This is usually the point at which you receive the results, but may also want to understand the meaning behind the model. To do this, you could examine the explanations or feature importances provided by the model.

source: Data Science Lifecycle | Microsoft Learn

Now that we understand better the process AutoML goes through, how could GPT-3 bring some extra juice?

Sure, one could easily think that GPT-3’s text generation capabilities could provide value by, let’s say, creating new features based on the data (e.g. feature engineering). But, that is just scratching the surface of what is possible. Let us revisit AutoML’s goal: “Making machine learning more accessible to non-data scientists”. From that angle, GPT-3 could bring immense value by generating human-like explanations for what is happening behind the scenes in your AutoML process. Nowadays, AutoML can handle all the steps and give you back a model with parameters, explanations, and results with just a few clicks. But for non-data scientists, those outputs can still be hard to grasp. So, some low-hanging fruit ideas on how GPT-3 could add value are:

• Explaining auto-generated code.
• Generating explanations for models, e.g. explain feature engineering steps and feature importance such that non-experts can understand the drivers of the model.
• Understanding your model errors in a more human-like way. Humanizing error analysis will help in understanding why a model is making certain errors and therefore also how to improve the model.
• etc.

In the future, this could become more advanced and interactive. Imagine, for instance, allowing your users to engage with AutoML through Natural Language queries, making it possible for them to specify their desired models or parameters in a more intuitive and user-friendly way and giving power to the business people. #EfficiencyAtItsFinest 😉

3. Putting it into practice: A real-world example of the integration

It is time to explore some of these capabilities by running a real example. We will be using Azure Machine Learning and a publicly available Azure Machine Learning — AutoML notebook to predict if a client will subscribe to a term deposit with a bank. After running all the cells, let’s dive into some specific ones. There is already quite some transparency and explainability included in the notebook, but let’s make it even more approachable. I will now do this in a quick and dirty way by copy-pasting the notebook inputs/outputs in ChatGPT (which is built on top of OpenAI’s GPT-3 family). However, if you want to re-iterate this experiment, you could use an Azure OpenAI GPT-3 model liketext-davinci-003and ask it to interpret or explain the outputs to get similar results. Demonstrating this integration is maybe be an idea for a next blog 🤔

In the following notebook snippet, a featurization summary for the best model is shown. For non-experts, the returned information such as transformations and transformation parameters can still be difficult to grasp. Let’s get help in gaining a better understanding of how different features were transformed and how they contributed to the final model:

Create transparency in Feature Engineering

With the preparation steps understood, time has come to set up your AutoML experiment. These settings may also require some clarification:

Generate explanations for AutoML code settings

Once the process is complete, we aim to identify the chosen model and gain insight into its details. This code output looks too difficult, ask for help:

Explain Model Details

If you are not feeling the technicalities, there’s always other things you could ask the model than the parameters.

To truly understand what is driving your final model, dive into the feature importance and ask for a humanized interpretation:

Interpret Feature Importance

There are many more possible examples and opportunities in general, but my goal is to bring across the message that GPT-3 can revolutionize your AutoML game. On top of that, I gave the model very basic questions. The more context is given, the better your results will be.

Thank you!