The two steps people forget to take when applying Machine Learning
For some applications, humans design AI systems without any ability to learn. Experts can design such systems and prove their correctness; these applications of AI are the best solution for cost effectiveness and performance (for example, when it comes to ensuring humans’ safety). In contrast, guaranteeing correctness for decision-policies generated by machine learning is a major challenge. We’ll take a deeper look into into why.
How to approach Machine Learning (ML)
As organizations hear more stories about how Artificial Intelligence (AI) and Machine Learning (ML) have transformed business practices, data science practitioners are being asked to “apply machine learning” with only vague or high-level business goals. There is a large gap between implementing AI and creating business value from data, and many disappointments in data science are the result of a failure to bridge that gap. This article attempts to show both AI practitioners and business managers how to bridge that gap.
The order in which you should approach AI is different from what most might expect, and should follow these steps:
1) Target envisioned solution
2) Design around your data
3) Define models
Most people forget the first two — seems counter-productive to miss them, doesn’t it? After you’ve gone through a detailed analysis of the requirements, as well as application and business constraints, you’ve arrived at a pretty detailed problem description. With that, you can approach your ML experts.
1. Target envisioned solution
Your desired results should drive machine learning — not vice-versa.
ML can’t give you the answers if you don’t ask the right questions. We need to be able to define, with specificity, what we are trying to accomplish. Defining and assessing the business case for machine learning involves many considerations (which we’ll dive into in our next article), but for now, here are the major criteria to define your desired results:
Defined questions
The targeted results requirements should be detailed enough to work against. You should ask yourself questions like:
- What should the ML model output? Is it a prediction of something you can observe, or is it a suggestion on how to act in a certain situation? For example, let’s say you needed a component for a bike-riding robot to prevent it falling on a slippery road. Do you want the model to predict if the bike is expected to slip for a given environment condition and inclination? Or would you want your model to predict the fastest way to get around the curve without slipping?
- How quickly do you need the model to compute its output? What hardware and software environment could be made available as a platform in order to execute the ML model?
- How transferable should your model be? Going back to the bike robot example: is the model only supposed to work for one specific bike or for slightly different bikes from various manufacturers?
Measurable questions
Assume that somebody gives you a pre-trained model. We need to think about how you would go about evaluating its performance. For a given input, you should be able generate the desired target value your model should output, and do so for a large number of inputs, predicting how costly it is to generate the target outputs.
You should also consider how to signal to a machine that the outcome is good. This can be as easy as using the true or false mechanism (e.g. whether something was successful or not), or it can get much more complicated quickly, depending on the application.
Useful questions
Is the problem worth solving from a business perspective?
- Will ML provide benefits that the customers are willing to pay for?
- Or could ML provide a reduction in manufacturing cost?
- Maybe, using ML will simply provide an extra marketing advantage? (Think of Apple’s Siri)
Application of ML will require a resource investment in ML engineering expertise, as well as infrastructure setup and maintenance. Make sure you’re considering whether the benefits of using ML are outweighed by the costs of data collection and infrastructure.
Training vs recall
When you train a model, you aim to improve its performance. During recall, you simply evaluate a pre-trained model to generate a prediction, but the model does not learn anything new.
It’s important to keep in mind that the computational requirements for recall are significantly less than those for training. To train a model, you need access to all of your training data, and many hours of training time. Depending on your ML task, a better gaming computer might provide enough computing power to train your model; in some cases a compute cluster might be needed. Once the model is trained, you can usually remove most of the training data and save the model as a small file. Many ML models require only very little resources, such as embedded devices or cell phones, to recall predictions. This already suggests separating training and recall.
Another advantage is the required complexity and agility of software infrastructure, which is much higher for training than for recall. This separation offers the business model of training as a service (also known as the subscription model).
2. Designing around data
AI can’t see around corners, so models require data. A lot of data. Major considerations include designing in data, data defensibility and collection, data structuring and cleaning, and privacy or legal issues.
Criteria for good data are as follows:
Quantifiable: Data needs quantifiable results for it to be useful training — that can be a simple yes or no, or may include a more detailed measure.
Identically distributed: All of your data points should have been sampled from the same probability distribution. Usually, that means making sure the data set you’re working with was consistently measured and recorded. As you gather more data, you’ll need to ensure consistent measuring.
Diversity: You should have a large enough sample size to cover your entire space of potential outcomes. This should include enough coverage of rare events that your model can tell the difference between a random error and a rare (but important) event. For example, if you’re trying to build an AI to predict the next stock market crash, using data from only years 2009 and 2017 would be too restrictive.
Captured external drivers: External drivers are the features that influence your outcomes, and those need to be captured in order to generate good predictions. As an example, it would be difficult to predict the price of a car if all you knew was the width of the steering wheel and the size of the gas tank, but much easier if you knew the model and mileage.
3. Models for success
This is where organizations tend to begin when thinking about AI. Models are the deeply technological part of ML, and ML scientists typically amass expertise in this area. However, successful implementations of ML require integrating deep strategic and design thinking from the beginning of exploration.
In short, ML techniques (models) can’t be designed until the desired outcomes and data constraints are known — only then can those problems be broken down into machine-learnable parts. Here are the criteria for good models:
Feasible and usable: Is it possible to solve the problem with this model, given the data, technical constraints, and design constraints? It may be hard to solve a priori without testing — an upcoming article will talk about strategies to speed up AI prototyping.
Scalable: This is the computational efficiency of your model. Are these techniques useable at scale, given the increase in computational and model-training cost when we increase data size? At scale, the infrastructure costs may be non-negligible.
Blockers to AI implementation are primarily problems that business and product strategy can remove. That’s not to say the technical parts are easy, but technical constraints are generally a known quantity; we know what current AI capabilities are, and what kind of problems it can be useful for.
Some uncertainties remain around the future of AI and ML. In general, it is hardly possible, even for an experienced data scientist, to predict how much data will be needed to train a model. Furthermore, it’s difficult to project how much tuning of the model’s training process might be needed and how sensitive the training process might be to outliers (anomalies) in the data.
Finally, there is a high uncertainty of the required time to move a training process from an early prototype to a mature product. In this context, it is important to keep in mind that most machine learning algorithms are stochastic; the models vary when the training is repeatedly executed on the same dataset, and, even more so, their performance varies when trained on different data sets. Generally, new data is acquired over time, and for a mature application, an automated model training is desired. This often entails automating either some or all of the process steps that are executed by the human machine learning expert during the initial prototype phase: data cleaning, tuning of the training process to the specific data set, and automated model selection.
So it is wise to treat each machine learning project, to some extent, as a research project with the associated higher risks and potentials.
Machine learning isn’t magic — it’s an incredibly powerful tool, but only if applied intelligently.
Written by Nick Chow and Alex Hentschel
Edited by Yasmine Nadery
