Machine Learning in the business context — how to find a viable project
While there are many posts and articles about what machine learning is, how to do a project, and technical courses, there aren’t that many that focus on the business aspect of it.
Disclaimer: Most of the content is based on the teachings in Analytics for Strategic Management at BI Norwegian Business School. A highly recommended course if you want to learn more on how to think about advanced analytics.
This article is targeted for business managers who either oversee ML projects or are in the process of starting one. It will give tips on how to identify viable project candidates and introduce frameworks for evaluating them.
The Cross Industry Standard Process for Data Mining (CRISP DM) is a methodology used within machine learning and data science, and that emphasizes iteration — hence all the arrows.
Some arrows represent process steps while others represent learning. And it doesn’t really state what happens after the deployment step, or where the business decisions are made, and is difficult to use strategically in an organization.
It is however the most common approach used to tackle data mining and modeling problems, and will serve as a guide throughout the article.
Business and Data Understanding
The first step in any predictive analytics project is to understand what you are trying to solve.
Will you be using the data at your disposal to predict something specific (a target — aka supervised learning), or is it to find some “weirdness”; an anomaly or some natural clusters (unsupervised). If it’s about the latter what will you use the gained information to do?
Most business problems are of the supervised kind: Based on all the data I have, how much will customer X spend? Or which advertisement should I recommend to you based on your browsing history?
Assessing if your business problem is solvable is highly coupled with the data at your disposal. Understanding the strengths and limitations of the data is important because rarely is there an exact match with the problem.
The stored data is only a condensed version of the business’ reality. There is so much context that’s impossible to store in a database, and therefore not all problems are suited for predictive analytics. Remember that you are trying to create a model, i.e. a mathematical representation of the “business reality”. It’s then best to find problems where there exists data to answer them.
Big problems like “We will use ML to improve our environmental impact” make for ill-fated projects.
Does your business have existing and historic data describing its environmental impact, and what exactly is the target here? The problem statement is of low precision and needs to be refined; some decomposed part of the problem might be a good candidate for machine learning. If it’s a bus company, predicting when a bus should be sent to service is more precise, and something you might have data to build. The less resources used on service rather than repairs is then a contributing factor to improving the business’ environmental impact.
Ideally, the business problem should be so well defined that you can deliver your technical team a model input file, describing the target and all contributing data to that target, along with examples of them.
Data preparation and Modeling
The data preparation and modeling steps are most probably where the most of the project’s time will be spent. It is essential to understand that these steps resemble closer to research and development than to engineering.
It’s not uncommon to use 70–80% of the projects time in these steps, doing multiple iterations as more and more knowledge about the data is obtained.
The key managerial points in these steps is to be attentive to which preparation and modeling techniques are used, as the project stakeholders might have other requirements for its application than the technical team.
- Is the data used drawn from a population similar to which the model will be applied? How does this impact the business statement?
- Which type of modeling technique is being used? Does it meet other requirements of the project: Generalization performance, comprehensibility, speed of the application, amount of data required?
- Should various models be tested and compared?
Evaluation
There are many ways a model’s performance can be evaluated; often the metric accuracy is used when describing how good a model is.
Unfortunately, this is usually too simplistic for models used in real business problems.
Take as an example a Telco firm who suffers from 3% annual customer churn, which wants to develop a model to predict customer retention. This problem is suffering from unbalanced data, and a simple and naive model which predicts that everyone remains (not-churn) will have a 97% accuracy.
Impressive, but not so useful.
Instead they could have incorporated their business domain knowledge and used the expected value (EV) framework. It is a practical method to determine the quality of the decisions made by models we wish to evaluate. It lets the business attach quantified costs and benefits to a prediction outcome, enabling us to evaluate the profitability of one model against another. Further, it makes the business think more thoroughly about the structure of the problem and the data elements that comprise it.
The EV framework should be used on both the newly built model, but also on the status quo model, i.e. the one that describes what is happeing now.
Even if your business doesn’t use a ML model for a given problem, there is a model in effect — albeit a very naive one.
From the Telco example above, the naive model could be that they belive the 3% of customers who churn is spread out evenly, i.e. the customers do not have anything in common. The action would then be to randomly incentivise customers to remain. And it is this model and its costs the ML model would need to be compared and evaluated against.
Deployment
When a model has passed all necessary evaluations and is deployed in your business application and makes predictions, the important thing now is to use the action for something.
When starting the project you should already map out what you will do with a prediction, how it will alter the business workflow, and get commitment from the stakeholders that it will be used.
Else if you get to the situation where the predictions are scrapped and the model put on the shelf, you could have saved yourself the trouble of doing it all together. Your data scientists and engineers who have worked hard on this will lose motivation, and your business will have lost time and money which could have been spent elsewhere.
In summary
When findig a problem suitible for a ML project the most important thing is to frame the problem on a row-level. Descibe the data needed to solve for the target, ideally providing examples of the data points needed. Then it’s easier to discuss with the technical team the viability, what data is in place and if something needs to be collected.
After a model has been made it has to be evaluated both technically and on the added value it brings the business. And it’s you — the management — who must distinguish what’s technically good from what’s operationally useful.
Problem specification guidelines
- Precision — what is the question?
- Does it stand on its own?
- Decompose until one question remains
- Can the problem be operationalized? - Value — Does this problem matter?
- What’s the baseline?
- What’s the result of the status quo? - Viability — If you find anything, can you act on it?
- Are there potential solutions?
- Can you execute or argue for implementation?
Reflection questions
- Target — what is operationalized and modeled?
- Input data — what data is used?
- Business action — what’s done with the target?
- Business objective — how can the action help the business?
- Workflow change — How do the organization use the prediction; what changes?
- Potential risks — Are there risks involved with collecting data, modelling the target, or using the prediction
- Ethical / Trust — How is the business affected from poor predictions - Results — what actually happens once the model is deployed?
Thanks for Reading!
Interested in seeing how a real problem has been identified, modeled and evaluated? Look no further than here Vegvarsel.no — a ML approach to avoid mountain road closures.
