When to use Machine Learning
Not all problems should be solved with models
Although it may be cool to say you have machine learning algorithms performing some task, they are not suitable for every kind of problem and shouldn’t be your first option in many cases.
Machine learning projects are complex, time consuming, expensive and usually difficult to maintain. Think of a machine learning initiative as a project that requires everything a regular software engineering project needs and more! It needs unit testing and software development, but also pipelines, monitoring, retraining (since the model performance probably will decay throughout time) and much more. Therefore, even though there are some problems where you can use Machine learning, it doesn’t mean you should use it.
There are cases where a simple SQL query, heuristics or IF statement is enough to solve the problem, or is at least a good MVP to validate the problem and the solution. It is better to start simple and then see if you really need to improve the performance by adding more complexity to it or not.
Before jumping into machine learning, ask yourself:
- Is machine learning the best approach?
If you only want insights about trends and patterns, then an analysis is good enough. Machine learning is primarily used for making predictions or inferences.
- Can’t I code a rule-based solution that will already give me good results?
Machine learning will probably be the best approach when you have some human task that depends on too many factors, because it would be difficult to get a good rule-based solution without rules overlap and would be also hard to code and maintain.
- Do I need to scale?
When you can solve this problem some other way, but it won’t scale the way you need or you would have to hire a lot of people to have the job done, then Machine Learning solutions may be the best approach. For instance, you can use humans to identify cats and dogs on images, but not on a large scale and with the speed needed.
- Can I wait for model development?
As mentioned above, it takes time to develop a model good enough for production, it involves Data Wrangling, Exploratory Data Analysis, Model Development, Model Evaluation, Model Productization, this is not a simple task.
- How will the model be evaluated?
Do I have a baseline or a metric to know if the model developed is good enough for production?
- Is the model development feasible?
Do I have historical data with power to explain the problem to be solved?
- Can I implement the model in the product or in the process?
If you can’t use it, you don’t need it or at least you still have some homework to do before model development.
- Will it generate value to the business?
Ideally you have a hipotesis on what the model can do for the business. For instance, the process y takes today x hours to be executed, by implementing the model we can speed up it by 70%, which will reduce the production cost by 50%.
- What is the data type you are dealing with?
When dealing with unstructured data such as videos, images or audios, machine learning will likely be the best approach. It would be very hard to analyze this type of data without using it, since we have a great variety of things that can be different. For instance, on images we can have different saturation, constrat, colors, angles and more variables that can impact on the analysis. Machine learning can generalize well to this kind of data if trained correctly. Furthermore, there are state-of-the-art pre-trained models that can speed up your model building process and also get great results.
Answering the questions above will help to understand if you should use machine learning or try a different approach. Furthermore, it will help you decide if your idea or project is already mature enough to be worked on.
Hope you enjoyed reading this article and that I could convince you of the importance of using the best approach to each problem. If you liked it, follow me for more tips and content on machine learning.