Pros and Cons of different Prescriptive Analytics technologies
In a previous post, I highlighted that not all Artificial Intelligence (AI) is Machine Learning (ML). There are other technologies, such as Decision Optimization (DO) or Business Rules, that can be used for Prescriptive Analytics. Each of these technologies have different benefits and disadvantages and the understanding of these is really important in order to decide which one to use in practice for a given application.
This posts digs into the different benefits and disadvantages of the different Prescriptive Analytics technologies: Business Rules, Machine Learning and Decision Optimization.
Technologies
In the previous post, I first explained that different technologies were available for the different intentions of AI.
Then I used a simple daily life example to introduce 3 existing types of technologies for prescriptive analytics.
Benefits and disadvantages
Let’s now dig into the pros and cons of each of these technologies.
Business Rules
One of the main challenges of Business Rules systems is, at the same time, one of its main benefits: the need to write the rules. Formally writing all the rules to be applied in a decision process can be seen as long and tedious. However, it is also a way to ensure that these rules are clear, that they are known and applied similarly in all parts of an organization. Part of the powerful functionality of Business Rules platforms is to provide easy ways to create these rules and share them among different stakeholders in human-readable forms.
Lack of Optimality. Most Business Rules platforms execute rules using forward chaining reasoning. By using this type of reasoning, rules execute in a correct order given by priorities. However, the final decisions can be very far from the optimal decision.
Imagine a case containing at least these two ordered rules:
if customer age is less than 20 and no product has been proposed
then propose a $20 productif customer age is less than 40 and no product has been proposed
then propose a $30 product
If a customer is 18 years old, the first rule will be executed before the second one (and prevent the second one from executing) and propose a $20 product. If we suppose that proposing a more expensive product is always considered a better solution, then the solution is not optimal.
The way rules are written, their orders and priorities will impact the final solution. Rules are descriptions of what to do, and not where to go (we will see optimization models describe where to go, i.e. what objectives to optimize, etc). Rules execution engines lack a global view of where to go.
Explainability. One of the main benefits of Business Rules is the explainability of decisions using the sequence of executed rules.
Imagine a very simple case with these two rules:
if customer age is less than 25
then refuse loanif customer age is less than 70 and loan duration is more than 20
then refuse loan
If a customer is less than 25, his loan will be rejected. One can explain this by simply exposing the first rule. If the outcomes of the rules are biased, then this is due to a bias that has been explicitely written as a rule.
Machine Learning
The main benefit of ML is that, in theory, no formulation of the problem is required (neither the what to do nor the where to go!). This is a data driven technique, and in theory you only need lots of data and some computation to train a model.
I said in theory as everyone who has played a bit more seriously with ML has understood that in practice, lots of expertise is required to select and organize the right features, algorithms, and parameters. What should be a simple use of some well known ML algorithm can end being as complex as formulating a DO model. The difficulty of Kaggle competitions demonstrates that.
On the other hand, the disadvantages are quite well known.
ML requires lots of data. This is not a problem with digital industry where in general the problem is more that too much data is available than the contrary. But this need of data becomes an issue when one wants to solve new problems where data is not yet available. For example, when a new situation arises (e.g. COVID-19 world), where everything is organized differently, no data is available to train models. Models trained on pre-COVID-19 data will not make accurate predictions for any post-Covid-19 situation. As mentioned in the previous post, my older kid, who learned to pack luggage from watching his mother year after year, going on holiday on summer, may have troubles with the way he packs luggage to go skiing with his friends. This is a completely new situation, and the “summer model” of packing should not be applied. Most times, the non-applicability of a model to a situation is not so obvious. Read more about this type of consequences in my post on Why Intelligences fail?
ML platforms answer concerns about correctness by supporting the monitoring of ML models over time, and providing ModelOps functionality to accelerate feedback loops. Many models become obsolete quickly after they are trained and require maintenance with new data and updated training. Some platforms allow to train models continuously.
Current concerns with ML includes, bias and explainability.
ML models may also have some bias and might not always be fair. ML models are created using historical data, so they are trained to reproduce some behavior, which can be good or bad. Any bias present in the training data will be reproduced in the future. As illustrated in my luggage example, my older kid will pack differently whether he learned from me or from his mother, leading to more or less success when going on holiday. Many examples of unfair models due to biased training data are well-known. A famous one is where an AI based recruiting tool favored men as it had been trained using older human decisions favoring men. Correcting for fairness and bias makes model training even more complex and even less automatic.
The second main concern of ML models is explainability. After AI prescribes decisions, human operators want to know why and get some explanation of why the proposed answer is YES or NO. With ML, in my opinion, it is difficult to consider any current work on explanability to be really satisfactory. We can explain the prescription by highlighting some features which are common to all the training records leading to this prescription.
Decision Optimization
Let’s now cover Decision Optimization. Decision Optimization is a set of technologies such as Mathematical Programming and Constraint Programming and is more or less corresponding to what was known as Operations Research.
The main challenge with Decision Optimization is the need of an optimization model formulation. For that, it is necessary to have a detailed understanding of the problem: data and decisions, constraints and objective. This is in general obtained after an incremental process between an Optimization expert and a Business Analyst. The Optimization expert then must construct the problem using mathematic formulations.
Decision Optimization, describes where to go, the objectives to be maximized or minimized, and the constraints preventing some combinations of decisions to be valid. Nothing has to be described on how to get there. Mathematical Programming will then use advanced algorithms to explore the huge set of solutions efficiently and return the best ones.
Here is an example of model on a simple and didactical example taken from Alex Fleischer.
Formulating the model using mathematical expressions requires some specific skills and expertise, which not every data scientist has. This bottleneck of optimization modeling skills is one likely reason that Decision Optimization is not used more widely in industry. Several initiatives have tried to break this bottleneck. One such initiative has been to provide reusable optimization models as assets to provide a starting point for data scientists without these skills. More recently, another initiative has been to make modeling easier using a natural language modeling assistant.
Just as with Business Rules, one might consider the need to precisely formulate a model to be a benefit. In this case, this is not a common formulation of how to make decisions, but a common understanding of what are the objectives and the constraints. Even if this approach is not as human readable as the Business Rules approach, this might change in the future with natural language modeling assistants!
By explicitly stating your target and what isn’t allowed, it is easy to update a model in DO when the objectives or constraints change. The world changes constantly, and it is much easier to modify a Decision Optimization model formulation than to retrain a ML model, as we discussed in our example on creating COVID-19 models.
Now that we’ve discussed the difficulties of formulating and managing the optimization model, we can delve into the benefits of Decision Optimization!
Decision Optimization can ensure an optimal solution. At any time during the execution of Mathematical Optimization algorithms, it is possible to obtain a gap between the current best known solution and the best possible achievable one. In some cases, the business user of the optimization-based application will want want a quick solution that is good enough and information on the gap between good and best. In other cases, even small improvements of the objective value will be significant for the business, and this gap will be more important
Decision Optimization prescriptions can be rationally explained. Sometimes, a given model has no feasible solutions for a given set of input data. Optimization engines will be able to return the minimal set of constraints which cannot be fulfilled altogether (known as conflicts), or a minimal set of changes to be done to turn the problem into a feasible one (known as relaxation). In one scenario, when an optimal solution is found, it is possible to get the conflicts which prevent a better valid solution to be obtained. For other examples and scenarios, see this post on explaining Decision Optimization prescriptions.
Conclusions
After understanding the benefits and disadvantages of each technology, it is much easier to choose which one to use depending on the user’s needs. These techniques should not be seen solely as competing approaches, but as complimentary ones that can often be combined to get even better results. As a consequence, AI platforms should not only focus on ML but integrate all these technologies and enable their combinations. See, for example, how ML and DO can be combined to better solve some problems in Cloud Pak for Data.
For more stories about AI and DO, follow me on Medium, Twitter or LinkedIn.
