5 best practices for implementing an ML-based recommendation engine to guide business operations

With the increased role Machine Learning plays in businesses, many companies try to implement recommendation systems to guide their business operations. With the focus typically on the quality of the prediction models, the business aspect of implementing such systems is often ignored, leading to poor user experience and, in some cases, project failure.

Below are best practices to follow when implementing such a recommendation engine. These can be used to increase the likelihood of success and expedite the implementation process.

Explain your recommendations

Seems trivial, right? And it might be a little trickier than it seems.

Many machine learning models and other recommendation generating algorithms are like black boxes: after they are trained, they produce results, but the decision rules are not easily interpreted. Being able to provide a rationale for each recommendation requires significantly more work.

As long as the business user agrees with your recommendation, they will not challenge its logic. However, if they disagree with it, they will want to understand it better, and the lack of explanation will frustrate them. Stating that the recommendation is based on historic data, simply will not cut it. You need to be able to clearly state a few rules that were used to generate the recommendation. These rules should be simple so that the user can understand them, but not too simple, since trivial rules generate trivial recommendations.

Avoid false positives to the extent possible

The cost of bad recommendations outweighs the benefit of good ones. Or in statistical language — precision is more important than recall.

The perception of your recommendation engine, when dealing with experts, is fragile. They know their business and, typically, find the idea of a recommendation engine insulting. They will initially take your recommendations with a grain of salt and gladly dismiss the entire system once they encounter bad recommendations. You need to be aware of this low tolerance for error, and make sure you set sanity checks to avoid bad recommendations. This is true even at the cost of sometimes missing out on a good recommendation.

Invest in internal marketing and create a feedback mechanism

We all like to think that the quality of our product will speak for itself, and that the users will quickly realize the benefit of the new system and adopt it. In reality, changing behavior requires effort. This is especially true when dealing with professionals, who have been doing the same things for years and have developed their own methodologies and habits.

Make sure you celebrate success, highlight the benefits, and regularly communicate the improvements. Also, use this internal communication to make sure people understand how the system works (remember — explain your recommendations) and how their feedback is helping to shape the system.

Creating a feedback loop allows you to identify bugs early on by receiving feedback directly from the frontline. It also has the benefit of correctly tunneling and positioning users’ concerns and pain points. Without a feedback loop, these concerns might surface in various channels with limited context and can create a negative buzz, which can put the success of implementation at risk.

Of course, feedback needs to be a two-way street. You need to communicate how feedback is being used and how the system has changed based on this feedback.

Define and track business (not statistical) success

You might have developed great models or algorithms that have very high accuracy. However, there is typically a long process between the model’s output and the actual business outcome. What you, and your business, really care about is what impact these recommendations had on the business performance, and for this reason, this is the only way you should measure yourself. Of course, there should be some process metrics to identify the drivers of business impact, but those should play a secondary role.

Incentivize business performance, not recommendation usage

If you have control over the incentive structure, it is very tempting to simply pay users (or provide some other incentive) to use your recommendations. This may seem like a sure way to get users to use these recommendations and realize the benefits of them. However, this can cause frustration when the recommendation conflicts with the user’s best judgment. In these cases, he or she will feel like you’re “pushing” them to do something that they think is wrong.

If possible, create an incentive system that is somewhat based on business success (ideally the same metric you use to track the success of your recommendations), and position your system as a tool to help the users to maximize their incentive. This system bears some risk, as your recommendations might be completely ignored, but if done well (remember — internal marketing) then all incentives are aligned, and the likelihood of success increases.