Machine Learning is the hot topic at the moment, though arguably it already has had a profound effect on our daily lives for sometime now. If you look at how Google and the major social networks use algorithmic learning to pick almost everything you see, it’s likely that you’ve already been the subject of ML optimized content today. This has had major implications for the tech and media industry.
Over the past year, we’ve been collaborating with Microsoft as beta testers for their new ML optimization framework called the Decision Service. The DS is a framework for employing machine learning-driven optimization on a small to enterprise scale, and provides the flexibility to be deeply integrated into any application. Many of the technical hurdles that would typically be encountered in developing such a system are handled for you; streaming events, redundancy, iteration, analytics, and expert data science.
So how is this different from A/B testing?
Often the DS gets confused as an A/B testing framework, and while it can be employed in places where you would typically see multivariate or A/B testing, it differs in that can take into account contextual information to predict how future options would perform. It also continually re-evaluates and trains itself offline using a method sci-fi inspiringly called “multi-world testing.” It then uses the results of this improved offline model to update the live model automatically. On the other hand, A/B testing requires manual intervention to both choose the variations and analyze the results to actually utilize the benefits, which can be very time consuming.
To think of it another way, the DS and A/B testing answer two different questions:
A/B testing answers:
Is A is better than {B,C,D}*
categorically
at the time of the testing
where {A,B,C,D} is a static set of choices?
The Decision Service answers:
Is A better than {A,B,C,D,..T}*
for User X
at the time of User X’s decision
where {A,B,C,D,..T} is a dynamic set of choices?
*The number of options in a multivariate test are limited by the amount of traffic necessary to reach statistical significance for conclusive results. With the DS, the limit of options per decision is about 20.
Depending on which question you have, you may choose the DS, or it may make more sense to adopt a more traditional A/B testing approach.
Contextual Features
The really neat thing about this service is that it is actually working with signals from other MS Cognitive Services to inform its decisions automatically. So where A/B testing does not typically have any contextual input, the DS can utilize a suite of cognitive analytics services to help generate features. Entity extraction, sentiment analysis, image recognition, to name a few. Furthermore it’s highly customizable, and you can actually manually feed it features too.
In our experience using the DS, we’ve been able to show significant lift (20+%) in conversion rates with our featured spots on the homepage and with video engagement widgets on our site. To measure the lift we actually used A/B testing with the DS as one variation so we could measure empirically whether using the DS actually performed better. In this use case, A/B testing answered the question “Is the DS working in this particular use case better than the control?”. You can read about it in the case study Microsoft did on us.
We’ve really only begun to scratch the surface on how ML can be used, and it should be worth noting that using ML does not guarantee success. It will only be able to optimize relative to the data and options you feed it, but nonetheless, when employed effectively, it has the potential to yield great benefits in whatever you are optimizing for.
The Decision Service is currently in open preview and so you can actually try this out for free right now.
