Why A/B testing will lose against advanced AI, all the time.
A short interview how AI will change the future of A/B testing.
How would you describe your business idea to a potential investor (who is not an expert in your particular profession)?
free machines provides Artificial Intelligence solutions to solve key decision problems along the customer journey.
What problem do you want to solve, what is your goal?
Higher personalisation leads to better outcomes. For firms, today there are no tools that personalise in a fast and optimal way and can be used with extremely low friction. We try to build this product with free machines.
Right now, in a lean experiment, we apply the technology as a zero-configuration solution optimising user interfaces like websites. You can check that out at hypermaus.free-machines.com.
How did you come up with your idea/concept?
We all came with a strong research background in Machine Learning and Deep Neural Networks. The team joined on the believe that Artificial Intelligence can have a very positive impact on humanity, and we want to sit in the front row when it happens.
When working at several A/B testing driven optimisation problems in different companies, we realised that this approach is fundamentally flawed:
- A/B testing cannot cope with dynamics in the environment. Often, when you’re done with testing and implementing the changes, the findings do not hold anymore.
- A/B testing takes huge amounts of manual work and thus financial resources.
- A/B testing is somewhat detached from the regular development pipeline. Findings have to be communicated forth and back between version-control driven engineering departments and performance marketers that use GUIs to come up with variants.
Based on these observations and our research background in AI, we realised that modern methods can solve these problems entirely. This is possible as they operate in a closed-loop way.
The figure above shows one of the fundamental differences between the two optimisation approaches: While A/B testing tries out test variants of constant proportion, our solution can exploit growing confidence about the best variant from the very start.
What is your business model?
We want to provide the services as a SaaS. In the HyperMaus experiment, we are targeting a lower-mid range price point, so we hope for a large amount of customers at a price range between 100,- and 1000,- $. Generally, we encrypt data before moving them to our AI models. That allows us to do transfer learning between the customers, expecting strong data network effects, and building artificial customers.
Why did you decide to work with XPRENEURS?
Since we started free machines, we always tried to get as much feedback as possible. One of the most valuable feedback meeting was with Dr. Martin Heibel, head of XPRENEURS. When approached us to join XPRENEURS as the first team, we were (1) quite honoured (2) and decided very quick to join, as we expect more intensive feedback from guidance of XPRENEURS to be extremely helpful.
Do you think AI will replace decision making on higher levels of companies in the long run?
It depends on the nature of the decision. We think today, AI can be extremely helpful if data size, dimensionality and frequency become too complex for humans to process. If you wait long enough it most probably will, and we shall do what we can to make that change a good one for everybody.
What differentiates your approach from the competition?
We have a proprietary AI core with the following properties:
- It operates in a closed loop. So you do not get analyses and recommendations what to do, which you then need to implement as an “actor” deeply in your business process. In that context, we are able to optimise problems in a full-scale and end-to-end way, eliminating most manual work and synchronising issues. (That has the downside of processes not ready for closed loop being out of our focus for the time being.)
- It’s capable to converge really fast. In A/B testing, you test for a certain amount of time, then analyse, and then implement the findings.
Do you see any additional applications for your technology?
That’s a tricky question. There are so many applications for our technology that our main challenge right now is to pick the right use case domain and the right focus in that domain. Basically, if you understand customer journeys broadly as websites, apps, or even contract properties (one could say that contracts are also a form of user experience), we could optimise all that.
