Lessons learned from working in an A.I. first approach for product development

We have recently opened a machine learning focused R&D center for Booking.com in Tel Aviv, Israel. We did that as we were looking to double down on our long-term investment in machine learning, so we could be able to contribute to the company’s ambition of leveraging even more AI to empower people to experience the world.

Developers, data scientists, designers, copywriters, user researchers are all part of this great group. Oh, and of course product people which are known in Booking.com as product owners.

In this post, I will introduce some of the challenges that any product person might face when trying to follow this kind a vision of an “A.I. first approach” for product development and how critical the role of the product person is in mitigating possible risks and pitfalls.

A quick view of how we work

We have a few fully stacked, autonomous product teams which are capable of taking each task end to end. In order to do so, each team has at least one of the required roles — data scientist, full stack developer or backend + frontend developers, UX designer and a product owner.

Each team is given a broad topic to focus on, while also being given the freedom to autonomously decide which exact features to develop in order to support the team’s objectives and KPIs.

We are working on various types of machine learning based products. For example, we are trying to personalize the booking experience for each of our users, according to their preferences. (for example, a 22 year-old solo traveler shouldn’t necessarily have a luxurious 5-star hotel suggested at the top of their search results).

We have learned a lot in the last year since we opened the development center, but there are a few key takeaways that stand out as the most critical ones:

#1 Start with the user pain points

We approach every problem from the customer’s perspective, regardless of the framework the team is choosing to work with, i.e. Jobs To Be Done, User Journeys, etc.

Instead of saying “let’s analyze text from user reviews and see what we get”, we start with the hypothesis that will guide us through the whole ideation to creation process.

A more correct way to move forward would involve proper user research which eventually could lead to a statement like:

“The customer need is to read relevant reviews in order to take a reassured booking decision as quickly as possible”.

Only then, we start to brainstorm about the possible solutions that could support that customer need. We ask questions such as: “What kind of data do we need to know about each user?”, “How can we define relevant reviews? Should it be picked by us? Or should we try an unsupervised approach to surface up hidden clusters?”.

The critical rule for a product person in this process is to ensure there is a clear and valid user pain point which might be addressed with a machine learning based solution, help the team understand what is the expected output from the model and envision how it will be used to serve the end user.

#2 Keep the principles of writing well-elaborated user stories

Don’t get distracted by the fact that data science is being used when writing user stories. Stories should have the same basic structure which includes the persona, the action, and the desired outcome. For example: “customers will see relevant reviews so they can make a quick and informed booking decision”.

Users usually don’t care if your team invested 6 months to develop a neural network based recommendation engine, or spent two days building a simple ‘Trending Summer Destinations’ table. It’s all about relevancy, and how the solution fits the user needs.

It happens so often that a team/startup gets carried away, trying to chase the newest, most sophisticated models, forgetting the basic principles of Agile development. In the end, we need to remember who are the users that we want to serve and what is the quickest way we can deliver value for them.

Having said that, there are a few differential points when working with user stories and machine learning that every product person must keep a close eye on:

In order to be able to prioritize between stories, you should try and estimate the addressable audience size for each model. What is the maximum impact for each of the ideas? In other words, what is the coverage of the model? Could it be relevant for many users?

As a good product person, you should aim to develop the minimum viable product (MVP) to prove or disprove the hypothesis behind the user story:

Try to narrow down the scope, for example, a relatively broad story like “customers will see relevant reviews so they can make a quick and informed booking decision” could be much more focused if stated as: “customers who care about the quality of the breakfast will see breakfast-related reviews”.

This kind of definition will enable much faster, agile development and testing processes which can be further scaled up to additional categories and use cases.

#3 Taking it live could be heart breaking

One of the biggest challenges for any machine learning based development is making the transition from a machine model which was trained and tested offline and getting it to work in a live production environment, facing real users.

Assume we created a perfectly working model that can predict for each of our users the exact type of reviews they would like to read: Benjamin will get more reviews around ‘breakfast’, while Jenny will be prompt with reviews about the level of the ‘staff’. Although offline metrics will show promising results, so many things could go wrong when we try to test it with real users. Just to mention a few of these:

- How can we ensure that customers trust our recommendations?

- The UX design or copy is not appropriate to the personalized message that we are trying to convey.

- The model takes too long to perform an online calculation which can cause delays in presenting reviews at all.

- We used outdated data to train the model, circumstances have changed and now the predictions are not relevant.

It might take five to six experiment iterations just to optimize all those variables before we will see a real impact on customer behavior.

To sum it up, the most important thing to remember is that similar to adopting a new responsive frontend framework or improving app loading time, machine learning is yet another technological improvement that might help deliver incremental value to your users and product.

Unless you work for a pure innovation and research company (lucky!), you are here to satisfy a certain business need. You must be pragmatic with your approach and try to understand what is the smallest step you can take in the attempt to deliver superior value, only then try to add more and more layers of complexity and granularity with advanced models and sophisticated features.