From ideation to production in 3 steps -
A case study on how we built our first Content Machine Learning platform at Booking.com
In the past year, my team has built an objectively great platform, from the very early phase of ideation to production at scale. This platform is widely used across teams within the Booking.com organization, with more than 30 use cases in production or under development.
In this post, I’ll share our 3-step recipe for building the Content Intelligence Platform and supporting a variety of use cases.
In order to better understand the steps we took, let’s take a look at what this platform can do.
Content Intelligence Platform (CIP)
CIP is a centralized service hosting ML-driven enriched content, both for images and text. Given an image or text, CIP provides ML-driven information, such as:
- Image ML Tags — by using an image classification model, the platform will output the different labels within an image and their confidence scores.
- Image Quality Score — what is the technical quality of an image? What is the attractiveness of an image?
- Focal Point (x,y) — what is the main point of interest within the image?
- Review Sentiment — what is the sentiment of a guest review?
So what were our 3 main steps for building this content platform?
Step 1: Primary Product Package
Our team worked on Computer Vision since 2018 and was familiar with the challenges around it, in general, and specifically within Booking.com. Hence, we had a concept in mind of building a content ML platform that would make our internal teams’ lives easier and save them a lot of time and headache. This was, however, still very vague and high level, and more exploration was needed to determine whether there was room for such a platform. To investigate, we did the following:
1.1 Mapping pain points
We started by asking ourselves “why?”. What are the current pain points around content that teams within the company are dealing with? We grouped them into 4 clusters depicted in the image below.
1.2 Defining our value proposition:
After having a better understanding of the main pain points, we created the CIP value proposition addressing these pain points.
Value propositions should answer questions like what is unique about what we are doing and what are we offering that others are not.
1.3. Vision and Mission
In order to build our identity, we needed to think longer-term — what will this platform look like when it is mature?
We envisioned CIP as a marketplace, where the sellers are our Data Scientists who build content ML models and our buyers are internal product teams that consume the model outputs for their different business needs.
From that, we crafted our vision and mission statements.
Vision:
Be the centralized marketplace for enriched content across Booking.com.
Mission:
Drive content-based decision-making by leveraging ML capabilities.
1.4. Defining the MVP
After we dedicated some time to the longer-term vision and identity, we needed to go back to thinking short-term and more specifically, how can we prove value immediately. A lot has been written about the MVP (minimum viable product), so I’ll jump straight to how we defined it:
One ML model to be used for a single use case within 3 months.
1.5. Defining our KPIs — how will we measure success?
The last piece of this product package was defining our KPIs. Below are 3 groups of KPIs we defined for CIP:
- Adoption: Number of teams using CIP, number of use cases using CIP
- Impact: Monetary value, cost savings, customer LTV (lifetime value)
- Average SLIs (service level indicators): Data availability, service success rate, etc.
Step 2: Design and Execution
While in the first step we were focused on the “why?”, this step is focused on the “how” and was primarily owned by our engineers.
During this phase, the engineers brainstormed together and built the different levels of architecture design, thinking of the relevant components, frameworks, and more. After having the architecture design diagrams in place, we focused on the “must-have” components required for our MVP.
At this point, our MVP was launched!
Having tested and validated that it worked successfully, we had the green light to scale it up.
Step 3: Scaling Up
Scaling up CIP involved two aspects:
- Adding more use cases from different teams to expand our reach
- Improving and adding technical capabilities in order to be able to expand the coverage of use cases we are able to support.
A platform product without customers and use cases with which to feed it, don’t add much value. Hence, one of my most important jobs as a platform PM was to carefully map our stakeholders in the very initial phase of the product and continuously manage and prioritize them.
CIP has 3 different types of stakeholders:
- ML model producers — these are the teams that have machine learning resources and are building content ML models which we would like to onboard to our platform.
- ML consumers — these are the teams that would like to consume content ML for their products or features, whether they have data scientists in their teams or not.
- Technical teams — IT, Cloud, Security, etc. In this case, we are the customers, and in order to provide a reliable and robust service, we must be well aligned with our technical stakeholders.
In order to showcase our capabilities to the different stakeholders and better explain what we do, we created a playground that provides a quick glance into the platform. In this playground, one can input any image from the Booking.com website and get the ML enriched content that matches it.
As mentioned, the second aspect of scaling up was adding technical capabilities in order to be able to expand the coverage of use cases we are able to support.
One common requirement that our first version of the platform couldn’t support was aggregation. CIP managed to digest and process content at the image/text level (item-by-item) but wasn’t able to support extracting the information on the hotel/destination level (for example). Since this was a repeating requirement, we decided to invest time in it, and by that being able to increase the coverage of use cases we are supporting.
In summary, building an ML platform is a long process that must include both long-term and short-term thinking. We must define a scoped MVP in order to prove our immediate value, and scale from that point. We should also dedicate a sufficient amount of time to our stakeholders, as they are the drivers of our impact.
