Expedia Group Technology — Data
Increasing Travelers’ Engagement Through Price Alerts
Helping travelers find the optimal deals for their planned trips
Authors: Oarinde Henok Yemam
Introduction
Expedia Group™ is dedicated to helping travelers find the optimal deals for their planned trips, guaranteeing maximum value with each booking. Expedia Group has made substantial technological investments to ensure trip planning and booking are seamless and enjoyable, resulting in an improved overall travel experience. One of the investments supporting trip planning and booking is flight price alerts. Flight price alerts are designed to address flight price fluctuations over time for price-sensitive travelers, so they book the flight of their choice at the price they deem fair. The way it works is travelers subscribe to their preferred flight searches through the price alerts campaign and receive notifications regarding price fluctuations. The traveler’s perspective on price alerts is illustrated in Figure 1. In this instance, the traveler is monitoring two different trips: a flight from Seattle (SEA) to Los Angeles (LAX), and another from SEA to Iowa (DSM). While both price alerts show a decrease in flight prices, it is important to note that this is not always the case.
Early on in our experimentation of price alerts solutions, we sent out push notifications (messages sent from the server to the user that contain information about a product or service) powered by a business rule which was later changed to machine learning (ML) so that travelers’ engagement with the alerts was enhanced. The focus of the transition to ML was to send alerts that are more likely to be interacted with while avoiding those that may cause user fatigue and disinterest. In essence, our goal is to strike a balance between boosting user engagement and minimizing any excessive or unwanted communications. A direct consequence of user engagement optimization is improved booking conversation rates. Beyond those metrics, we wanted to establish trust in price tracking and incentivize more travelers to return to Expedia for flight search and booking by providing timely alerts on relevant price trends at crucial stages of the shopping journey. We invite readers to the advertisement below (video 1) for a practice use case of price alerts.
Now that you are up to speed with the basic concept and purpose of price alerts, the remaining part of this blog will focus on exploring the technical specifics of our iterative ML solution approach.
Message Relevancy (MR) Model
As mentioned previously, our initial approach for delivering price alerts push notifications relied on predefined business static rules. However, static rules have inherent limitations as they are not adaptable to changing circumstances or user preferences. Additionally, the static approach lacks the ability to optimize push notifications, which can potentially lead to suboptimal outcomes.
In contrast, machine learning offers a superior alternative for push notifications in our travelers’ flight prices context. It provides an adaptive, personalized, optimized, and scalable approach. By harnessing the power of data and advanced algorithms, machine learning enables us to deliver timely and relevant notifications that enhance the user experience and yield better conversion rates. Unlike static rules, machine learning algorithms can dynamically adjust to evolving conditions and leverage insights from data to make informed decisions. This empowers us to provide a more tailored and effective push notification experience for our travelers. We used four guiding questions to build an effective ML system for price alerts. These were:
What outcomes are we trying to achieve?
Let’s take a step back and answer this question from the perspective of our travelers. They subscribe to price alerts because they want to wait until the right time to book a flight that meets their financial goal. Therefore, the model needs to meet this demand. From a business perspective, there is a cost associated with every price alert so if travelers are not interacting with the notifications, it means they don’t find the content or updates relevant. Hence, we designated the traveler’s interactions with our notifications as the target variable. This achieves both the traveler’s demand for relevant updates and business cost optimization. This binary indicator determines whether the user opens the notification (1) or does not open it (0). We focused specifically on the “open” action because we observed that only a small number of travelers clicked on the notification. This is because opening the notification provides travelers with all the essential information they need. We considered different types of “open” actions, including direct open (tapping on the notification) and indirect open (opening the notification later by accessing the app). However, we only considered direct open as the positive class for our modeling approach.
What data do we have to accomplish our objective?
The price alert campaign tracks the number of subscriptions and their accompanying interaction actions, trip route information and flight price fluctuations. From this data, we create input features such as the traveler’s past interaction with notifications, subscription profile such as the number of subscriptions and subscribed days, and, flight price delta since the day of subscription. By using these input features, we optimize the likelihood of a traveler opening a notification. If this likelihood is high, then we send a notification. This assessment is done every 24 hours.
What are the evaluation and business success metrics?
We evaluated the performance of our classification model using the recall metric, which measures the ratio of true positives to total actual positives. This metric allowed us to focus on identifying the proportion of travelers who are likely to open the notification but did not receive it. In production, we achieved a 98% recall rate. Our business success metrics encompassed increasing the percentage of subscribers receiving notifications, improving the notification open rate, reducing the subscription opt-out rate, and enhancing subscriber retention rate.
How can we leverage ML techniques on the available data to attain the desired outcomes?
For our modeling approach, we chose the Random Forest Classifier due to its simplicity and ease of parallelization. After deploying the initial model, we observed positive results, including an increase in the opt-in rate, the percentage of subscribers receiving notifications, and the notification open rate.
To improve the model further, we introduced both direct and indirect opens as the target variable. We also employed an in-house wrapper called Automated Model Tuning (AMT), which leverages Hyperopt and MLflow for hyperparameter tuning and experiment tracking. With AMT, we achieved significant performance improvements, increasing the F1 score by 20% compared to the production model. Initially, we utilized scikit-learn’s RandomizedSearchCV method to search through a grid of hyperparameter ranges. However, this approach had limitations, such as assuming prior knowledge of the hyperparameter space and lacking efficient parallelization for the hyperparameter search. To address these issues, we incorporated the AMT wrapper, optimizing the allocation of computational resources during model training and streamlining the hyperparameter search process. This enhancement made the model training process more efficient and effective in selecting and optimizing the best parameters.
Despite the model improvements, we acknowledged the bias toward past interaction behavior as the primary feature. To mitigate this, we implemented safeguard rules. These rules ensure that subscriptions that have not received a notification within a certain number of days after subscribing will receive one. Additionally, notifications will be sent to subscriptions where the total price change is equal to or greater than a specific threshold. These safeguards aim to ensure that approximately 90% of subscribers receive at least one notification during their subscription lifetime.
In summary, our initial model successfully delivered price tracking notifications to travelers with a minimum subscription period of 10 days, while defaulting to the business logic rule for shorter subscriptions. The improvements made to the relevancy model eliminated the need for the default rule, enabling proactive engagement with travelers and timely notifications starting from the second day of their subscription period, regardless of the subscription duration.
Conclusion
Since implementing the message relevancy model for price tracking, we have observed noteworthy improvements in various key metrics. There has been a substantial increase in the opt-in rate for the price alerts campaign, indicating a higher number of users expressing interest and subscribing to receive notifications. Additionally, the subscriber share receiving notifications has significantly risen, ensuring a wider reach and engagement among our subscriber base. Furthermore, the notifications open rate has experienced a notable boost, indicating that more users are actively interacting with and opening the notifications they receive. Overall, these positive results demonstrate the effectiveness and impact of the message relevancy model on the price alerts campaign.
Author Information
Dammy Arinde — Machine Learning Scientist; Henok Yemam — Machine Learning Scientist
