Contextual Calendar Reminders

Successful hosting has everything to do with an up-to-date calendar.

By Tao Cui and Yuting Gu

Airbnb’s mission is to create a world where anyone can belong anywhere. So when hosts forget to update their calendars and are booked by guests on dates that don’t work, the host has to cancel the reservation. This definitely does not make the guests feel like they belong. Inaccurate calendars are also detrimental to the business. Being cancelled on is a frustrating experience that makes guests think twice about using Airbnb for a future reservation. Airbnb also compensates the guest in these situations. An accurate host calendar saves all parties a lot of frustration and advances the company mission of enabling anybody to belong anywhere.

Calendar Reminder Email

To help improve hosts’ calendar accuracy and avoid canceling bookings, we’ve created handy calendar reminder emails to give hosts extra reassurance. Hosts can opt out of the reminders if they like.

How Should We Send Reminder Email?

We want to provide the host with appropriately timed reminders such that the host maintains a substantially up-to-date calendar, but is not overwhelmed with excessive reminders such that the host unsubscribes or opts out from receiving calendar reminders to update his or her calendar.

We developed two strategies to send calendar reminder:

Smart Calendar Reminder:

We want to provide “smart” reminders to hosts based on how often each host typically checks his/her calendar. We first look at how often hosts look at their calendar by plotting the distribution of two adjacent calendar checks. It appears that the distribution follows an exponential distribution, which indicates the calendar check may follow a Poisson process.

Fig. 1 Probability distribution of days between two adjacent calendar checks

The probability that a host checks his/her calendar t days from previous calendar check follows the distribution

where a is the average interval between two adjacent calendar checks.

This indicates the calendar action ratio

could be a good indicator of whether a host’s calendar is outdated. For example, if a host typically looks at their calendar every 5 days and he/she hasn’t looked at their calendar for 15 days, it is very likely that the calendar is outdated.

The average interval of two adjacent calendar checks is based on a few samples, which may not be reliable. For example, if we only see one sample that the user checks their calendar 5 days apart, we cannot say he/she will check his/her calendar every 5 days. We can show that the estimated average interval follows a chi-squared distribution

where n is the number of samples used to estimate the average interval a and c is the estimated interval.

This indicates the probability, that a host checks his calendar t days from previous calendar check over estimated interval c is greater than some threshold, is given by

Therefore, given a probability reliability and number of samples, we can estimate the real interval as the product of estimated interval with a multiplier. The figure below shows this multiplier as a function of the number of samples and reliability.

Fig. 2 The multiplier to multiple the estimated average interval as function of number of samples

We look at the impact of calendar action ratio on booking decline and cancellation. We can clearly see that decline and cancellation rate increases monotonically as calendar action ratio increases, which shows that the ratio is a good signal to predict decline/cancellation.

Fig. 3 Reservation decline/cancellation rate vs calendar action ratio

Now the question remains on how we should set the calendar ratio threshold to send reminder emails. Sending calendar reminder emails has two impacts. On one hand, calendar reminder emails may help hosts keep an updated calendar and reduce cancellation. On the other hand, excessive calendar reminders may make hosts unsubscribe our emails, which may lead to an increase in reservation cancellation. Since cancellations are closely tied to revenue, we obtained a final threshold by maximizing revenue through balancing potential increases and decreases in revenue.

Regular Calendar Reminder:

As stated above, maintaining an accurate calendar is crucial to all parties on Airbnb. However, new hosts may neglect the importance of having an accurate calendar and do not connect their earnings with calendar accuracy. Moreover, the smart calendar reminder does not work for new hosts as we do not have data to learn their typical behavior. Therefore, it is important to help new hosts to form a habit to check calendar regularly.

Regular calendar reminders can help hosts form the habit, and we can send reminders to hosts if they forget to check their calendar in the past week. Now the question becomes exactly when we should send the emails so that they are the most likely reacted by the hosts.

We analyze data and find the probability that a host will view their calendar on each weekday in different countries in Fig. 4 below. Many hosts are likely to check their calendars near the weekend, such as on Friday, Saturday or Sunday. The rationale is that hosts may want to check their calendar to prepare hosting in the next week.

Fig. 4 Probability of checking calendar vs weekday across week

For the calendar reminder emails we sent, we would like hosts to open them as much as possible. We analyze the data and find the probability that a host will open an email from Airbnb cross weekday over different countries in Fig. 5 below. Interestingly, hosts from different countries do not have a common email opening behavior.

Fig. 5 Probability of opening email vs weekday across week

Finally, to determine the optimal time for sending reminders, we want to select the time that corresponds to the highest likelihood that a host will read a calendar reminder email and access the calendars in response to receiving the reminder. We multiply the probabilities in Fig. 4 and Fig. 5 and get the probability in Fig. 6 below. We can see that the highest likelihood is hit on Sunday across all countries. Therefore, we send calendar reminder emails on Sundays.

Fig. 6 Probability of opening email times probability of checking calendar vs weekday across week

Implementation

We build a data pipeline using the platform Airflow open sourced at Airbnb. The data pipeline is run daily to process the data logged in the previous day. It takes several hours for the data pipeline to finish and the finish time also varies from day to day. However, hosts may have already checked their calendars when the pipeline is running. It will be awkward if we send a calendar reminder email to a host who has just looked at calendar.

To solve this problem, we adopt the computation framework AirStream developed at Airbnb. Airstream is a real time stream computation framework built on top of Spark Streaming and Spark SQL. It allows engineers and data scientists at Airbnb to easily leverage Spark Streaming and SQL to get real time insights and to build real time feedback loops.

The architecture of Airstream is given in Fig. 7. Airstream takes a very simple config. It defines sources (e.g. kafka), computation (e.g. SQL query), and sinks (e.g. kafka, redis, hdfs, hbase).

Fig. 7 Architecture of Airstream (By Liyin Tang and Jingwei Lu)

In our implementation, we config an Airstream job to collect calendar view impressions. We define the source as kafka and the sink as a hbase table with TTL 24 hours so that it will only keep data in the past 24 hours. The Airstream job is run every minute. As soon as the main calendar reminder pipeline finishes, we join the generated host ids with the host ids that appear in the Airstream sink table where the later host ids are excluded in the final list of sending calendar reminder. Finally, we send reminder emails by the Mochi, which is a service developed at Airbnb to send high volume marketing emails.

Fig. 8 Implementation of calendar reminder

Conclusion and Future Work

Our experiment results from this calendar reminder email have shown great impact on the Airbnb business. Calendar reminders have helped hosts with successful hosting and lead to guests feeling like they belong. This is also a good example on how real time data can be useful to address users’ feedback immediately.

For regular calendar reminders, we are sending emails at a particular day of a week for all hosts that need to be reminded. We are working on a machine learning model to find personalized email sending time that is the best for each individual host. We are also working on integrating the data from this project into other products. Please stay tuned!

New technologies have made it easier to keep each other at a distance. We are using them to bring people together and to make people feel more belonging.

We work together, across different teams, to fullfill the mission and deliver our commitments. Special thanks to Andrew Chen for helpful discussion on email sending criteria, coordinating cross functional teams and reviewing this article, Tracy Kuwatani and Cecilia Yang who contributed the beautiful design, Max Davidson for heartwarming content, Logan Ury for intriguing user research, and Stephanie Pancoast for great support.

We’d like to thank Liyin Tang and Jingwei Lu for their generous support on Airstream. Shoutout to Trunal Bhanse, Lenny Rachitsky for the input and Angela Zhu for reviewing this article.

Interested? We are hiring!