Building an experience framework

Zhibo Wei, Steven Liao & Jay Kim, Pinterest engineers

At Pinterest, we’re committed to delivering a great experience to more than 175 million monthly active users. In addition to core features, like home feed and search, hundreds of in-product experiences are presented to Pinners every day, including new user onboarding flows, user education, tooltips and more. These experiences are built by multiple different product engineering teams. In order to ensure consistency and proper user targeting, we built a system called Experience Framework.

The challenge

As tens of new experiences are delivered to Pinners each week, there are a few issues we need to avoid:

1. UI conflicts. We want to prevent multiple experiences from showing in the same page and potentially conflicting with each other. For example, it would be a bad experience if a tooltip, banner and modal show together in the same page (as illustrated below).

2. Spammy user experiences. Sometimes we need to show experiences repeatedly without being spammy. For instance, we educate users on creating their first Pin a few times (until they actually do), but we don’t want to be too aggressive.

3. Building experiences takes time. It usually takes the following several steps:

  • Implement the custom targeting feature for specific user groups.Implement the logic to control the experience’s frequency and duration.
  • Coordinate with other teams to implement the product experience and mitigate potential conflicts.
  • Implement the server layer to gather the content data shown.
  • If needed, take care of metrics and stats logging.
  • Implement the client (i.e. mobile, web) to render the feature.
  • Testing and QA.

These are the steps for most product experiences, and the logic behind them is very similar to each other. Before our framework, developers would recreate the wheel each time they added an experience to the product.

System goals

As we built our product experience framework, we wanted the following properties:

  • A centralized system to manage all product experience configurations and solve conflicts across teams and functionalities. For any given page, the system decides which product experience to show the user. In Experience Framework, all product experiences are grouped into different collections by their showing locations, and conflicting features are bundled together so the system can easily solve conflicts by selecting the best one for a user.
  • A system with a unified schedule for showing product experiences. To work with Experience Framework, all product experiences are represented as a JSON configuration, which includes the target user group specification, render frequency, the UI content and more. Based on these static configurations, the system selects eligible experiences to show to the current user at a given time.
  • A system that speaks a common language across different platforms with generic client library support. Experiences on different platforms should follow the same protocol to ensure the same recipe can be displayed across platforms like web, iOS and Android. Standard client libraries should be built so that developers don’t need to re-implement components multiple times. This approach will significantly boost developer productivity and make the user experience more consistent.

Example

Before we jump into the details of the system we built, let’s take a look at an example of a banner experience on web:

To build the blue banner at the bottom of the above image, the web developer needs only to write the following code:

As you can see, the Experience Framework only requires two lines of code. All the developer has to do is input the experience definition. Every other detail on how and when to render the banner are completely encapsulated in the Experience Framework, and the client library is decided on the server-side.

Architecture

Now let’s get into some technical details on how we built this system.

The core of Experience Framework exposes RESTful APIs to end users, so all clients talk directly to Experience Framework. To determine the features shown to a user, Experience Framework relies on data from two different sources–the experience configurations on S3 and data stored in HBase.

We also built a web tool for developers to edit the configuration of their experiences, as illustrated below.

We use MySQL to store draft configurations. Once the developer is ready, it’s easy to publish the updated configurations into production in real-time.

To log requests and user actions into Kafka (which are then loaded into Hive through a data pipeline), we built an analytics tool so developers can understand how experiences are performing.

There are few critical points in this system worth a closer look at:

  • Data Storage: There are two main considerations we had when selecting the data storage system. First, we need a linearly scalable data storage system to support the massive traffic we have every day. We also need to dynamically expand the data schema to add new columns on the fly since experiences can be deployed in runtime. We decided to use Hbase since it meets our requirements.
  • Configuration distribution system: Experience Framework will run on thousands of servers, so it’s critical to be able to synchronize the configurations across all the servers in real-time. We take advantage of Pinterest’s in-house s3+ZooKeeper system to make this happen. With the help from the configuration distribution system, we can update product features in runtime without deploying any code.
  • Client library: We spent a lot of time on building and improving the client libraries, because a properly designed and implemented client library can save a lot of trouble. For example, we encapsulated most of the technical details and hid them from client developers which allows them to focus on business logic. It also increases developer productivity and reduces the risk of bugs.

Lessons learned

Each time a user opens Pinterest, a request is sent to Experience Framework. Overall, the whole system is receives around 1.5B requests every day, and hosts more than 300 experiences from multiple engineering internal teams. We learned a lot during the process of building such a critical system that supports a variety of of teams, hundreds of developers and billions of requests. Here are the main lessons:

  • Always modularize the code and de-couple different components as best you can. As we need to support various product teams, new feature requests are common. It’d be painful to restructure a large portion of code to support feature requests.
  • As a central and critical system in Pinterest, the framework connects with many components, and any piece could cause trouble to the whole system. For example, a poorly designed client code can double or triple the traffic to Experience Framework. Instead, we need to aggressively put safeguards in place.
  • System visibility is important. You have to understand how every part of the system is working at anytime, especially if an incident happening. That’s why our team spent a lot of time building a comprehensive alerting and monitoring system.

Future work

The current system decides which product feature to show based on static business rules and rankings. Sometimes those static configurations can be set arbitrarily, and it’s hard to tell the best values to set for those config. For example, the max display count was set to three, but the user may only need to see a feature once. Additionally, given there are many different fields available in the configuration, it can be hard and time consuming to set up different experiments to test all the variants. To solve this, we’re investigating the possibility of adapting the machine learning mechanism in our system so that we can better target users and decide whether to show the experience dynamically. This ensures experiences can be shown with their full potential and ultimately makes a better experiences for Pinners.

Acknowledgements: Many thanks to NUX, State Machine and many other product teams for giving us valuable feedback and helping to improve the system every day. Special thanks to Helen Fu, Victoria Kwong and Vivian Qu for building the client libraries for Experience Framework.