Interning with the Airbnb PX & SX Teams
By Anna Matlin
My name is Anna Matlin and I’m a rising senior at Princeton University studying Operations Research Financial Engineering. I spent 12 weeks as a data science intern at Airbnb, working on the Product Excellence (PX) Team, which is responsible for the Help Center, the Resolution Center, and any features that empower users to help themselves on the platform. I also worked closely with Service Excellence (SX), which builds tools to maximize the productivity of Customer Experience (CX) specialists.
I had an amazing experience at Airbnb. As an intern, I was assigned a challenging project and supported fully along the way by team members on PX, SX, and CX. Over the course of my internship, I designed and implemented a new taxonomy for CX data, creating new ways for PX and SX to understand the “life of a ticket”. The data I introduced will enable cost savings on the order of magnitude of tens of millions of dollars.
Some background on PX & SX
While many of the teams at Airbnb are found across many companies — for example, Payments — PX & SX operate on a less familiar plane. Moreover, since only a fraction of users experience issues on Airbnb, it is easy to forget that Airbnb has a Help Center, a Resolution Center, and thousands of CX specialists working around the clock to serve the Airbnb community. Therefore, before I jump into the details of my project, it will be valuable to create some context about the high impact work that PX & SX are performing every day.
I quickly learned that the sphere of PX and SX is especially challenging because it is so human. The realm of Payments, for example, is incredibly complex and intricate because of the sheer volume of transactions, currency exchange, bank routing, and more. But PX and SX are tasked with understanding human issues of every shape and form, at scale: the first time host who can’t quite set her calendar properly 3 months in advance, the loyal user who arrived to a poorly cleaned listing and can’t get in contact with the host, or the well-intentioned guest seeking a short term alteration after a car accident. When SX tools enable a CX specialist to save the day, or PX recommendation algorithms empower a user to find answers seamlessly, they enhance the user experience and the feeling of belonging that drives the entire Airbnb platform.
In addition to serving the mission of the company, PX & SX generate enormous cost savings because the teams reduce the number of users contacting CX, as well as time spent solving each case. Before PX & SX were formed, Airbnb was simply hiring and training more CX specialists, which was not a scalable solution.
Introducing structure into CX ticket data
My project for the summer was to “introduce structure” into the vast realm of CX ticket data. On one end, PX maintains data about the help users search for before contacting CX. On the other, SX maintains data about ticket routing and cost metrics. However, the actual process by which a CX specialist solves a ticket, the how, is recorded in the format of unstructured text notes peppered with CX shorthand such as GCI (guest called in). As a result, data scientists on both teams are limited in the kind of analysis they can perform: PX can’t predict what kind of resources would be most useful to a help-seeking user; SX can’t identify particular CX actions that correspond to higher cost metrics.
The open ended nature of this task, when combined with the domain knowledge learning curve of PX & SX, made for an extremely challenging project. For the first few weeks, I shadowed CX specialists to observe their workflows, studied internal resources for CX as if I were a new CX specialist myself, and dove into existing text notes data. Before introducing structure into CX ticket data, I needed to develop an understanding of if, and then how, human actions mapped to the current unstructured data.
Therefore, my first few weeks as a data science intern were highly unusual; I felt more like a PM, shuttling between teams from meeting to meeting and designing mocks of possible taxonomies for the data. I found that “out of the box” methods like clustering and simple regex parsing in Python were useful to get a sense of different themes in the text notes. Through this combination of qualitative fieldwork and quantitative analysis, I broke down the space of all actions CX specialists were describing in their notes into a finite set of categories and corresponding subcategories.
Implementation and logging
The next step was to translate the taxonomy I had designed into real data that would be accessible to PX and SX for analysis. I wrote the code for a directed acyclic graph (DAG) in Airflow, Airbnb’s open source platform, to schedule and monitor data pipelines. Airflow jobs are written using a combination of Python and HQL scripts. Since I had limited time to build the DAG, which I hoped would be extended and maintained after I left, I was extremely careful about the data I chose to include. The DAG I wrote covered as many of the categories of the new taxonomy as possible, with just enough new data in each category to showcase potential for PX & SX insights.
After I built the DAG, the next step was to make a case for continuation of the DAG as a long term project. The data would be valuable to both teams: for PX, understanding concrete actions CX specialists use to solve routine, simple tickets presents opportunities for automating help, in turn freeing CX agents to work on more difficult, impactful tickets; for SX, the same data could be used toward workforce management, routing optimization, and feature roadmaps for internal CX tools. In order to prove the value of the information, I needed a simple example of how the DAG could drive cost-saving insights.
I dove into the new data and performed an analysis of ticket routing, identifying a small group of tickets that were routed inefficiently. Digging deeper, I identified trends among the CX actions for tickets in this group that could explain why the tickets were routed a certain way. These trends shed light on possible causes of the inefficient routing, enabling CX to step in and eliminate the problem. I then estimated the potential cost savings to be $1M annually. With this case study for the value of the data, as well as a detailed roadmap for expansion of the DAG, I was able to transfer ownership of the DAG to SX for the long term.
My three months at Airbnb were a fantastic learning experience, both technically and non-technically. As an intern, I worked on a high impact project that will be continued and expanded after I leave. I had the opportunity to visit the Portland office to meet team members on SX and CX, who were a wonderful and welcoming group. In general, I was constantly in awe of how thoughtful, kind, and supportive everyone was on PX, SX, and CX, in addition to being incredibly productive and hardworking.
It’s difficult to express concisely what’s so special about Airbnb’s culture, but I’ll do my best. On a high level, it’s a result of the mission: to create a world in which anyone can belong anywhere. And then there are the little things, like fresh squeezed orange juice in the Eatrium every morning and the 1:7 dog-to-employee ratio. As cheesy as it sounds, I was truly excited to come into work every morning because of the boundless optimism, inclusivity, and can-do attitude of the people around me. Looking back on my summer, I’m grateful not just for the chance to learn new skills but also for the opportunity to engage with the extraordinary Airbnb community.