Systematizing customer feedback
At Paperless Post, we believe that creating a great product comes from having a unique vision, executing and relentlessly responding to user questions, issues and feedback.
Customer Operations has always been more than a service we offer— it is an important source of insight. The fact that our users care enough to write us is something we see as a great sign, and an opportunity to learn.
But how do we separate the signal from the noise?
We get a lot of feedback. Paperless Post offers a design-driven online events product, and since we launched, our users have sent over 500 million invitations. Managing what we hear from users in a systematic, prioritized way hasn’t always been easy.
Actually, when we first launched, prioritization wasn’t that hard because bugs were often urgent and needed fixing. Most of our time in the early days was spent keeping things afloat as we grew. Once we were a more stable business with a bigger team, it got harder to prioritize user issues, especially when compared with new features. At first the Customer Operations team met with every product team every other week. Later we implemented a “Support dev rotation” where one engineer was always on call and it was their job to fix or escalate the most important issues as they came up.
None of these processes solved the problem of how to prioritize “the most important issues.” It took launching a new product, Flyer, a few years ago, to realize that there was no getting around the fact that we needed a more objective, data-driven way to decide how we should prioritize bugs and user experience issues coming from users.
Systematizing by necessity
Like our original product, Flyer originated from a creative vision about what users would love. We knew they would respond well to self-expression tools, so we built a design tool where you can customize your own GIF and choose a variety of well matched fonts and colors for the event page. We built modular event blocks, text message distribution, and many other long-requested features from users.
Flyer’s growth after launch was faster than our original product launch, and on a bigger base. There were too many user tickets to use the same grassroots method of listening that we did in 2009, where everyone at the company was either answering a user email, taking a call or fixing an issue at any given time.
Our Customer Operations team was now bigger and more sophisticated than our whole company had been in 2009. The heads of Cust Ops at the time, Christina and Sarah, knew there were 2 main problems:
- We needed to track what we were hearing back from our users in a way that left no user feedback out, and was centralized.
- The information needed to be saved and organized in a way that could easily be read and understood by the rest of the company.
So they proposed a system that could do these two things.
Customer Ops Tagged Issues
Our heads of Customer Ops worked with the Product team to create a system of tags that they would use to describe user issues. Each kind of unique issue would have a tag, e.g. “write a message to guests”, and it would be related to an umbrella tag like “Distribution & Event Management” that described a part of the user experience. It was very important that this system aligned almost completely with how the Product and Engineering teams viewed the functionality.
Once the tagging system was agreed upon, the leaders of the team went back and tagged every historical issue from the beginning of Flyer to stress test the tagging system, to make sure nothing was missing. This was brutal work, but they knew the data had to be complete to be trustable.
From those tags, they created a set of macros (templates that all Customer Ops agents use to write back to users), each of which corresponded to one or more of the issue tags. They trained the Customer Ops team to always use macros so that no ticket would go untagged.
Today, each time a user writes into Paperless Post looking for help, not only do they get helped, but their request automatically becomes a tagged issue.
Every evening, thanks to our data team, the issues that came in that day show up on a dashboard where anyone in the company can see all of the issue tags and how they have changed over time. They can also click into an issue tag and see the contents of the tickets written in real users’ words. Product managers for example, don’t need to hear from our Customer Ops team about what issues users are having on the tracking page, they can read for themselves.
Goal: Getting off the top 10 list
At any given time we have a Customer Ops “Top 10” list, a.k.a. the issues causing the most tickets and confusion. It’s one top 10 list where success means getting off quick! Every day there is a lot of fresh data and it is sensitive to changes in the product. After a new feature or bugfix goes out, the list can completely change, and what was the main problem drops way down the list. It is motivating to see that instant feedback if you’re the one who fixed the issue!
Every time a top issue goes down, a new one or set of ones will rise up. The goal is not to have issues go away (that’s not possible), but to have the top user issues list change as often as possible. Ideally with the curve starting to flatten out , as the product becomes more mature…until you release something new, and the cycle begins again. Users will always need help and have issues. Our goal is to make sure our velocity in solving those issues goes up and thus the product is getting better.
This form of tracking issues also reveals subtle changes like 3% increases or decreases in certain issues week-over-week — changes that we wouldn’t have found without a tag system.
No one data source tells the whole story
As product developers, we need to know the strengths and limitations of each source of data. Often the answer to a user or business issue comes from triangulating between data sources to see what’s really going on.
Users who write into Customer Ops give us a very clear and critical view into what the strongest points of friction are for those already trying to get through our process. They will write in because they are invested already but don’t feel capable or confident figuring out a roadblock on their own. They are willing to write about what they are having trouble with, and even willing to speak and answer questions and make clarifications.
This is great data, but it doesn’t capture a few things. Numbers wise, users who write into Customer Ops are between 3 and 6% of the events we serve every week. It would be a shame to lose them and not to learn from them, but we know there are many more who don’t write in! Studies have shown that for each person who does write in, there are between 10 and 20 people having the same issue who do not write in.
There are also other sources of data that we track, which help capture the user experience in other moments, for example NPS, and Product Analytics. None of these alone is perfect, but when they all point to the same issues, it’s powerful signal. And of course none of these data sources capture things people haven’t thought of yet (like a new creativity tool). That is where vision comes in and plays a role.
Why this matters
Our team took something that was historically very emotional and hard to be data-driven about (qualitative customer feedback) and abstracted it into a bigger set of data that gives us a read on user experience and sentiment and how it’s changing.
Besides using data to make product decisions, our other goal in systematizing user feedback was internal alignment. As a consumer company, our users are collectively, our boss. Not even the most brilliant person inside our company knows better than the sum of all our users. Our goal is to enable anyone in the company, a designer, developer, business person to get in the mindset of the user, learn what they want from us, question why we’re doing something a certain way, think about how we can serve them better, and thus really excel.
