Surveying for Insights at Scale
At Wattpad, we aspire to enable writers to create and publish their most treasured stories and to help readers to discover the stories that will most captivate them. The journey to create a story and to have it reach its right audience is a long one with many obstacles along the way. Our job from an experience design perspective is to smooth out that journey so that more users willingly spend more time, effort, and money on Wattpad.
To sift out the most critical user obstacles, we regularly survey our users on how they’re able to use Wattpad and how they feel while using it. We’ve found some survey methods to be quite unrevealing which has compelled us to be much more pointed in what we ask of users. Through much trial and error, we have seen the importance of crafting questions that align with how users use your product as a way of opening up the more candid responses that will help us identify the real user problems that matter.
But who cares about surveys when you’ve got data?
The goal of a great design is to drive user behaviour. We begin to understand user behaviour through data. Data by itself, however, is just presumed behaviour, and presumptions in design are dangerous. Data tells us where users have clicked, how many are engaged, and the percentage that are converting. Analyzing that data helps us to map out the user journeys where there is engagement, conversion, as well as drop off — what data can’t tell us, however, is far more important: why.
Let’s take for example the use of User Profiles on Wattpad. As a social storytelling platform, readers and writers can connect with each other through following one another via their profiles. Users often make comments on each other’s stories and many strong friendships have formed over the connection that stories provide. We noticed that some users are very active in following others which can lead us to presume that they are extremely social. What we uncovered through talking with users, however, was that users were following others not so much for connecting with them but more for following their reading list recommendations. It turns out that many readers find that following others with similar reading interests is a very reliable way to find relevant stories on Wattpad.
What data is invaluable for is identifying problem areas within our experience. However, it must always be complemented by the context of what our users’ intentions are if we are going to be able to take meaningful steps towards fixing those problems. And the only way that we can understand our users’ intentions are, is to create a way for them to tell us.
Starting with Usability Benchmarking
The first type of user surveying we conducted was centered around usability. We remotely recruited users and asked them to complete a series of tasks on Wattpad: Can they use the advanced search? Do they know how to tag and categorize their story? Are they able to customize their profile? We tracked their success rates and then asked for their overall perception on usability using the System Usability Scale (SUS).
What’s great about SUS is that it yields a percentile ranking (aka “SUS score”) out of 100 whereby 68 is the average. Whether you score above or below the average can give you an indication of how much you need to focus on improving the fundamental usability of your product.
Although this method proved helpful to highlight feature usability enhancements, it didn’t provide a more holistic view of Wattpad in the context of being a social storytelling platform. Usability is a necessary foundation, but it wasn’t going to be the differentiating factor in engaging readers and writers to choose Wattpad for their stories.
Capturing Multidimensional Sentiment
While usability is a good, fundamental benchmark, we knew that it was only a starting point. Wattpad is differentiated by its diverse library of stories and the community that surrounds those stories so we needed measures that captured those differentiating dimensions. We discovered a measurement approach called AttrakDiff which is effective in capturing the emotional feedback for a product. With this method, users evaluate your product against a series of opposing word pairs that represent the different dimensions of your product. For Wattpad, we developed word pairs aligned with the dimensions of Usability: Can people use it?, Stimulation: Are people motivated to use it?, and Community: Do people feel like they belong?
The results helped us to holistically monitor the impact of product updates and new feature launches. We could track whether or not the perception of usability had improved, if Wattpad was becoming more attractive or compelling in its use, and whether or not social interactions were being conducted in a safe and inclusive manner. While this method gave us an indication of overall health, it lacked precision in linking us to specific problem areas to focus on.
Getting Critical with Engagement
In order to be able to pinpoint critical problem areas, we needed to evolve our surveying methods to ask more specific questions that would help to catalyze more detailed responses from our uses. We aligned our questions to concentrate on the pivotal tasks that would lead to greater or reduced engagement of our users. So instead of asking their general feelings about Wattpad, we would ask them about their ability to fulfill a key task or goal.
For readers, their critical interactions revolve around their ability and effort to find stories:
- Ability: “I can find a story I want to read on Wattpad”
- Effort: “How long does it take you to find a story on Wattpad?”
However, it was our follow up question that has proven to be the most useful:
- “What makes it more easier / difficult for you to find stories?”
The primary difference with this follow-up question is that it is open-ended where it enables users to share their approaches without being filtered out by our presumed answers. Through these open-ended responses, we learned that users employed multiple search strategies (search by tags, search for similar stories, search by author’s reading list) to form their own story-finding system and that they’ll go to great lengths to find the next perfect story. Instead of viewing search as a singular task, the open-ended questions helped us to uncover that search was a triangulated set of tasks. By being open-ended, we became open to discovering new insights.
From Net Promoter Score to Net Value Score
One of our earlier measures that we incorporated to gauge overall satisfaction was based on Net Promoter Score (NPS). NPS is a standardized and well-adopted measure where you can compare yourself against industry standards within your vertical. NPS measures the loyalty of users to a company or product by asking one simple question:
How likely is it that you would recommend [company/product] to a friend or colleague?
Respondents provide an answer on a scale from 1 to 10 where the proportion of respondents at the higher end of the scale are weighted against those at the lower end of the scale. A score from -100 to +100 is computed where a higher number is desired.
For us, we found that NPS didn’t vary all that much from month to month despite changes being made to the product. We also heard through user interviews that readers preferred to keep their reading interests private and thus didn’t necessarily promote content platforms in the same way they might with other products. And given our focus on enabling more paid and subscription offerings, we needed a more direct measure of the perception of value from our offerings.
We discovered a more relevant measure called Net Value Score (NVS). NVS measures the perceived value of a company or product relative to comparative options:
How would you rate [company/product] on the benefits it offers, compared to the product/service benefits offered by other suppliers of similar products/services?
We’ve adopted NVS to measure the value of our free, ad-supported experience and for our paid subscription offering. NVS helps us to gauge our comparative value relative to alternative choices: Do free readers find enough value in our stories that they’ll continue to engage with an ad-supported experience? Do readers understand the value of our Paid Stories relative to stories that they could purchase elsewhere? How much value do subscribers see in our subscription offering compared to other content subscription offerings? NVS helps us to discern if we are ahead or behind in the value we provide or if we need to focus more on the awareness of the value we offer.
Making Surveying Work for You
Surveying can be an incredibly efficient way to get a lot of user feedback all at once and will continue to be a key tool to better understand opportunities for driving desired user behaviours.
Through our experiences across multiple surveying approaches, we have learned how to make surveying more useful and valuable through the following:
- Be multidimensional: A product needs to be evaluated across multiple dimensions beyond utility or usability — identify the dimensions that are important to your product
- Be open-ended: Allow users to tell you what’s important to them and how they go about using your product — don’t box them into predefined answers
- Focus on critical interactions: Ask specific questions around the most critical interactions that foster or compromise engagement — what makes them succeed or fail?
- Be comparative: Ask questions like NVS to assess whether you’re leading or lagging your competitors in the value you provide
While it has been a journey to get to our current surveying approach, it has been rewarding to be able to unlock new insights from our users in a way that is meaningful to Wattpad. Researching amongst users is very much like designing for them where it takes a number of iterations to get it right. By being persistent in finding the right questions that would help our users to open up to us, we are now much more sure that we’re solving the right problems that would make Wattpad an even better part of their lives.
