8 steps to beta study success
Here at The Times and The Sunday Times, our design team prides itself on making informed decisions based on user research. So, what do you do when you’re tasked with overhauling an app that’s fiercely loved, part of users’ daily routines and cherished for its unchanged format over the last decade? A lot of research, that’s what!
Wondering how we took on the challenge? Read on to discover the essential steps we followed in our beta study to ensure a smooth transition to our newly designed app for our valued subscribers.
Our approach
Before kicking off the beta study, we ran numerous diary studies to understand habits and behaviours.
With a direction confirmed, we began to define and piece together the experience, ideating and designing features to minimise disruption and boost adoption, and testing with users in the lab.
Our process
Before any potential launch we wanted to measure true adoption with wide-scale validation and longitudinal testing of the experience.
These are the steps we took to run a 7-week beta study with our users:
1. We planned. A lot
We created a detailed plan for the 7 weeks that allowed users to adjust and adapt to the new app, while allowing us to improve issues and functionality over the same period.
A detailed research plan outlined quantitative and qualitative methodologies, with all surveys scripted, programmed and ready to go.
The beta’s key success metrics, reporting format and timeframe were all agreed before launching the beta app with stakeholders.
2. We identified what success looks like
Our key success metrics were set in stone at the start of the project, and monitored throughout via surveys embedded within the app banner. This feedback allowed us to clearly see changes over time.
Our key success metrics were:
- Customer satisfaction: probably the best barometer of the quality of the user experience provided by a product.
- NPS (Net Promoter Score): like satisfaction, recommendations are also a great barometer of UX. It goes without saying that a user who has had a great experience, is more likely to recommend that product or service to someone else.
- Likelihood to churn: an important metric to monitor paying subscribers.
3. We recruited a representative and statistically valid sample
We tested with our actual subscribers, as their input was the most important and helped us much more than developing a “beta” group.
For our research to be statistically valid, our magic sample number was 385, however we recruited well over this figure to allow for users dropping off over the seven weeks, and to allow us to segment our data.
4. We benchmarked existing attitudes and behaviours
It was extremely important for us to understand attitudes and behaviours on the existing app to compare against the beta.
After recruiting a representative sample, we surveyed them on the agreed key success metrics and captured their analytic behaviour and demographics.
We were also able to identify the users on an individual basis during the whole research project. This allowed us to observe changes in behaviour and attitudes over the testing period.
5. We brought our users to life
We recruited users for in-depth interviews straight from our in-app surveys.
We conducted on average 8 interviews per week and targeted a mixture of users who were satisfied and annoyed. Stakeholders were shown short video clips of the users talking about their experience.
6. We encouraged engagement and feedback
User engagement throughout the study was important.
We created a sense of community through our communications and were responsive to user emails, making users feel important and valuable. We also ran weekly competitions, which encouraged engagement and feedback.
7. We tracked user behaviour through analytics
User behaviour played an important role in telling the story of our beta study. Dashboards allowed us to easily monitor weekly activity and self serve data.
8. We combined all our data.
By comparing final key success metrics and analytic metrics with our benchmarks, we could clearly see if the app was a success. Overlaying behavioural data with survey data and digging deep into our user segments really helped us understand our users.
Key takeaways
Based on the steps we followed, here’s what we’d suggest keeping in mind for your next beta study:
- Planning is key
- Use a valid sample — and recruit more users than you think you’ll need
- Understand your users’ existing behaviours and attitudes
- Define success early
- Speak to your users directly
- Incentivise feedback and engagement
- Track your users’ behaviour
- Bring all your data together to get a complete picture of your users
