How we conduct user interviews and user tests at Clue: Part One
I find it fun to see exactly how other organizations handle certain design processes.
It’s the small the little details that I find interesting.
So, I thought I’d lay out exactly how we do work with users for research and testing purposes at Clue.
Clue is a female health tracking app based in Berlin. We’re now at the stage of about 50 employees, and we’ve raised over $30 million from VC firms such as Union Square Ventures. More than 10 million people use Clue.
The reason I give these details is that I think your organization’s stage will have a big impact on your approach to user research and user testing.
To give a quick example, we’re not at the stage where we test a single flow with hundreds of users to get quantitative data on the usability.
But we are well beyond the stage where offering free coffee in Starbucks in return for chatting to us for 10 minutes is enough for our research and testing processes.
In this two-part article, I’m going to focus on the practical, step-by-step process we use for talking to users, week-in, week out.
The design process
Our design process at Clue focuses both on designing a product that is scientifically-valid — our app lets people know when they can expect their period and other menstrual cycle symptoms and stages — and responds to the needs of the people using it.
That means that the two product designers at Clue work closely with scientific researchers and data scientists. This is how we make sure that the product meets rigorous scientific standards.
We also talk to the users at pretty much every point across the design process.
Here’s how I’d break up the stages of talking to users:
Blank page interviews
Even when we don’t have a particular feature idea in mind, we’ll carry out some user interviews. At this point we’re just looking for interesting problems to dig into that could be the basis of future feature ideas.
Feature research interviews
When we have a feature idea, we’ll kick off a round of user interviews. Here we’ll focus on people’s pain points, what their current solutions or what are the missed opportunities.
Prototype testing
At this stage we’ll have a prototype that we can show to people. It can be as simple as paper wireframes put together in an app like POP. We’ll quickly move to high fidelity mockups with Marvel or Invision.
I’ll give people activities to do with in a context. I’ll ask them to tell me what they are thinking as they do the activities.
For example, I might give this activity:
“You want to tell Clue you are now on the pill.”
I’ll watch to see if people understand the prototype and whether it sparks their interest.
Usability testing of existing features
We also carry out tests of existing features to get ideas on how to improve them.
What about quantitative user testing?
Currently, we don’t do quantitative user testing of prototypes. We’re usually testing a prototype on 3–5 users per round. For quantitative results, we’ll use surveys instead.
At the same time, we roll out releases in stages after spending a week at beta stage, so we get feedback very early on.
We also feature flag releases, so we can A/B test their impact on growth and retention.
The recruitment problem
My first-ever interviews were in a Starbucks in the Sentier district of Paris. I went down with a Windows tablet to test out an app I was designing for Windows 8.1, offering people in Starbucks a free coffee in return for their time.
I think this type of guerilla user testing is great when you’re starting out. But, after a while, you grow out of it.
To give a quick example, what if you want to talk to people who have had a very specific life experience? You can’t just hope to bump into them at your local coffee shop.
You don’t want to just test out prototypes on the same homogenous group of people in your coworking space. You’ve interviewed everyone there anyway.
If you want to make talking to users a regular occurrence, you’ll have to build out a process for regular sessions.
At the same time, recruiting a diverse set of people who have experiences that are relevant to the particular feature at hand is a time-consuming challenge. When I attended a Nielsen Norman Group training in London, they recommended outsourcing recruitment to an agency for this very reason.
How we Recruit
In our case, we mostly recruit our own candidates. We need to find a large pool of people who are willing to spend an hour with us in the office or remotely.
First, we asked in the Clue newsletter whether people would be willing to take part in our research group. In the form, we asked for their contact information, whether or not they could attend on-site in Berlin, and which languages they speak.
We then followed up with a second survey to qualify these people based on their experiences. We sent them a Typeform survey asking questions that asked for information such as:
- their medical history such as pregnancies;
- their relationship with Clue, such as how long they’ve used Clue and which device they use (Android or iOS);
- demographic information such as their age, relationship status, or children.
We wanted to identify people who’ve had certain experiences such as birth or usage of specific birth control methods for the purpose of user research and testing in respect of related features.
On the other hand, we also try to make sure to have as much diversity as possible so we avoid interviewing too many people who have had a similar experience because we aren’t looking for patterns in each group.
“We are looking for patterns that connects, idealistically, an infinite amount of variety. We want to increase diversity instead of segmenting diversity.” (Read more about the topic in Segments of one by Mike Lavigne).
We also occasionally recruit via an agency so we can find people who are not using Clue.
What about remote recruiting?
Our offices are based in Berlin, so if we only did in-house sessions, we’d obviously skew our findings to people based in Berlin.
Being able to interview and test prototypes in a remote setting means we’ll get a far more diverse set of people to talk to. In addition, not having to travel to our offices reduces the work involved for participants. We’ll also get access to cultures beyond those present in Berlin.
At the same time, it’s fair to say that you lose some information when interviewing someone over a Google Hangout or testing a prototype on a mobile device remotely.
You don’t see their body language as clearly. You’ll have technical issues that get in the way of focusing on the session.
We still do a lot of remote sessions, because new apps have mitigated some of these drawbacks. For example, we have used Marvel together with Lookback.io so we can record the sessions, hear user reactions and see their face as they interact with the prototype.
Scheduling the session
When you want to do it on an ongoing basis, rather than a once-off challenge, getting a smooth, repeatable process is critical.
Once a month, we send out an email to a batch of people inviting to choose a time via Calendly in the next four weeks.
People choose their own date for their session via Calendly. These confirmed sessions then show up in the product and design team’s calendars.
I put four placeholder slots on my calendar, so I make sure I don’t schedule other meetings at those times.
First, we set expectations. We write in an email a line saying, “We’re looking for feedback from people like you so we can improve Clue in the future.”
We provide exact instructions for how they’ll attend the session, be it remote or be it at the Clue HQ in Berlin.
A few days ahead we’ll email to ask if they can confirm that they’ve read the email. If they don’t reply we’ll follow up one day before the session.
To read what we do once we have someone in the office: read Part Two.
