Research Team of One: How DonorsChoose.org Learns More With Less
Here’s a familiar scene: You’re a product researcher gearing up for a usability test. You draft a research plan, get feedback from stakeholders, and finalize edits. Your recruiter finds and schedules six participants, all of whom, per your specifications, have made at least 2 purchases on their mobile phones in the past month. 10 days later, you’re running a participant through some tasks on a prototype. Your research partner is sitting behind the one-way glass in the lab, taking notes on the session and operating the recording tech. In a week, you’ve analyzed your data and summed up the findings and recommendations in a comprehensive report that will be presented to the product team.
Here’s the thing, though. There is no lab. There is no recruiter. And there aren’t any other researchers. At DonorsChoose.org, a crowdfunding charity that helps teachers get the classroom resources they need, we follow a lean and nimble doctrine. But that doesn’t mean we have to sacrifice rigor in our research. In fact, we test just about everything we build to make sure our teachers and donors can accomplish their goals on our site.
At DonorsChoose.org, a crowdfunding charity that helps teachers get the classroom resources they need, we follow a lean and nimble doctrine. But that doesn’t mean we have to sacrifice rigor in our research.
What Research Requires
Let’s first talk about the steps that need to happen when we conduct usability testing at DonorsChoose.org. The basic research workflow goes something like this:
- Identify research questions
- Recruit participants
- Write a moderator’s guide
- Hold the sessions with participants / Collect data
- Analyze the data
- Report out the findings
Some of these steps require more time, while others require more effort. Some, like recruiting, take a lot of time and a lot of effort. Some steps happen in tandem, while others only begin when another is finished. So, how do we manage it all?
If you, like us, don’t have the luxuries of a dedicated lab space, recruiting services on retainer, or multiple researchers, read on. Here are some ways our one-person research team spends less time on logistics and more time on research. We’ll focus on the areas where we’ve had the biggest wins: recruiting, conducting sessions and collecting data, and reporting findings.
Recruiting the Right Participants
Finding and scheduling participants for your research is resource-intensive. Some people mitigate this by practicing “guerrilla”, aka “coffee shop”, research. In addition to making sure our designs are usable, we’re often trying to validate that we’ve designed the right solution for teachers and donors, so taking our chances at the nearest Starbucks isn’t a great option. We want to be exact in our targeting while striking a balance that requires less work than traditional recruiting strategies.
Our process starts with our data, made available to everyone at at DonorsChoose.org thanks to our brilliant Data Science team. Once we identify our recruiting criteria (e.g., teachers who’ve posted projects on our site in the past month), it’s fairly simple to pull a random sample of our users based on that criteria. We’ll email a short survey to this group asking them to indicate their interest in participation as well as their availability. Interested and eligible parties get a Calendly link to our calendar so they can schedule themselves. When a study is closed, we track the users we emailed and those who participated. That data gets uploaded back to our database so that we don’t reach out to the same folks when the next project rolls around.
The process isn’t seamless, and it still takes some effort, but it cuts out the the manual back and forth emails while maintaining a high standard.
No Lab? No problem.
We run most of our moderated usability testing remotely, using Validately, and have identified three key advantages over in-person research.
- The DonorsChoose.org mission is to “move us closer to a nation where students in every community have the tools and experiences they need for a great education.” If our teachers and students are distributed, our research needs to be too, so we use Validately to connect with teachers and donors all over the country, in a more natural context than a lab provides.
- Our users are often busy teachers who, by virtue of using our site, are already going the extra mile to get resources for their students. Conducting research remotely respects their time, reduces our no-show rate, and requires less of an incentive.
- As an added benefit, Validately (and other tools) have built-in recording functionality, removing the need for a camera, software, and a human to operate it all. Have I mentioned that I’m only one person?
Collecting Good Data
Our research is only as good as the data we collect. That said, when I’m moderating a session and fielding questions from observers, it’s easy to miss things in the session or to take sloppy notes.
I picked up a surprisingly elegant solution to this problem from David Juhlin during our time working together at the User Experience Center. Rather than resort to a combination of faulty memories and handwritten scribbles, I use a simple checklist to capture data as a participant progresses through a task. I still do plenty of scribbling, but there’s a forcing function ensuring that I capture the right data every session.
Having an easy way to track these behaviors in real time provides my team with structured and quantifiable findings with less of the hassle, reducing bias and making it easy to prioritize changes to the designs.
Document, Yes. Report, No.
While we do practice lightweight documentation for posterity’s sake, our team tends to favor quick debriefs over lengthy slide decks or reports. This usually takes the form of a quick conversation with a designer soon after a study ends, followed by a larger conversation with a wider team of stakeholders. Our focus is on identifying actionable insights quickly so that we can iterate and test again.This type of reporting is collaborative in nature, and is why we insist that all stakeholders watch the research sessions. Which brings us to…
Involve your stakeholders
If you’re going to be lean, it’s a good idea to find some people to lean on. I’m lucky to work with an amazing team of designers, product managers, and other stakeholders across our organization who get involved in the research process. They review the scripts that I write up, sit in on sessions when they can (or at the very least watch them later), and sometimes help out with note-taking and data collection.
Why This Stuff Matters
I’ve worked at companies where I was fortunate to have access to full-time recruiters, dedicated lab space with fancy observation rooms, and a team of researchers helping each other out. There are a ton of great benefits to these kinds of environments and, if you have the resources, I highly recommend investing in recruiters and additional team members. You’ll be able to do more research faster, which I’ve heard is the goal.
At DonorsChoose.org, I try to achieve this goal by gaining efficiencies when I don’t think I’ll need to sacrifice the quality of the data. I’ve at times gone too far in that direction, trying to speed up a recruit so much that the participants didn’t fit a particular study, rendering the findings less than useful. When that’s happened, I’ve had to take a step back and accept that it might take me a bit longer to find the right participants, so that I can collect the data that will drive a project forward.
Overall, working for a scrappy non-profit has shown me that you can still do meaningful research without a massive budget. If you’ve found other tips or methods that help you pursue similar goals, let me know! I’m always looking for ways to be more effective in less time, while maintaining a high standard of rigor.
P.S. One last plug for my awesome teammates who gave me feedback on this piece and helped steer me in the right direction.
