Rolling Research: Keep the Insights Coming

Quick Answers to Lightweight Questions

Rolling research is a process based on ongoing, regularly scheduled research sessions. It’s a lightweight, flexible way to establish a steady stream of answers to your organization’s latest usability questions. Because it happens on a predetermined schedule, uses the same basic methods every time, and can cover a variety of topics and products, it’s more efficient to implement than conventional session-by-session research.

The session cadence can be weekly, monthly, or anywhere in between. And because rolling research is limited to evaluative questions, you can get it up and running fast, often within a week or two. In addition to the obvious benefits of frequently talking with and learning from customers, rolling research can also provide insight on topics that might otherwise have to wait for greater researcher availability — or might never get the research attention they need.

Here at Facebook, insights gained from rolling research have helped guide product development across many of our teams. For example, this past fall, when we were creating our Jobs product, the Jobs team quickly tested an early version of their entire job-posting flow with five admins of small and medium-sized businesses ( SMBs) through our Pages Rolling Research program. The session enabled the team to make usability improvements to the admin-facing flow, and to better tailor the tool to the specific needs of SMBs. Because of the lightweight process of submitting products for testing, the Jobs team received user feedback early in the design process, when they otherwise would not have.

What it can and can’t do

Rolling research is a quick, flexible way to increase exposure, insights, and collaboration. It works best as an additional channel for feedback on existing stimuli — not as a replacement for generative, foundational research. Questions beyond usability increase the cognitive load on participants, and a rolling research moderator typically doesn’t have enough time for exhaustive probing.

The clearer your questions are, the better rolling research will work. If you’re not looking to answer a clearly defined question, such as “what tweaks need to be made in order to launch this feature,” it won’t be very useful. Also, because rolling research is designed to cover multiple (often unrelated) topics, it works best for general population insights, not for exploring niche segments.

Who can implement it?

Whether you’re a seasoned professional or new to user experience research, you can conduct rolling research. The process has been successfully used by individual researchers, groups of researchers working on similar products, whole research organizations, and even by teams that lack a researcher.

• Solo researchers: It may feel daunting, but setting up a rolling research program helps you scale yourself by operationalizing research logistics.
• Teams of researchers: Collaborating on a rolling research program exposes each member of a team to different product areas, enabling you to observe, critique, and learn from one another.
• Teams without research: If you don’t have a researcher on your team, a rolling research program can serve as a band-aid. This channel will help you get signal on your product or designs — and help make the case for more exhaustive, rigorous research.

Promoting research

Beyond the fundamental benefits of a continuous source of user insights, rolling research can also demonstrate the value of research throughout your organization.

• Demonstrate deep user-base expertise. Each time you run a round of rolling research, you’re building on existing knowledge about your users. By socializing these studies, you’re conveying to your organization that you and your team are committed to understanding your users.
• Dispel research myths. Do you have a resistant stakeholder who complains that research is a bottleneck or just takes too long? Rolling research can provide another channel for feedback, which can help diminish those perceptions.
• Represent the research-deprived. Rolling research can serve anyone in your organization, including teams that don’t usually have access to research. When you add a question from one of them to a rolling research session, you don’t just help them get the answers they need. You also demonstrate the value of research — and might even help them make the case for adding researchers to their team.

A few best practices

Rolling research can be established quickly, but the more care you take in setting it up beforehand, the more efficient your efforts will be. While you can — and should — make adjustments after the program is in place, establishing a consistent, predictable process up front will help stakeholders understand what you’re doing and encourage them to take part.

Recruiting:

• If your sessions will cover research topics from different product teams, look for themes across all the groups and find a base participant that will work for all of them.
• Keep recruiting simple to ensure consistent execution. Consider friends and family, super users, ask biz dev or sales to invite participants, or use a research recruiting firm.
• Your goal should be to interview 5–6 participants over 1–2 days.

Session prep:

• Keep testing lightweight, limiting it to participant background and focused workflow questions, concept evaluation (design comparisons), and usability tests.
• Limit each hour-long session to 3 products or concepts. More can be overwhelming.
• Educate and evangelize relentlessly. Develop a rolling research elevator pitch — and don’t hesitate to use it. Brand your program and advertise it often. Put sessions on the group calendar, along with purpose of that day’s session, to encourage attendance or viewing.

Session:

• If you have the resources, use a dedicated research room as well as a separate observation room so stakeholders can keep engaged. This fosters stakeholder empathy with users, in-the-moment feedback, brainstorming, and real-time iteration.
• If resources are more constrained, use a streaming service like GoToMeeting or Join.Me (which is free) to give stakeholders a window into the session. You can also use real-time chat to stay connected.
• Record the sessions if possible for later viewing.

Post-session:

• Debriefs are essential but don’t have to be exhaustive. Focus on major themes and time-sensitive questions stakeholders flagged before the sessions. Debriefs can happen after each participant leaves or within a day or two after sessions. Debrief in person, by chat, or with whatever method fits the setup of the sessions.
• In addition to the debrief, make sure you’re documenting the insights. Keep it lightweight (bullet points in an email, for example). The point is to make sure that your research lives on after the sessions end. As with any research, the most impactful insights include 3 pieces: a direct connection between insights and the goals stakeholders have set, video clips or quotes from the session, and recommendations — strategy and design next steps.
• Follow up about recommendations in lightweight ways like checking in by email or message, or posting in groups.

Rolling research is all about repeatability. Throughout the process, look for ways to reduce friction and become even more efficient, such as using the same screener, booking the lab space in advance, and reusing key questions.

Ready to roll

If your organization could benefit from a continuous source of quick evaluative insights, we encourage you to give rolling research a shot — and to stick with it even if its value isn’t obvious right out of the gate. The benefits of rolling research have a way of snowballing over time as you streamline the process and continue to build stakeholder awareness and buy-in.

After you’ve completed your first round of rolling research, take some time to assess what worked and what didn’t with your team. (And if you can spare a minute, let us know how it went in the comments section below!) Inevitably, you’ll find ways to tailor the process to better suit your organization’s needs.

After a few rounds you might find yourself running a well-oiled research machine.

Authors: ‎Jamie Kimmel‎, Researcher at Facebook; Akilah Bledsoe, Research Program Manager at Facebook; and Beth Lingard Research Manager at Facebook (from left to right)