Supercharge Your Agile UX Research with Informal Cognitive Walkthroughs

Part one: The Internal Product Team Review

Do you feel like you’re spending too much time on tactical research, and have too little left for strategic research? If so, and you work in close support of a single product over time, let me introduce you to your new best friend: the Informal Cognitive Walkthrough (ICW).

Over the past decade, I’ve bootstrapped over a dozen large enterprise software development teams with this Agile UX research method. For researchers who are embedded in product teams, ICWs offer a way to significantly reduce the considerable time it would take to plan, conduct, and present results of a traditional usability study. All this while still talking to the same number of customers, having confidence in the accuracy of your results, and having more impact than ever with the product team.

I co-created the original version of the ICW method with members of the Microsoft Intune team in 2009, when Agile UX research was in its infancy. Over the years, we’ve applied it at Microsoft to products for IT professionals, developers, and consumers. I still use it on all three products I cover in my current research, in Microsoft’s Mobile Experiences space. While I tweak the method for every specific context, it’s remained unchanged at its core. In honor of the tenth anniversary of the ICW, I want to share out a high-level view of what it is and how to conduct one.

The ICW consists of two parts: the Internal Product Team Review (discussed here) and the External User Review (which I’ll describe in part two of this series).

Conducting an Internal Product Team Review

To create the Internal Product Team Review, we optimized the Streamlined Cognitive Walkthrough, which measures usability by having the product team simulate a user’s problem-solving process. This approach has many strengths, encouraging the team to take a UX orientation early in the design process and work out key usability issues internally. Through trial and error, we’ve increased its efficiency, honing it for Agile.

One important change we made is requiring attendance of all stakeholders working on a given scenario. If we don’t have a quorum, we don’t run the review. This allows the team to make decisions about what changes to implement as part of the walkthrough, which saves time and ensures impact. Another change we made was to eliminate training of the product team. That step, which includes getting the ground rules and prepping the team to take on various roles, originally took 20 minutes of each 90-minute review. To make the review work without the training, we’ve increased the responsibilities of the researcher, as described in the step-by-step guide below.

The researcher (left) takes the role of customer during an Internal Product Team Review.

(*For brevity, some preparatory steps have been omitted.)

1. Set up an internal review of your team’s artifacts with the product team members: The product team members kick off the engagement by proposing a scenario for customer feedback (in the form of user goals) and provide related artifacts. The artifacts can be at any stage of the design process — implemented code, high-fidelity interactive prototypes, low-fidelity designs, or even a workflow of the steps you plan to design.

2. Assemble key stakeholders: We immediately invite all stakeholders working on that scenario to the ICW. This reduces the need for video clips and reporting later; stakeholders will have the context they need to make decisions if they’re present. It also fosters collaboration among disciplines, an important tenet of Agile.

3. Run the walkthrough: We typically schedule 1.5 hours for the walkthrough, enough time to cover three larger topics. The researcher takes the role of the customer. They announce the user goals and walk through the individual steps to accomplish them, taking a golden-path approach (the ideal approach as envisioned by the team) for each goal.

• The researcher answers two basic questions: 1. As the user, would I know what to do at this step? 2. If I do the right thing, as the user, do I know I’ve made progress toward my goal?
• After each step, the team shares observations on the customer experience and makes decisions on the spot about what changes are needed. The researcher monitors discussion, bringing up ground rules like “no defending your designs” only when they’re broken. The researcher takes notes on key observations and action items.

4. Share notes: Notes are the only form of documentation. The researcher sends them out quickly after the walkthrough, specifying who will make what changes before the next session, to improve the experience.

5. Repeat as needed: In the context of an ICW, we typically run two Internal Product Team Reviews per feature area: one kickoff meeting with designs as they are and another with the exact discussion guide and designs we’ll be showing users in part 2.

Impact of the Internal Product Team Review

When we first began experimenting with Agile, I wondered how many usability issues a product team would catch during a single Internal Product Team Review as compared to a traditional usability study. The answer, based on our research, was 80 percent—in only a small fraction of the time, and with those issues immediately being addressed by the product team.

Thus, these are the key insights I’d like to leave you with:

1. Do not skip the Internal Product Team Review. You’d also be skipping an activity that can immediately improve the product and build team cohesion. The review allows you to not waste users’ time finding out what the team already knows.
2. Only hold the review when all the main stakeholders can be present, so you can immediately and rapidly iterate on the uncovered issues.
3. Deprioritize report writing — prioritize changes to your product or artifacts. A follow-up email with key insights and action items is all you need.

There will, of course, be important findings in that last 20 percent of insights that teams won’t discover on their own. Not to mention, the empathy-building piece of customer interaction only happens with real participants in the room. So we added a second component to the ICW that allows us to address real-world scenarios in an Agile way.

Are you interested in performing a complete ICW? Stay tuned for part two of this series. Thoughts so far? We’d love to hear them. Tweet us @MicrosoftRI or comment below.