Supercharge Your Agile UX Research with Informal Cognitive Walkthroughs
Part two: The External User Review
This is a two-part series on the Informal Cognitive Walkthrough (ICW), an Agile research method. To most quickly identify and address the majority of usability problems in your designs, see Part one on the Internal Product Team Review.
In the second phase of the ICW, we gain real-world context through an External User Review: a method that combines multiple parallel one-on-one observation sessions with focus group discussions. For the External User Review, we typically bring in five users per session and set up two back-to-back lab sessions — allowing us to talk to 10 customers in a single day.
To define the External User Review method, members of the Microsoft Intune team and I modified the Pluralistic Walkthrough, a usability testing method that involves adding representative users as participants in the cognitive walkthrough. Our approach differs from the traditional method most notably in the roles the product team members and usability experts play:
1. Because we always precede this session with an Internal Product Team Review, the feedback from the product team members and usability experts has already been integrated in the artifacts presented to the users. Thus, the External User Review focuses solely on learning from our study participants.
2. To run 5 one-on-one sessions in parallel, each participant is paired up with a product team member. Each pair runs through the major sections in the study at their own pace. This allows them to traverse the designs as they wish, as opposed to working from printed task scenarios (as is common in the Pluralistic Walkthrough) — which provides more natural feedback.
3. Our note-taking structure and group discussion method (described below) eliminate the need for post-session data analysis. They also guard against stakeholders overgeneralizing one or two points of view.
How to conduct the External User Review
By this point, you’ve identified the five product team members who will be pairing up with participants. The rest will sit behind the one-way mirror (or watch remotely).
1. Product team prep on study day
a. The researcher reminds the product team members that their primary job is to listen/observe and take notes. Next, walk through the discussion guide, so everyone knows their tasks and specific areas to focus on. Notetakers then go to their station within a usability lab. (There should be five stations with two people each.)
b. The researcher brings the participants into the lab and pairs them with the product team members who are already in the room. They then kick off the session with a quick round of introductions.
2. Run the 5 one-on-one observation sessions in parallel
a. Broken into the five individual pairs, participants think out loud as they complete the first set of tasks. Interviewers jot down their observations in a template prepared in advance.
b. After completing each task, the participants write down what they liked and didn’t like about the experience, and provide subjective experience ratings.
3. Transition to group discussion
a. After each main study section, the researcher asks participants to share their feedback out loud.
b. The researcher notes these pros and cons to the group, preferably on a whiteboard. The researcher then tallies how many users experienced each of the items listed. These notes become the working draft of the preliminary study report.
4. Repeat steps 2–3 as scheduled
a. For each one and a half-hour session, there is room for two or three larger end-to-end topics. Repeat steps 2–3 for each topic.
5. Post-session debriefs
a. Once the participants leave, the product team and researchers meet for 30 minutes to discuss insights from the session, and recommendations for next steps. These notes are added to the preliminary report.
b. When there are obvious problems to address, the designated team members make those changes. There are two hours in between sessions for design iterations.
6. Report on study insights
a. The researcher cleans up the notes from the day’s sessions, surfacing insights and recommendations in the preliminary report.
b. The researcher leads an hour-long presentation of the preliminary results the next day, and the team agrees on action items/next steps.
c. The researcher adds more context about the study (e.g., details about the methodology) and shares it with the broader team as a final report.
A note for our current COVID-19 situation
Instead of in-person lab study sessions, the same method can be run with remote moderated technologies (using software such as UserZoom, UserTesting, or online meeting technologies). Everyone joins a common conference call at preset times for the focus group discussions.
Maximize impact of ICWs with a regular cadence
Ideally, the entire ICW repeats once per month. This cadence allows all product team stakeholders to be present at both the Internal and External Reviews, which increases the decision-making speed and product’s impact.
No further analysis is needed beyond what’s built into the focus group discussions and debriefs. When we empirically tested the efficiency of this “during the session only” notetaking method, we found that it identified 85 percent of all the insights that post-session analysis of the data would provide, including the highest-priority insights. And because another ICW is coming up, you can continue to iterate.
With a regular cadence of ICWs in place to handle usability testing, the researcher will have more time to tackle foundational research questions. Furthermore, the product team’s empathy and trust in your work will make them more willing to shift the team’s direction based on your strategic research.
Note that, the ICW methodology is flexible, and can be modified to match your research needs. For example, I’ve run an ICW with as few as two participants (developers pair-programming) and as many as 80 participant-product team pairs (in a large event where customers visited the Microsoft campus). Similarly, while I find that one ICW per month is the perfect cadence for the long-term, I have run as many as three per week when a greater foundational understanding of our users was needed.
Are you interested in performing a complete ICW? Let us know your plans in the comments below, or tweet @MicrosoftRI or Valentina.
