Published in


Pilot, iterate, repeat

Attention UX designers, this is your pilot speaking: you need to iterate on your user research activities. Also, your cheesy jokes.

Therese piloting our sketching activity with one of our Capstone classmates, Ming

Last week, Aditi wrote about the prototyping mentality and the value in iterating not only on your product, but on your design process, too.

Over the past couple weeks, I’ve been working on both our usability test and user interview plans. At the same time, we’ve also been putting these activities in front of real people as a group to fine-tune them.

Each time we pilot, we learn from our participants about how we can make the activities clearer so that we can get the richest possible data in our user research next week.

A case study in piloting: sketching search patterns

As part of our user research, we are conducting interviews with participants to get a sense of their information search routines online. We wanted to pair each interview with an activity so that in addition to reflecting on past information-seeking and use experiences, our participants could also show us their search process. (NN Group warns that interviews are limited in their usefulness as a user research method — showing is better than telling.)

Pulling from the IDEO Field Guide to Human-Design, we decided to try a sketching activity similar to their “Draw It” activity (which you can find on page 65). Users are provided with paper and markers and given a scenario wherein they would need to search for various types of information. They are then asked to sketch out how they would find that information. Here’s what we got in our first pilot:

Our participant’s sketch from our first pilot — while she did a great job of mapping out her process, it was difficult to get her to actually draw.

This didn’t work. Therese suggested that the scenario needed to be more clearly defined in order to give users the details they need to visualize how they might respond to it. She also noted the participant wasn’t sure where to start with sketching and had to continue to be reminded to pick up the pen. The blank sheet of paper was too intimidating and out-of-the-box for our half-hour sessions.

The storyboarding worksheet we piloted.

Back to the drawing board. I enriched the scenario with greater detail and narrowed down the ask for our participants to make the activity clearer and more concise. At Therese’s suggestion, I also reframed the activity as a storyboard rather than a sketch. I quickly threw together a storyboard worksheet for participants to try to combat blank page anxiety.

We piloted again, and our users were still not able to visualize their search process. They understood better that we wanted drawings because we provided the worksheet, but they asked: “what is a storyboard?” And they kept coming back to the same conclusion, which was also echoed in our first pilot: “I would use Google. How do I draw that?”

All in all, putting this activity together and piloting it took just a few hours of the team’s time — and we learned a lot from the exercise. We learned that sketching may not be the most well-suited activity for getting users to visualize navigating information online. (To be clear though, more research would be needed to validate whether that’s a solid rule, so I’d encourage folks to try this kind of research with more users!)

Sian Townsend, director of research at Intercom, suggests on their blog that sketching is better suited for digital products that involve:

“Spatial relationships (e.g. maps, local search)
People and relationships between people (e.g. Facebook, Snapchat)
Time and Sequences (e.g. Path, Moves)
Complex systems (e.g. Health insurance, Salesforce).”

We wanted to use the sketching activity to understand our users’ mental models for searching for information online—instead, we are now getting users to show us how they search for information using a “show and tell” activity, inspired by Work Differently (all methods cards, page 9).

This activity has proven effective with piloting as we get to see in real time how people search for information. By encouraging users to think out loud, we are able to parse their thought processes and gain an understanding of their mental models as they navigate their search.

So why pilot? Sure, our first activity didn’t work — but we learned a lot from piloting and trying something new, at a minimal cost in terms of resources. For future, we know that sketching may not work for this kind of research question. For this project, we have validation that the activity we landed on for our research is going to provide high returns in the form of useful insights for our product development going forward.



Aditi, Therese & Danielle are UX students in Toronto finishing our Master’s with a capstone project in partnership with the MaRS Discovery District, reimagining the Entrepreneur’s Toolkit. We’re using Medium to capture our design process and all the lessons learned along the way.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store