A step-by-step guide to conducting design research analysis
A hyper detailed breakdown of what we do at Xero
Design analysis is the process of making sense of data collected from design research sessions. At Xero, we use design analysis to identify findings, patterns and themes that we can use to safely draw insights from, and make assumptions about participant behaviour.
At the end of a research session we’re surrounded by Post-It notes, transcripts, recordings and various scribblings, and our job is to take this beautiful chaos and organise it. We group it, re-group it and then label the groupings.
Why do we conduct design analysis, anyway?
There are a few reasons we conduct design analysis. One is to identify patterns and themes, but another is to help us check our own biases.
When you’re working in an environment with lots of like-minded people, all with a deep-rooted passion for what you’re building, your sentiment towards it may very well be clouded in partisanship. Similarly, you may think it’s easy to use, beautiful and intuitive, but your intended audience may not. Design analysis allows us to get rid of any biases, and build confidence about what actually happened in the session.
When we see findings that have commonality and impact, we can safely assume that they need to be considerations for the design. And because it’s a collaborative process, there’s a shared understanding among the team about why we chose the design solution. The way we conduct this analysis at Xero might be a little different than elsewhere, so I wanted to take this opportunity to run you through it step-by-step, in the hope it might offer you some wisdom in how you conduct your own.
Step 1: Prepare and organise your data
It’s easy to get muddled at the end of a lengthy, lively research session, so preparing and organising yourself before you begin collecting data is the key to success. Here are a few steps to help make analysis easier on yourself.
- Choose one Post-It colour per participant, with their name and a number, and put them all up on the wall
- Use questions and tasks from your script, or research objectives to create a note-taking template
- Take notes on behaviours (something they do, or don’t do) or attitudes (how they think, what they want, and how they approach your designs)
- Be sure to leave space for extra sections like ‘About’, and ‘Other’
Remember, keep the notes of what you write on each Post-It simple — it’ll make it much easier to group similar behaviours and attitudes later.
If you are working as a team, assign participants to team members who want to be involved in the analysis. It’s best if:
- they can listen, read and make notes for at least two different interviews (it’s useful to get different perspectives from different participants)
- more than one person reads and listens to each interview to broaden the analysis discussion
- you use note-taking templates to organise early
- you run a dedicated session for team members, spending 20 minutes presenting their participant’s notes to the group (this will help everyone get a better sense of feedback if they haven’t had a chance to listen to or read all of the interviews)
Step 2: Generate findings
At the end of these discussions, you’ll ultimately try to put a laser focus on the findings.
Your findings are the common behaviours and attitudes that popped up in the words, actions and sentiments of more than one participant — although a good general rule is that they ideally come up in at least three to make a solid case. Was someone particularly vocal, or passionate about their response? This also counts as a finding.
At this stage, you should have extensive Post-It notes for every participant. As you start to organise them, get rid of the duplicates first. Do this by finding the same colour Post-It’s and throwing away any that feel repetitive or unclear.
Next, look for categories within the larger grouping of notes. At this point, really clear themes should be emerging for each category. Maybe it was common feedback about where a button was placed? Or how obvious it was to get from one section to another? Maybe it was about something that felt clunky, and why — or feedback on what would make XYZ infinitely better. Give these groups headings that are really clear.
Remember, there will be other categories for things that aren’t part of the research goals or objectives. Identify and name these too. With a bit of fine tuning, it should look something like this:
Finally, find the larger themes that cut across your initial categories or screen groupings. What can you see across multiple categories? Bring those together again and name them (with the origin group clearly labelled for the purposes of backtracking).
Describe each grouping in a short paragraph, and use impactful quotes from participants that sum it up. Then, give yourself a massive pat on the back — congratulations, you’ve just generated findings for your study!
Step 3: Form meaningful insights
Insights come from your analysis and interpretation of research findings. They point towards major behaviours and attitudes that have surfaced, and explain why that attitude or behaviour exists. Often, they are a particularly significant finding (or group of findings) that have uncovered something important for the product, and your understanding of how users might approach things.
Rule number one of forming an insight: they must be actionable. There has to be a clear pathway from the insight itself to how it will be used to inform product and design. If that doesn’t exist, it’s worth noting down for context but it’s not an insight.
I recommend writing down your insights using this formula: this <behaviour/attitude we saw> was because <the reason why participants did or said what they did>, which <the impact it had on their experience>.
If needed, add additional detail to support it, such as examples or smaller findings that didn’t warrant their own insight.
Some insights will show a deep relationship to one another, and will often come with lots of detail about why this occurred. In some ways, this is a ‘super insight’ — a clear signal that something must be considered for the final product. Always look for the profound ‘why’ behind everything and what impact that would have for participants.
Some good questions to ask yourself while doing this:
- Which insights seem more important than others?
- How can I order these by relevance to the design learning objectives?
- What was really interesting that I wasn’t expecting to find?
- Where did my expectations match up with reality?
- Is there any connection between the findings and previous design research, support tickets or analytics?
- What do I feel really confident about? On the other hand, what’s in contradiction where I can do some further exploration?
With all of this legwork, you might like to turn it into a final report with your approach, aims, participants, an executive summary, and next steps.
Once your design analysis is finished
Ultimately, design analysis is crucial to making sure we’re acting upon the findings and insights that are most significant to the design — not just the last thing we remember, or what our confirmation bias might tell us is most important.
Hopefully this resource will act as a great starting point for you and your team to make some powerful interpretations of what to keep and what to change, based on concrete insights and findings. Remember, the whole process can take anywhere from a day (for rapid, lighter analysis) to a week (for in-depth analysis with a lot of data).
If you run design research sessions, I would love to hear what your key takeaways are from this or what you might implement that you haven’t before. Pop your feedback as a comment below, I’d love to hear from you.
