A story of running research in a Growth team
Sharing stories is an integral part of how we learn. This story is about how we worked together to conduct an extensive research study in one of the growth teams.
In our growth team, sometimes designers run research projects, but we are always supported by our amazing research ops team and in parallel to the fantastic research team here at Atlassian.
⚠️ This is by no means a comprehensive guide of how you should conduct a research study but instead some learning moments and techniques.
Our story starts in…
In late February this year, one of our Product Designers did some concept testing to determine what our customers thought of three approaches. She reached out to me, a Studio Designer. I help other designers across various projects.
We followed a tried and tested process: Plan > Recruit > Prep > Conduct > Analyse > Synthesise to ensure we got the most out of our study. By using this process, we were careful not to miss anything critical as the learnings and structures of previous studies are built in. It’s harder to undo something when you’ve already started interviewing participants.
We used a research plan to ensure we were asking the right questions from different perspectives — product, engineering, and analysis. We did this by holding working sessions with the team I was working with. This allowed us to work together to write the important questions. This allowed for a well-rounded view of the qualitative data we were about to collect.
📑 Templates were pivotal to structuring our study; without them, we wouldn’t have been able to move quickly or accurately. They helped us standardise our outputs and provide a cheatsheet to kickstart our work.
We used Confluence, due to its centralised page structure and collaboration features, it allowed our team to collaborate in real-time, but you can use other tools with similar features.
Planning
We filled out the research plan, which gave a good base to work from when the team and I collaborated. We focused on the Why? and the What? to ensure the qualitative data could be used to make critical decisions across the project lifecycle.
Objective, outcomes, and outputs
One part of planning across projects is focusing on objectives, outcomes, and outputs or “ooo… (yeah)”, as I like to call it. It helped us understand what this study would help us achieve (objective), what we want to learn (outcome), and what data/materials we needed (outputs).
Consider outlining the three O’s with your team before you start the research project.
Research questions
We separated our research questions into two main areas:
ℹ️ What’s our overarching research question? e.g. what is one high-level question that this study will answer?
ℹ️ What are our key questions? e.g., lower-level questions that will help answer the leading question or validate a hypothesis.
We grouped and classified our key questions into four areas:
- Access
- Permissions
- Use-cases
- Other
Once we refined the questions we wanted to be answered, we were able to understand the type of research we’d be conducting and who we were going to interview.
❓ Consider having an overarching question that structures the main topic of qualitative data to be captured. Also, try to structure the other questions around secondary topics.
Building a list of assumptions
An essential part of our research process is to build a table of assumptions or hypotheses. This allows us to test and measure assumptions we had about the designs and attitudes of our users. Generally, these assumptions are created during the Discovery or Definition phase through an empathy or journey mapping exercise. After each interview, we reviewed these assumptions and reaffirmed whether they were believed to be true, false, or inconclusive.
Adding a list of hypotheses or assumptions can enrich your research and uncover how well you think you know your users.
Screening
Screening participants is an art in itself, especially when trying to find specific types of users.
Ways to structure screening questions:
- determine if interviewees use your product through a list of options, then ask them to give you an example.
- allow the initial screener questions to appeal to more people, then have the questions become more detailed to ensure you’re getting the correct type of participant.
Think of your screener as a funnel; the deeper your potential participants go, the more specific your questions become.
We got help from our analysts to clarify who our participants needed to be and why we needed to target them. We separated our research into two separate screeners to find:
Group A
- At least 5 of the people who [do this type of activity]
Group B
- At least 5 of the people who [do this other activity]
We did this because we were trying to understand the perspectives of two different user groups that might work together but are from separate companies.
Generally, try to interview six to eight people per cohort but as low as five can be enough. You can read more about this specific number in this Nielsen Norman article.
We separated the participants into three further buckets: 🟨 LIGHT 🟥 DEEP 🟦 BROAD:
This gave us meaningful ways to categorize our participants and ensured we were holistic in our research.
For big projects, find ways to group your participants so that they stay connected to what you’re trying to learn.
Organizing interviews
We were interviewing participants in many different time zones. We created four daily sessions (morning, lunchtime, afternoon, and evening) based on the largest group of interviewees who were located on the west coast of the USA (PST). We used the Australia timezone to our advantage by having AUS Mondays (most people’s Sundays) to cater to those participants who were busy during the weekdays.
We used P + [a number] to label and anonymize each participant's details. You’ll see an example of this later in the article. Keeping track of information and data became much easier for us this way too.
As we got participants booked in, our scheduling page became our source of truth and allowed me to keep on top of organizing and managing our participants and the data we captured. It became my source of truth until we finished conducting and synthesizing the qualitative data.
Conducting
It can be challenging to only have a limited amount of time with someone in which to get them to talk about their experiences and views or even get them to understand a feature or concept. We brought visual cues (or concepts) and asked them to show us how they do things, to solidify the conversation and get on the same page as the participant.
Designer using a visual cue “This is what I mean.”
Participant seeing visual cue “Oh yeah, I get it now.”
In previous studies, our notetaker was there to write and capture notes throughout the sessions. As we moved towards using Dovetail to help us highlight transcriptions, we noticed that we were doubling up.
This transition into Dovetail marked a more effective way of working and meant the notetaker was less focused on writing everything, but instead could focus more on the key takeaways, observations, observations, and pain points.
Organizing your sessions
I prefer to have two sessions, before and after each interview — a session to prepare for the interview and another to talk to any observers to gather further observations.
Prep for interview
Generally, I leave 15 minutes to prepare for the interview — to go through the participant screener and any other data. I then send a quick Slack message to all the observers:
Example:
Reviewing interview
Post-interview we take some time to unpack the session with everyone who was involved. We discuss any key takeaways, pain points, gains, and opportunities or ideas.
Synthesizing
We used Dovetail to handle all research data; we wanted to move towards using a single tool to manage all our research data so that at some point, we can use it as a library. This involves getting the right fields and using consistent tags, which can be done using Dovetail’s extensions feature across all our projects.
Tagging everything, everywhere, all at once
This took us the longest time to finish as each session was about an hour and a half, sometimes a little longer as we packed in many questions and concepts. It’s been my most extensive study in a long time. We need to do the hard work upfront and ensure we tagged every interview appropriately to achieve our dream of having a research library at our fingertips. This approach involved tagging: the type of feedback (observation, expectation, pain point, positive, opportunity, etc.), what part of the experience the feedback was for, and what tools they were referring to. We also used this time to capture feedback based on the project and the designs. This way, it would be easy to go back to specific screens and see what our users mentioned.
Two main axes of highlighting:
- What kind of feedback is this? e.g. positive, negative, opportunity, etc.
- Where does this feedback belong to? e.g. part of a concept, strongest concept, usability, IA, content, answering a key question, etc.
We then shared a page of links to specific tags in Dovetail with the rest of the team. This approach allowed them to look at videos that captured qualitative data about different areas the project was concerned with. e.g. the individual pillars of the project, opportunities, the strongest concept, design feedback, etc.
Dovetail tags & extension
During this process, Anna defined a way to categorize a logical and useful set of tags. We then refined those tags after we ran through a couple of studies, which looked to improve uptake and reduce disorientation. We worked towards creating an extension where the whole growth team can share the same tags and fields, thus moving towards a more standardized way of categorizing and accessing data.
Turning tags into insights
At this point, we tagged 11 videos into 800 highlights. 800? Yes, that’s right! We wanted to be very thorough to inch closer to our dream of having a better research library. How do we move into insights now? We were looking forward to using Dovetail’s new views to help us affinity map, but those features weren’t released when we were synthesizing our study. I used a different method, where I created broad insights that were refined over time. This was hard to keep track of but allowed us to move highlighted data around quickly. The analogy I used is going from large buckets to smaller and smaller buckets until we arrived at a specific insight.
With nearly every insight, I added a “How might we” statement to help.
e.g. “How might we allow these distinctive concepts to scale and flex as our users’ needs change throughout the lifecycle of a project?”
Help your insights become actionable pieces of information by adding a How Might We (HMW) statements.
You can learn more about HMWs from this article by Ideo.
Dovetail became a powerful tool because it reduced the gap between the insight and the source.
Insights just for designers
We also used the insight feature for design-related feedback (opportunities, feedback, IA, etc.) to group information together to go back to when we were designing concepts.
Reporting
We structured our report around answering the research questions and covering learnings about the concepts we showed participants. We split the work up. I formed the general insights while my colleague looked at what we learned with each concept, and another colleague helped fill in the remaining insights.
What makes this report different from the others we did? We used (one or two) videos to cover every question and angle of our research. This method bridged the context gap; we didn’t have to show screenshots or quotes.
A video of a quote or highlight from an interview tends to be a better method of giving context to a reader than a quote or a screenshot.
Consumption convenience
We’re conscious of how each person prefers to consume information. Some of us like to read, others prefer to watch, and some want a summary. To ensure we could surface this research to a broad audience in our team.
- We held a share back of the report (which was recorded) — this became our way of watching it
- The report we put together was on the Confluence page — this became our way of reading it
- And finally, Nadine created a presentation — this became our way of summarising it
Based on the scope and audience of each research study you conduct, you can gauge how varied your outputs need to be.
Example:
What stories or takeaways do you have when you’ve run a research study?
Stats
- we interviewed 11 participants for 1:30mins each, from two different cohorts and across six different time zones
- we tagged over 800 data points in Dovetail
- we analyzed qualitative data across 27 screens and four hypotheses in Figjam
- we synthesized 27 insights in Dovetail that created one report in Confluence, a Zoom video, and a presentation on Google Slides.
