Guerrilla research
Going beyond A/B testing to see our email designs at work
People are a critical part of our development process, with their input guiding all stages of our product development. Several Twitter features are the result of people creatively hacking the platform for a specific function, e.g. Tweet threads, hashtags, @usernames. Along with observing behavior on Twitter, we use qualitative research, on-platform surveys, and experimentation to better understand how people use Twitter, and how we can better meet their needs.
Typically, several of these research techniques are used to iteratively learn what product solution we need to build and validate if what we’re building is on the right track. However, constraints sometimes prevent us from sticking to our ideal development process. And although these constraints can be seen purely as restrictions that dictate how work will get done, we find they encourage innovative thinking. When time, budget, or other resources are not on our side, we think creatively about how we can work around them to build high-quality, people-centered solutions.
Within Research, we have a few different methods to consider when we need to get insights that were required yesterday. Rather than turn away teams eager for feedback, we have a conversation about the pros and cons of different research approaches, eventually finding a path forward that is inclusive of customer feedback.
We’re sharing a recent example of how we addressed one of these situations, in the hopes that you’ll be encouraged to think creatively about how to include people in your product development process.
Our team was interested in learning why people engage with our emails, specifically the Highlights email, a selection of Tweets from their network. Though metrics told us a lot about email opens and clicks, it didn’t tell us why people opened it or why they clicked into Tweets and accounts in the emails.
- Did the subject line catch their eye?
- Were the people mentioned important to them?
- Why did they click if we already show them Tweets in the email directly?
- Do they just open all Twitter emails out of habit?
We needed user research to complement our experiment findings, give us insight into people’s motivations, and help develop new hypotheses we could continue to test through experiments. However, the team was also trying to move fast — at the time, we were beginning 2018 planning and wanted to understand and prioritize the areas we should be addressing to improve the email experience.
Given the time constraints we were dealing with, we discussed the possibility of conducting guerrilla research, specifically a variation of intercept interviews — basically walking up to people, showing them our prototype, and asking for their feedback — to get directional insights. We wanted to catch them quickly, out in the world, and let them share their thoughts, based on a few questions, in less than 10 minutes. Unlike some other qualitative research techniques, this method saved us time by skipping screener emails and spending a short amount of time with participants.
There were also limitations that we considered when we were deciding if this rapid research was the right approach:
- We wouldn’t get an understanding of the participants’ backgrounds.
- We weren’t going to be able to build a meaningful rapport.
- We wouldn’t have time to follow up on their responses.
Since we planned to validate our findings at scale through experimentation, we decided to focus on speed and lightweight investigation, and thought guerrilla research would be the best tool to get quick feedback and answers to our key questions.
We put together a plan for our team of designers, product managers, and engineers to collect feedback on the Highlights email. We drafted an interview guide based on our goals, and walked the team through some tips for reaching out to people: have a compelling opening line, be respectful of people’s time when you interview them, and let participants know their feedback will contribute to the Twitter email experience.
We decided the Ferry Building in San Francisco would be the perfect place to conduct our research.
- Bustling place filled with locals and tourists? ✅
- People from different backgrounds and ages? ✅
- Delicious post-research lunch options to bribe your team with? ✅
We set off for the Ferry Building with interview guides, notebooks, mobile prototypes, and gift cards for people as compensation. We split into six pairs, an interviewer and note-taker, to approach folks willing to answer a few questions about our Twitter email prototype. In order for people to understand we worked for Twitter, we also wore some Twitter t-shirts.
We wanted to get a range of perspectives, so we spoke with a diverse group of participants, including two young sisters vacationing from Australia, a gentleman who wasn’t fond of small fonts, and woman who was killing time until she could catch a bus to Santa Rosa.
Watching people interact with your product is its own reward, but our team also appreciated the lunch break halfway through the day so we could debrief and re-strategize for the rest of the day. Between mouthfuls of burgers, we shared what we’d learned about our participants in such a short time. We learned about the importance of relevant subject lines and of personalized content within the email, and noticed the different ways people would interact with the email.
The day after our research outing, the team gathered to compile what we learned while it was still fresh in our minds. Each team member wrote an observation on a sticky note, and then presented what they learned to the group. We noticed patterns in how people consumed the email content and grouped the notes into categories. This led to an experiment plan where we could see if our findings scaled to a larger set of Highlights email recipients.
Here are some of our team’s takeaways from the field:
- Guerrilla research helped us validate or disprove our initial hypotheses quickly while also learning new ideas to test. For example, within the Highlights email, we hypothesized that people tap on the first few Tweets in the email because they’re seen first, and then decide if they should continue reading the entire email. However, we noticed participants curiously scrolled to read the entire email first before deciding which Tweet to tap on.
- The qualitative feedback from this study combined with our previous quantitative learnings identified high-level opportunities for us. This helped determine the roadmap for the upcoming quarters. One theme that emerged was finding an easier, faster way for people to login. We made logins from email a top priority that quarter and let people login through SMS to help transition them from their inbox to Twitter more seamlessly.
- Getting all team members involved helped develop greater empathy for the people using Twitter. And the team is even more invested in the decisions going forward. It’s surprising and rewarding how often our team recalls data from this research to help support any hypotheses we want to test.
The experience with the Email team highlights one of the ways we’ve adapted to the circumstances to bring customers in to inform our product development. In other scenarios, we rely on different research approaches, depending on the research questions, timeline, scope, and needs of the product and team, to ensure we have a thorough understanding of customer needs and whether or not we’re addressing them.
In the future, you’ll hear from us on how we’re making an impact through research, specifically our shift towards a more customer-centric culture, adopting the “Jobs-to-be-Done” framework, rapid prototyping and usability testing, and ethnographic field research around the world. Until then, we hope sharing some of what we learned will help you and your team overcome constraints to build better products for your customers.
Special thanks co-author of this post, Daniel Machado.
