Leveraging a Small but Mighty User Research Team
A deep dive into the process of scaling, optimizing and democratizing the User Research Team at AppDynamics.
The Beginning
In 2017 AppDynamics hired our first user researcher. Three years later we now have 5 awesome researchers and our amazing user research intern. Naturally, with that growth comes a unique set of growing pains. In order for us to do what we do best — research — it was critical that our team dedicate time to focus on the structures and processes that would enable us to do just that. Although building out these structures is a work in progress, there are 3 key areas we prioritize in order to scale, optimize and democratize user research at AppDynamics: Build Your User Panel, Build Your Process and Build Your Impact. Let’s dive in!
Build Your User Panel
Recruiting users for user research studies can often feel like trying to find a needle in a haystack, especially when the haystack is composed of siloed teams, layers of upper management and incredibly busy professional workers. Such is often the case when recruiting in the world of B2B user research. In many cases, the target user is not the buyer or the decision maker. The target user is unlikely to be in contact with your company and finding these users can feel like an impossible feat. For this reason, our team knew it was critical that we build a robust User Panel of dedicated users who are excited to hear from us and share their feedback. So, how did we create our User Panel?
In order to grow our User Panel, we rely on several methods.
- Collaborate internally:
Each year, my team partners with our colleagues in marketing to send out an email campaign to thousands of folks across our customer companies to invite them to be on our panel. We received 290 sign-ups from our first email campaign and 160 from our second. We are also dedicated to building relationships with our Sales and Customer Success organizations. These folks spend months building trust and good rapport with each and every one of our customer companies and have a direct line of communication with target users. Our team leverages these preexisting relationships by asking our friends in sales to pass along our User Panel invitation to the users they work with.
2. Extend the interview invitation:
After scheduling a user interview, we let the user know it’s ok to invite other members of their team to the call, even as silent observers. Many times folks take us up on this offer and when they do, we make sure to remind them all to sign up to be on the User Panel at the end of the call.
3. Include panel sign-up in thank you email:
After every user interview, we email the participant(s), thanking them for their time and asking them to please forward our User Panel sign-up to other folks in their company. Word of mouth is a powerful way to expand your panel.
4. Ask during site visits:
During site-visits, we ask folks in-person to sign up to be on our User Panel. It’s helpful to have the sign-up on hand!
5. Leverage internal experts for internal User Panel:
At large B2B companies, chances are there are folks within your company who either a) use your product in the same way that your target users do or b) work extensively with end users and have enough field expertise to speak to user needs and pain points. This is the case at AppDynamics and is why we chose to create a second user panel — our Internal UX Advisory Board. This Panel is composed of dozens of Sales Engineers, Professional Services, Customer Success Engineers, Field Architects, etc. who are all dedicated to sharing their own expertise and their insights from the field.
We sent our first email campaign during the summer of 2018. Now, two years later, we have over 500 users on our User Panel.
Build Your Process
Even though our external User Panel continues to grow, so does our demand. During 2018 our team spoke to a total of 73 users. In 2019, we spoke to 331. In order to maintain our growing demand and increase our velocity, we needed to put a process in place to standardize our workflow. To do this we streamlined our process in 3 key areas: research requests, recruiting pool, emailing.
- Create intake process:
Step 1. Choose the Tool
As both our User Panel and demand for research grew, we needed to create a way to track all of the research requests. An off handed “Hey can we do research on this?” was no longer sufficient. To streamline this process, we decided to leverage a tool that the rest of AppDynamics was already using — Jira. To be honest, Jira is very engineer friendly, but I would argue not inherently designer / researcher friendly. Originally, our team tried out tools such as AirTable or Google Forms. These tools are extremely user friendly so naturally they appealed to our team. However, we ultimately decided that in order to increase visibility and track our user research across the company we needed to stick with what our stakeholders were already using, not introduce yet another tool to the bucket.
Step 2. Determine intake questionnaire.
After landing on Jira we needed to determine what information is important for us to receive when stakeholders request research. We worked with our Atlassian team to create a User Research ticket system with custom fields such as “Persona” and “Products.” We use these tickets to track and assign research projects.
Step 3. Enforce the process.
The hardest part of implementing a new process is enforcing it. After we introduced this new system, there was some initial resistance from our extended product team. We found ourselves reminding folks time and time again to “just submit a ticket.” However, though adding this process took some initial warming up to, the long term benefits outweighed the break-in period.
Further, other team members, such as our UX Writer, adopted a similar intake process for their own work. If it’s not in Jira, it didn’t happen.
2. Leverage external recruiting tools:
Our User Panel is amazing and for the majority of our projects we recruit using this panel. However, in order to scale our research we needed to find other ways to recruit users to minimize panel fatigue. To do this we turned to two external tools: Validately and Respondent.
Validately is a powerful tool that allows you to run both moderated and unmoderated studies. You can use your own user panel or you can recruit users from the Validately user panel. Our users are highly technical professional users and we find it challenging to find these users via the Validately panel. For this reason, we often use Validately with our internal users.
Respondent is another tool we leverage to recruit users. Unlike Validately, you cannot run studies using the Respondent platform. However, our team finds the Respondent user panel much more robust than the Validately panel and we are able to find users that fit our unique personas despite the technical nature of our product.
3. Automate:
As with any process, the more you can automate the smoother the process will be. There are three key areas where automation is critical to the efficiency of a User Research team: Outreach, Scheduling and Tracking. In an ideal world, user researchers could dedicate 100% of their time to doing what we do best — conducting research. However, the reality is the coordination involved in maintaining a high-performing user research team can often be a full-time job. This is why it is critical to automate this part of the process as much as possible.
Recently we decided to switch from Gmail to Qualtrics for our email outreach to utilize their email template capabilities. By using email templates, we are able to more efficiently and automatically send emails to our user panel.
Currently, we do not have a way to automate participant scheduling. Instead, we include a list of possible dates and times in our participant screener so that participants can select which dates work best for them. This takes out the initial back and forth emailing that occurs when trying to schedule a participant interview. Though this method works well for our team, it is an ongoing effort to automate this process more.
Lastly, it is critical that our team tracks which users we reach out to and which users participate in our studies. Now that we use Qualtrics to email our users we are able to track who has been reached out to and when to make sure we are not oversampling. However, we are still working on a way to automate how we track users that actually participated in our studies, not just users we reach out to via email. Currently we use a google spreadsheet that our team manually fills out after conducting each study. This piece of the puzzle continues to be a challenge for our team but something we are continually trying to improve.
Build Your Impact
Growing a B2B User Panel and running a bunch of studies is great but what does it matter if it doesn’t impact the product? Ensuring the aha moments are not lost is critical to any great research team. In order to truly champion our users and evangelize our research, we are consistently working on 3 key areas:
- Create user research repository:
Documentation is everything and yet it can often be a challenge for teams of all functions. As user researchers keeping findings alive is a critical piece of the puzzle. To do this, we decided to leverage a tool that was already widely used at AppDynamics — Confluence. We worked with our Atlassian team to create a User Research space. In this space, we create page trees by product line and a nestled page for each study. On each study page, we include the link to the Jira ticket, research objectives, executive summary, key stakeholders, and links to prototypes, notes and recordings. Our research repository is incredibly valuable not only for documenting past research, but we also use our study pages to upload user interview recordings. This way, stakeholders who can’t attend sessions live, can catch up quickly by watching the recordings. In the future, we hope to create a search feature in order to allow our stakeholders to quickly search for key terms and product areas.
2. Publicize the insights:
It’s great to have a research repository but what’s the point if no one sees it? In order to increase visibility of our research, we decided to create two public slack channels:
#user-research — We use this slack channel to publish findings, answer questions, and share knowledge with internal stakeholders.
#user-research-daily — This channel is connected to our UX Research calendar, which is a public calendar that we post all of our user interviews to. We use the Google Calendar for Team Events plug-in to pill the calendar info into the slack channel and encourage everyone to join as silent observers.
3. Democratize research:
Over the past year, as more and more people discovered the value of user research, the demand for research skyrocketed. As a team of 5, it can be challenging to keep up with these requests. Due to the increase in demand we decided to focus on enabling our designers and product managers to lead research. To do this, we increased the number of Qualtrics and Validately seats and we encourage our internal partners to take advantage of these tools.
In the case that a designer or product manager wants to conduct their own research, someone on our team often takes on a consulting role. For example, if a designer wants to run a quick unmoderated study to get feedback on a specific piece of a design, we may suggest conducting a study using Validately. We will help write the tasks and participant screener but leave it up to the designer to launch the study and analyze the results. By doing this, we’ve reduced the risk of being seen as a bottleneck. It’s impossible to dedicate full-time to all research requests but we don’t want to say no either. By consulting our designers and PMs we are able to teach them valuable research skills while also maintaining our own workload sanity.
Democratizing user research is beneficial not only to the research team, but to our stakeholders and users as well. By leading their own research, our designers and PMs are able to connect to our users by hearing their pain points first hand. For our users, democratizing research has led to increased user research, which means we are able to incorporate more user feedback than we could with just 5 researchers. At the end of the day our user is our top priority. By democratizing user research we are able to broaden our reach and spread the voice of our user across the company.
Final Thoughts
Over the past 3 years our team has doubled in size and quadrupled our user reach. Despite this, we are still a small team with so much room to learn and grow. In the coming months, years, and beyond I see our team building more bridges with our internal stakeholders and users, diving deeper into our complex user personas, expanding our ResearchOps practice and so much more, all while navigating our intricate product space. Growing a rich and robust B2B research practice is an ongoing process. I have no doubt in my mind our mighty team of 5 will continue to take this challenge head on.
