A Custom UX Research Workflow For Less Than $6k/year
You simply cannot build a successful product today without understanding your users to a certain degree. The purpose of this article is not to convince you of the need for customer discovery, it is to provide an insight into how we established a custom research workflow at MX. Based on our company, team, and budget, this was the smallest amount of investment we found for a meaningful return. I don’t mean to prescribe our workflow to all product teams, by any stretch of the imagination. I understand all companies, teams, and budgets are different, but I hope some of the insights will help others build their own successful workflows.
Why build a custom workflow?
Not every team needs a custom research workflow. There are complete research solutions available. For example, UserTesting.com is a wonderful tool. During my investigation, I was also quoted $30k/year just to get started with UserTesting. If your organization is like mine, increasing spend from $0–30k is practically impossible. Piecing together a custom workflow allowed us to build up our process at our own speed. It didn’t actually start out at $6k/year like the title suggests, we built up the expense over time. This gave us the opportunity to demonstrate value incrementally.
A custom workflow also gives us a considerable amount of flexibility in our process. Below I outline the four main challenges we identified and the tools that provided the best solution for us. If we become unsatisfied with a specific tool, or a better one comes along, we can simply replace that portion of the workflow without disrupting the rest of our process. We fully plan on tweaking our process over time in order to improve efficiency.
Consider building a custom workflow if you have a small budget, or need to demonstrate the value of user insights before you can purchase enterprise tools.
Four research challenges
To make sure we were finding the right solutions, I had to boil down the main problems, or challenges, we were facing in our research efforts across the product teams. After an audit of the current solutions, and a bit of discovery with the product teams, I was able to identify 4 main challenges we were coming up against. The 4 challenges were:
- Recruiting
- Recording
- Analyzing
- Distributing
Recruiting
While the target audience for MX is really anyone with a bank account, recruiting appropriate candidates is truly unique to our organization. MX provides white label services to financial institutions, who turn around and offer those services to their consumers. Our business model is B2B2C. As a result, it is difficult to get direct access to our end-users without working closely with the financial institution. While we are working with clients to facilitate access to end-users, the process is slow and difficult. In many cases, the financial institutions don’t necessarily want their customer base to know that they are using third party services.
While we continued to work with clients to recruit actual end-users, we also felt like there was a lot to learn about the populations general perception of their own financial institutions. When I joined MX, the team was recruiting locally via Facebook and Craigslist ads. This was time consuming and a limited participant pool. Many times the participants were already familiar with MX and our services. Sometimes they were friends or family members who were overly agreeable. On a few occasions, it became clear that participants were interested in becoming hired at MX.
Based on the time investment and the quality of participants, it was clear that we needed a better way to recruit. I began looking for a solution that would make finding, scheduling and paying participants much less time consuming. We also needed a wider pool of participants, which meant we would have to perform remote interviews as well.
Recording
In our previous process, participants would come to the MX offices and we conducted moderated, in-person interviews. We used the company VC software (BlueJeans and Zoom) to record sessions and allow team members to watch in real-time. We had a desk camera to record the participants mobile device, if we were testing a mobile feature.
The greatest challenge of recording these sessions was that the VC software simply isn’t made for
research interviews. The video quality was poor and the audio would often skip, missing valuable responses. When team members joined, they would forget to mute their mic and distract the moderator. The desk camera had a hard time focusing on the mobile device and the participant would move inevitably move it out of view anyway.
We needed a solution that would capture A/V at a higher quality. We wanted to be able to refer to the recordings later and distribute important clips throughout the organization. Since we decided to expand our participant pool to include remote sessions, we also needed a reliable remote solution that would allow us to test mobile experiences. Finally, it was also important that team members could sit in on live sessions, if they wanted, and not interrupt the interview.
Analyzing
Once we had captured qualitative data from qualified participants, it was important to be able to analyze the feedback. We also have several passive feedback channels that provide us ongoing qualitative data. We wanted to be able to combine all the feedback channels and run analysis across all the datasets. With thousands of open text responses from reviews and customer service tickets coming in every month, this is by far the most time consuming step in the process. It is also critical to find themes and make sense of the data we were collecting.
Since we are a relatively lean organization, without dedicated researchers, it was important that we could find a solution to perform quick analysis of our data, even at a very high level. Manually transcribing the interviews we were conducting was also extremely time-consuming. Automating the transcription process would also free up valuable time.
Distributing
Lastly, none of the data and analysis would be valuable if they weren’t distributed and digested by the rest of the teams. Created insights and reports is another time consuming activity. Someone has to curate the most valuable insights and produce a report. Usually that report is created as a PDF and sent around in an email. If you’re lucky, some of the team members will actually open the PDF and glance at the report you painstakingly put together, only to lose it in an email thread when they really need the information months later.
We needed a solution that would help us create valuable insights quickly. We also wanted these insights to live in their own space, so that they could easily be accessed and discovered by team members on demand.
The Tools of the Trade
After identifying the challenges and needs of the team, I dug in and researched the research tools. I spent several months taking notes, and signing up for dozens of free trials. Below is a list of the tools that we settled on to accomplish our goals.
User Interviews
Userinterviews.com is a service that specifically handles recruiting for, you guessed it, user interviews. They charge $30 per participant , or $70 if you need a specific, professional user. The incentive is added to that cost, but is automatically distributed by them once you complete a session. They pull participants from their pool of users, and provide you a list of qualified candidates. There are basic characteristics you can choose from for the $30 tier. You can even create a simple screener survey. Once you choose the participants from a list of qualified candidates, User Interviews coordinates the scheduling and allows you to message the participants. After that, you’re on your own. You choose how you meet and what you talk about. Once you’ve completed the interview you just mark the participant as complete and they send out the predetermined incentive.
What we like:
We are able to recruit a variety of different users from across the country in a matter of minutes. User Interviews handles the scheduling and the incentive. Time to recruit has gone way way down and the quality of participants has gone up.
Lookback
Lookback is a recording platform specifically built for user feedback. Starting at $49/month Lookback has free “observer” accounts that let team members tune into live sessions without interrupting the interview. The observers and moderators can take time stamped notes in real-time. Once you’ve recorded a session, you can easily make clips of important feedback. With the Pro subscription ($99/month), you can export those video clips. The iOS SDK will record the screen of the iOS device AND the users face with the front facing camera.
What we like:
The SDK allows us to record remote sessions on mobile devices. The quality is also much higher than the previous VC tools we were using. We love the ability to take time-stamped notes and quickly create clips or highlights.
Storage
A quick note on storage. Our company uses Google Drive, so we store all the media files on a company drive. This is not an added cost to the company, but it is something you may need to consider. As part of the Google Suite, we received permission to privately publish interviews to YouTube. This made it easy to share with the product team, and everyone else in the company. We create a “highlight reel” of a testing session with clips from Lookback and post them privately to YouTube. This turned out to be the easiest way to share an overview of each session, as it was easy to distribute and doesn’t require hours of the teams time.
What we like:
A central place for raw recordings. Everyone on the team knows where the files are and has access to them.
Otter.ai
After we have the recording of a session, we import the audio/video file to otter.ai to automate the transcription. They give you 500 free minutes every month, which we found to be plenty. The transcription is not perfect but it’s pretty good. It takes some time to cleanup but not nearly as long as it would to type the full interview by hand.
What we like:
It is free and very fast.
Dovetail
Dovetail is a new, intuitive research platform. Starting at $100/month for 5 users, it can perform sentiment analysis and basic tagging on your data. It also allows you to easily create insights and share reports with your team. It may not be the most powerful tool but it is extremely easy to use. When we added team members to our Dovetail project, they were able to orient themselves almost immediately.
What we like:
We found that Dovetail serves two functions for us. There is sentiment analysis and theme tagging to help with our analysis challenges. They also make it really easy to search themes across data sets and create insights. I can create a monthly report on customer service responses in a matter of minutes, not hours. Sharing that report with the team is as easy as posting the link in Slack or email. It was easy to make Dovetail the single destination of all user information. If someone loses a report link, they know they can find what they’re looking for in Dovetail.
Cost breakdown
Below is a quick cost breakdown of our workflow. Calculating a consistent price for recruitment is difficult. User Interviews is a variable cost since you pay per recruit. The number of participants every month and the incentive amount will vary. The bare minimum would be 5x30 minute interviews a month. $30 recruitment fee + $30 incentive = $300.*
Userinterviews.com: $300/month*
Lookback.io: $49/month
Google Drive: No additional costs
Otter.ai: 500 free minutes
Dovetail: $99/month
Total: $5,376/year
Based on the challenges and constraints put on our team, we found this workflow to provide a tremendous amount of value for a reasonable cost. It is not perfect, by any means, and will not work for every team, but hopefully it can help those of you looking to build a custom workflow for yourselves.
What do you think of my workflow? Is anyone doing the same amount of research for under $6k? I would love to hear what other teams have implemented.
Cover photo by You X Ventures on Unsplash
