User Testing for a Niche Product
Our framework for user testing
User testing is a critical part of the design process. It enables designers, engineers, and product managers to prioritize and make decisions for new features and functionality. But what do you do if you have a niche product with very few users? Is user testing still valuable? What kind of testing should you do?
The DNC Tech Team supports several data systems for use by campaigns and organizations to learn about voters and their needs. One of these systems, Portal, connects multiple data sources in a centralized experience.
In comparison to typical consumer tools, the user base for this product is small. According to analytics, on average we have about 180 users and 115 new users in a monthly time frame. While this user group could be considered small, the product supports essential campaign work.
We are constantly enhancing current features and providing new functionality. With each iteration we review analytics and gather feedback to see how the changes are resonating with users.
Small-scale Tools and Frameworks
When working with niche products like Portal, it’s important to match your research techniques to the scale of your user base. One Nielsen Norman Group’s analysis suggests you only need 5 participants for your user study. This keeps the investment proportional and the feedback actionable.
Use Quick Tests for Basic Answers
When we need to think through a basic design decision, such as what types of icons to use for a design or how a dashboard homepage will look, we often turn to Usability Hub to do preference testing. These short tests are under 2 minutes in length and are fun for people to take. Seriously, people always tell me they have fun taking these tests.
Capturing feedback quickly enables us to be nimble and to create rapid iterations and improvements to a design. These tests are also useful when you find internal discussions going in circles with no agreement. In these cases, I find myself saying “let’s ask our users and see what they think.”
Metrics help support a decision and cut out unnecessary conversations.
Talking to Your Core Users & Building Relationships
Having a small user base means we work closely with our users on a daily or weekly basis. Frequent communication and our close relationship with users helps recruit testers. Whether we are in the exploratory research phase of design or ready to implement something, we always try to talk to at least five users to gather feedback. Remote tools such as Zoom help us to facilitate these conversations.
Before each session, we set up a testing script and scenario we walk through with our users. We usually set up calls with two users, one moderator, and one note-taker. Having two users allows us to gather different perspectives at once. Calls last roughly 30 minutes so we don’t take up too much time in busy schedules and to keep the meeting short and interesting.
Communicate Your Findings
People want to know that their voices are being heard and to see what is being implemented on their behalf. Internal stakeholders want to see the value of what you are doing. Reporting out your findings gives both audiences what they are looking for.
Updates don’t need to be a formal slide deck or require a large amount of effort to complete. The DNC Tech Team uses Slack as our primary way to communicate high-level findings and interesting user anecdotes.
Ensuring our users see how we value and incorporate their feedback is another reason they’re willing to participate in user tests. Speaking of which…
Following Up
Don’t stop once a product is shipped. Follow up with your users and check in once they have used a feature for a while.
One way we do this is by setting up face-to-face check-ins (via Zoom). During these check-ins, we have our users show us how they are using the product. Doing this allows us to get specific input and to test our assumptions by digging deeper into how well the product is supporting them.
When not doing face-to-face check-ins, we send out an evergreen survey a few weeks after every major release or update. This allows us to track how well our products are performing over time, and quantify the results in a trackable way. The graph below shows a view of how much our product has improved in the past few months.
We also continue conversations with our core user group informally in Slack to gather their feedback directly. Surveys are great, but they don’t provide anecdotal feedback such as:
[Portal] is 5 million times better than what I had to do in 2016, and 2018.
Data Director, State Party
No Product Too Small
Making assumptions about your users and their needs is a dangerous way to design. Users best know how they use your product and can provide you with insights you may have missed. Use their feedback to make your products better. Continue talking to people and providing visibility into the process.
Have been watching Portal develop for a long time. I like the direction Portal is heading in. The features are appearing before I know I need them.
Data Director, State Party
While there are multiple ways to approach user testing, we have found this formula to be quite successful for our DNC Tech Team.
