Getting Value From Customer Testing
Curating Your Own Customer Testing Group
[NOTE: This originally appeared on MindTheProduct]
A customer testing group can provide value in a variety of situations, ranging from a gut-check to getting long term deep insight. Here’s a guide on how user testing and consultation, alongside a tiered approach to segmenting your customers, can add value to your role as a Product Manager.
Continuous User Testing & Consultation
When solving customer problems like growth, retention and usability, it’s important to have a group of users to poll quickly for their opinions. A member of leadership once lobbied the Product team to take our push messaging strategy from once a week to several times a day. We saw great engagement on a single push, we should see higher engagement when we increase frequency... Simple right?
This seemed excessive to me, but I didn’t have great data or customer sentiment on hand to push back with a testing strategy. After speaking with only a handful of customers from different age groups/engagement patterns, I found that we didn’t need to be as conservative as I had argued. The insights were surprising and within a day I had great data to bring back to the business to start experimenting with an approach.
The Tiered Approach
Discussions like this come up on a daily basis and it’s hard to conduct user research in every problem area you want to solve. Think about the time you spend as a Product Manager on a graduated scale. It’s important to keep communication with users tactical and focused. Just as you have close friends and general acquaintances, you should group the users you reach out to. Here’s a framework that can be tailored to the needs of your organization:
Group 1 — Infrequent Contact (~2000 people)
Think of this as your catch-all group for people you’ve engaged in the past. These can also be non-customers you poll in a blind study.
This group is great for:
- Competitive research
- Early rollout of new builds and features
- Infrequent surveys, NPS, awareness
Group 2 — Advisory Committee (~200 people)
These are generally users that have opted into surveys in the past, submitted feature requests or bugs or even users culled from a Tweet, comment or review of your product.
This group is great for:
- Pool of users for testing
- Frequent surveys
- Prototypes
- Gut-check on screenshots of near-term releases
- Feature usage data
Group 3 — Personal Group (~20 people)
This is your group. If you are on a team each of you should have your own group of 15-20 users that you have constant contact with. All of these groups should have a healthy mix of demographic, but this group should have the most variance.
This group is great for:
- Questioning the validity of new features and the removal of old ones
- Product health of patient questions
- Sharing rough prototypes
- Beta testing
Each group is not mutually exclusive, but remember that you want to limit your communication with everyone except Group 3, so that your response rates are healthy. Feeling out how often that engagement should happen comes with time.
Constant Grooming
The beauty of this tiered approach is that you can generally graduate users from one group to the next (or skip them ahead). Don’t forget that you need to constantly groom this collection of users. Ensure that your groups consist of varied demographics, have different use cases and are a mix of old and new customers (this is where the hard work comes in).
