How to Analyze Your Qualitative Data
Principles of Good Categorization Schemes for Product Managers
Analyzing free-form text customer feedback is an essential skill for great Product Managers. Great feedback analysis can uncover insights, help you prioritize, and shine a light on unknown issues.
But analysis of qualitative data is typically only as good as the categorization scheme you use.
From our experience, a good feedback categorization scheme should have 5 characteristics:
I. Useful
Do your data categories create value? This happens in one of two ways:
1) Data categories lead to action
Do your categories enable you to take action?
There are several valuable action types that your data categories can lead to…
- Routing the comment to a team/department responsible for the feedback — these are typically function- or product-based categories that map to your organization
- Routing a comment to a part of your process — these are typically type-based categories (bug, feature request, design improvement)
- Closing the loop — Problem or Feature-based categories can help you tie a customer’s feedback to issues you will address. When you do address the issue, by categorizing the comment, you can “close-the-loop” with the customer that left the feedback by telling them you’ve addressed their feedback. This leads to more loyal, engaged customers.
2) Data categories lead to decisions
Do your categories help you learn something or be aware of something that you didn’t know previously? Do your categories help you make informed decisions?
Hare are three ways good categorization can help you make better decisions:
- Better Prioritization decisions — Analytics on feature, bug, problem, etc. based categorizations can help you prioritize what to build and what issues to address
- Customer understanding—Learning something important about your customer that you didn’t previously know. This can be what they are trying to accomplish by using your product, what they like/dislike about your product, how they spend their time, what they wish your product could do, what criteria they employ when considering whether to use your product vs. other options, and many other insights.
- Issue Awareness —Uncovering problems with your product (bugs, process breakdowns, usability issues, etc.) that you were previously not aware of. Additionally, a good categorization scheme can alert you of specific, critical instances of problems.
II. Accurate
Do your data categories actually represent the context and content of the comments?
This seems straightforward/obvious, but the point is critical. If you mis-categorize comments, then aggregate-level takeaways will be meaningless at best and misleading at worst.
Two common mistakes here are (1) forcing comments that don’t fit into a pre-existing set of categories and (2) having individuals or algorithms without proper contextual knowledge categorize your feedback.
It’s important for you to have a way to audit the accuracy of your categories.
III. Mutually Exclusive
Are your categorizations along each dimension mutually exclusive?
It’s OK for comments to have multiple categorization tags, but your buckets along each dimension (type, team, problem, etc.) should have limited overlap. If there’s conceptual overlap, it means you probably haven’t defined your categories well enough and your analytics on those categories will be less meaningful. It also could be that your categories are at different levels of a conceptual hierarchy (up next).
IV. Taxonomical
Are your categories coherent along a conceptual hierarchy?
That question may sound dense, but the point is simple: categories should generally be along a consistent conceptual level of abstraction. Here’s a bad example: Canada, Utah, Africa. In this example, you have a country category, state category, and continent category. It’s not to say that you can’t have these various levels, but it’s best to begin at a consistent, lower level, and then roll up from there. For example, Utah, California, Texas, Arizona —> USA —> North America.
V. Adaptable
Is your scheme able to adapt to stay accurate and relevant as new data comes in?
Stale schemes typically become useless and inaccurate. It’s critical that your categorization scheme is flexible and able to adapt as you get additional data. Your approach should be able to adapt by adding new dimensions (function, type, problem, product), adding new categories within a dimension, and adding new levels of abstraction along the hierarchy.
In upcoming articles, we’ll share some specific categorization schemes that we’ve seen work well, but we wanted to start with the foundational principles. Additionally, we’ll address why to be wary of surveys, forms, and voting boards as your primary source of customer qualitative data.
