A Qualitative User Experience Research
Talk to 5,000 customers in just 24 hours!
The story of our product research’s methods with a new solution!
A few years ago, we made up a product team, responsible for implementing a product roadmap based on user research.
This is the story of the team’s efforts in the way of being a data-informed one. But how?
We consider three steps of research:
1. Quantitative’s Research
2. Combination Quality and Quantity in Research
3. Qualitative’s Research: Talking Directly
Three Steps of Researches in Our Team
First Step: Quantitative Research
On these days, we don’t have enough knowledge about user research methods. But we study about it and start with Quantitative Methods:
- Counting users’ interactions with GA Events
- Define Google Analytics Goals and Conversion Rates
- A/b Testing or Multi-Split Testing
Quantitative days of our research was so useful and had a positive impact on our product. But we found that we need something Stronger, or maybe more human-generated data!
Second Step: Combination Quality and Quantity in Research
We added some Qualitative Methods to our researches:
- Session Recording
- Heat Map Analysis
- Funnel Analysis
Heat Maps visualize all users interactions and its combination of Quality and Quantity research.
In the article “3 important steps in session recording analysis” I wrote about Session Recording. It’s easy and useful.
But as a Product Manager, you always want to find more qualitative feedback.
Third Step: Talk directly!
This article is about this step of our researches. In the step, we found a way to get purely qualities feedbacks. In the last days of the previous step we thought about qualitative methods:
🤷♀️ Net Promoter Score
What you can understand from 0 to 10?!
Ok, 7 is better than 5, but how does this number discover problems?
💁♀️ Feedback Polls
For example, you ask “What’s your most important problem with my product?”; You have a good question but you cannot afford the best option for customers’ problems!
Assuming that we picked these problems:
- Pricing
- Support
- Mobile Applications
- Content Quality
OK, there are two main problems with Feedback Polls:
- Are you sure these are the most important options?
- This type of question with predefined answers changes the customers’ mind and they can’t reflect their real problems.
🤦♀️ Surveys
Who has the time and patience to fill out long forms?
And again, repeated previous hesitations:
- Are the questions good?
- Are the answer choices good?
- What about the “changes in customers’ mind”?
All of these methods are functional, but we still do not feel good with these methods.
✍ Direct Feedback
One day at launch time, one of our team members, @payamshakibafar had an idea:
Let’s give our customers an opportunity to say whatever they want!
We create a feedback form with a powerful question: “Criticize Filimo!” and an empty text box to write whatever they want!
Before running this research, I thought that our customers will write more different topics that we will not be able to classify and analyze the feedbacks.
But the result was unbelievable! We received 5,000 feedback in just 24 hours and our customers provided so similar problems which made us able to categorize the problems under five main topics and surprisingly less than 1% of the feedbacks were inappropriate.
We find the perfect way to get the customer’s needs directly, and the real needs of customers will pave the way of growing!
Benefits of Direct Feedbacks
Tone of Voice
The first thing that will attract your attention is the Tone of Voice of your customers:
- Are they teenagers or talking formal?
- Do they use the same words which exist in your site to ask about features?
- Are they angry or not?
- Are they thankful or not?
No Feedbacks Orientation
Direct Feedbacks let people express their own opinions, in their own words, instead of choosing between predefined answers.
Don’t use Textarea, use simple and small Text Field. When you are using Textarea, people think that they have to write a letter for you, but if you using a small text field, they will try to say the most important things in short.
We use the small text field but without character limitation to make those wants more space, can write multiple lines.
Product Problems NOT Design Issues
One of the important stages you have to pass if you want to switch from UX Designer to Product Designer is thinking about main problems more than Design Issues.
Listening direct feedbacks will show you that most customers don’t think about Pixel Perfect Design or Color Scheme and etc. They have deeper problems with the heart of your product strategy!
For finding design issues, Session Recorder tools are much better while Product, Marketing, Support Team as well as Stakeholders need to have Direct Feedbacks.
Problem Details
We always know the general area of problems, for example:
We know some problems with Information Architecture but,
- What’s this problem exactly?
- Witch content of our product is missing?
- What content do customers want to see on the Home page?
You can find the details of the problems in direct feedbacks.
Competitor Analysis
- Are they talking about competitors?
- Which of your competitor’s name is repeated and which one is more famous?
Loyalty Check:
- When customers talk about the competitors, about what specific feature, they’re talking?
- Are they want to switch to your competitors? Or just want small features of the competitor’s product?
Create Roadmap
We run this research every six months and the best part as a Product Manager is that you can create a Roadmap and its priorities based on frequent responses.
How to Record Customer Feedbacks?
We easily implement these feedbacks with Google Tag Manager and Google Analytics.
This implementation way is free and easy and you have more additional information about your customers in Google Analytics that can merge with feedbacks such as Device Category, Referral, User Types(New vs Returning), Goals. You can submit various information with feedbacks; For example, user state(log in or guest) and etc.
Don’t think about it, let’s trying this method to understand benefit of Direct Feedbacks:
- Open Google Tag Manager Account
- Before any change in your GTM, export of current version in Admin > Export Container
- Now, go to Admin > Import Container
- Download latest version of my GTM export.
- Choose JSON file and select Existing workspace to upload.
- It’s ready! Go to Variables and change your GA ID on “HXA — Google Analytics ID” variable.
- Click on Preview button to see Feedback Box.
I define unique names for all Tags and Variables, so don’t worry about conflicts! Also, there’s some settings to customize Feedback Box on “HXA — Direct Feedback — Settings” Tag.
You can check result in Google Analytics > Realtime > Events > Event Category: HXA — Direct Feedback.
Caution: You may not upload any data that allows Google to personally identify an individual (such as names and email addresses), even in hashed form.
How to Analyze Customer Feedbacks?
We export Google Analytics Events of users feedbacks to Google Sheets(thanks Google services!) and then start reading and add one, 2 or 3 tags to each feedback.
Which Tags?
Read feedbacks, and be attentive to repeated problems; These are tags.
For Example:
- Content
- Pricing
- Bugs
- Feature Request
- Competitor
I created a Feedback Template on Google Sheet to import data from Google Analytics
Make a copy of this Sheet and start to analyze your users feedbacks:
How to Present or Visualize Results?
If you tag all of the response you can create a useful chart for the numbers of tags, like the picture above.
How to respond to your customers?
The best response to your customers is in solving their problems.
If you answer the customers’ feedbacks right away, the next time you do this research, the type of problems has to be changed, and the same problems must not be mentioned again.
The first time we did this research, the customer satisfaction rate was 30%. We follow the product roadmap, based on the feedbacks and repeated the survey after 9 months. The satisfaction rate increased to 60%!
Let’s try it out and feel free to discuss this subject in comments or @parhumm.
