The Data Source You’re Missing
How to use customer interviews to get deeper insights into your data science projects
A Different Kind of Research Team
Doing customer research in a pure data science mode typically means creating numbers and insights based on customer behavior data that is collected. This generates great insights, but it also leaves a lot of potential information out of the analysis.
At When I Work, we have formed a cross-functional team to research our customers’ needs. We wanted to find out what friction our customers were facing while adopting our products so that we could make the adoption process smoother for future customers. The team used qualitative research techniques to augment the quantitative techniques that the data team is skilled at.
The team includes members from User Experience, Customer Advocacy, Marketing, Growth and Data teams. Having a group with a diversity of experiences brings skills from a wide range of disciplines to the table. The powerhouse combination of qualitative and quantitative research helps us learn about our customer needs in ways that would be impossible for either one of these methods alone.
Qualitative versus Quantitative Research
Qualitative research comes from the social sciences. It is used to understand the why and how of human behavior instead of just the what.
Rather than by logical and statistical procedures, qualitative researchers use multiple systems of inquiry for the study of human phenomena including biography, case study, historical analysis, discourse analysis, ethnography, grounded theory and phenomenology.
Qualitative methods in a business setting also include interviews, surveys, user observation and content analysis. Since When I Work is a Software-as-a-Service company, understanding people is a primary driver for success.
Qualitative research has some advantages over quantitative research in a business setting. One advantage is that you can get in-depth information from your customers about their day-to-day concerns. Rather than being limited by the data your company has collected, any topic you can imagine is available for research. It is also possible to find out about customers’ motivations instead of just their behaviors. While a quantitative analysis can reveal that users prefer one product feature over another, only qualitative research can help us understand the reasons behind this preference. Finally, through interactions with customers we gained empathy for them and better understood their struggles.
On the other hand, there are some disadvantages to qualitative research. The primary disadvantage is that data collection is a labor-intensive process. We focus on interviewing to collect information about our customers’ preferences. For our research interviewing a customer takes between thirty minutes and an hour. Even after hours of work you might only have a handful of data points to analyze. It also is more emotionally taxing than quantitative research. While it is easy to stay detached when analyzing a spreadsheet of numbers, talking to people with all of their quirks requires higher amounts of energy.
A Match Made in Heaven
The method that works best for us is to combine the insights from qualitative and quantitative research. Findings from each method inform each other and build on each other. For example, we might hear from several customers in interviews that they found a feature difficult to use. We then analyze our clickstream data to see if this was a wide-spread phenomenon with our users. If we find out that it is, we can use the clickstream analysis to create more specific questions for the next customer interviews to get more insight into the problem.
Being hypothesis-driven makes both types of research more efficient. In quantitative research, floundering around in large sets of data without a clear direction is a waste of time. In the same way, conducting qualitative research without a clear idea what you’re seeking to learn will end in frustration for both you and your customers. The scope of possible topics you can interview your customers about is vast so they need to be narrowed down before the interview begins. The topics should be hyper-focused on the research questions that you want to answer.
Interview Preparations
One of the benefits of combining typical data science with qualitative research is that we can be highly selective about the type of customer that is interviewed. Since our data infrastructure is well-developed we have a large amount of information about each customer. We leverage this data to select the exact type of customer that we want to talk to. For example, we wanted to talk to a group that had tried our Time Clock product but didn’t purchase it. With our clickstream data, we were able to find dozens of these customers to talk to. This is much more efficient than random sampling from the entire population of customers and hoping that we find a few who have the relevant experience.
Once we have a list of potential customers to interview, we send out an email asking them to sign up for a time to talk to us. We use Calendly so that customers could pick the time that works best for them. There is typically a 10% response rate on invitations for interviews. Since we run interviews in batches of 10, this means that we need to email 100 qualified customers to fill the time slots.
In order to make the interviews systematic, we create a script to use during each interview. This ensures that we ask similar questions to each customer. It also provides a roadmap to the interviewer so that they always have another topic to discuss and don’t have to be anxious about what to say next. While the scripts provided a structure to the interview, it is also necessary to let the process wander into whatever the customer’s largest concerns are. There is a delicate balance between listening to the customer, following them where they want to go and directing the conversation towards the topics we are researching.
Interview Process
We always have at least three members of the research team in the room for each interview. The interviews are by phone so having extra people in the room didn’t interfere with the process. The interviews are scheduled for 30 minutes; although if things are going well and the customer has more time available we extend these up to 45 minutes. This is long enough to gather valuable information, but not so long that the customer wants to disengage.
One team member is designated as the facilitator for the call. They are responsible for starting the call, asking questions from the script and directing the conversation. The other two team members take notes. One person takes electronic notes in Google Sheets. These are meant to be complete documentation of everything that is discussed. They include facts like company size, interviewee responsibilities and software tools used. In addition, the phone calls are recorded with customer consent so any gaps in the information can be filled in later.
The third person writes down insights on Post-It notes. These are less about the nitty-gritty facts and more high-level insights about the customer. Each note contains a compact idea. For example:
- Prefers to learn of new products on social media
- Preparing payroll is a hassle
- Most important priority for next year is to grow business
Each interview generates between 20 and 40 of these insights.
Interview Analysis
One method that we use to synthesize the information that we gather from the interviews is affinity diagramming. This is a more physical and manual process than anything done in a typical data science problem. It organizes the Post-It notes that are generated during the interview process. We group them together on the wall based on the topic of the notes. This helps consolidate the insights learned over many interviews. While occasionally you can make a connection with another interview during the middle of an interview, it is impossible to keep track of everything in your head that you learned across many interviews. By using affinity diagramming we found hidden themes that didn’t jump out at us during the interviews themselves.
Persona creation is another method that we used to synthesize the information. This consists of integrating the information that we find into the outline of a representative yet fictional person. The persona lists the attributes, behaviors and concerns of a typical person. Since our research is into business managers we might have personas that represent small business owners, operation managers and payroll accountants. Thinking through the concrete attributes that each of these personas helps to organize the many facts we have learned. One guideline for creating personas is that you can only use the information gained through the interview process. Any speculation about what the personas are like should be left out.
Conclusion
The final product of all of this research is a set of insights that we could use to make strategic decisions with our product direction. Each of these insights is backed by the comments from dozens of interviews with the exact type of customer we wish to serve better. The insights provide clarity on the most important customer needs that we can address.
Qualitative research is a powerful tool to add to the typical quantitative data science methods. By including data sources outside of structured data you can expand your topics of research beyond what is stored in your organization’s databases. When researching problems that are both complex and critical to your mission, it is important to include as many sources of information as possible. Using qualitative methods will increase the amount of knowledge you have and can deliver key insights that aren’t available using only traditional data science methods.
