Product Managers: User Research on Your User Research
Findings from a nationwide product manager survey on user research demands and practices
Understanding users matters to everyone in tech organizations — product people along with other business and operational partners — regardless of geography, organization size, or customer type. Organizations are hungry to learn about their users and expect to get this information in order to achieve objectives.
We surveyed product professionals (product managers, designers, user researchers, etc.) from more than 100 tech organizations across the United States about their user research demands and practice. We uncovered insights and identified best practices for various research methodologies, collaboration techniques, and prioritization.
TL;DR:
- Usability testing, interviewing, and surveying appear to be the three most commonly practiced user research methods today.
- While we all have many ways to improve our user research effectiveness (e.g. breadth of tools used, frequency of practice, sharing of insights), the biggest opportunity for more impactful user research is to better share the research we’re already doing. User research is often cooped up in the product organization or not shared as well as it could be when it is communicated out, and non-product people aren’t getting the user understanding they expect.
- Product people in consumer-user organizations (B2C organizations) do more user research and understand their users better than product people do in B2B only organizations.
- We found some indications that Bay Area tech organizations and product people value user research more than their colleagues do in other geographies.
Background on This User Research on User Research
XO Group recently hosted the New York Product Conference for the New York product community, with the help of our friends from Cornell University and Cornell Tech. We spent the day exploring the theme of Users, and learned from product and design leaders across New York’s leading tech companies including the New York Times, Jet.com, Foursquare, Blue Apron, Invision, Glossier, Ringly, and many others. Here’s an article with key takeaways from the conference.
Given the day’s theme, I collaborated with Krista Plano, Lead User Researcher at XO Group, to study how product builders (product managers, product designers, etc.) learn about their users today. We believe that knowing your users is a superpower, and we sought to baseline how product groups and their non-product colleagues around the country currently value, conduct, and share user research. We acknowledge that the survey results are representative of our own product networks and we view these results as directional, not precise.
We conducted a survey across our product networks in October 2017 and received responses from about 100 organizations of varying sizes, mixed across the United States though over-weighted to New York City and California. Respondents were mostly product managers, and their organizations serve a healthy mix of user types across consumer, enterprise, and small business.
Below are interesting key learnings we discovered after looking for differences across respondent role, organization size and location, and user category type.
What Type of User Research Are We Doing?
As a baseline, this chart shows most commonly used user research techniques in tech organizations today. The scale is directional, not precise, but shows that usability testing, interviewing, and surveying appear to be the three most commonly used user research methods practiced today. Check out this User Science article for an introduction to different user research methodologies.
Meeting User Insight Needs
Tech organizations want to understand their users: “Talking to my users is a high priority at my organization” received a consistently high score across organization size, user type, and geography. We found less consistent and lower scores for user research practice and user understanding. Respondents report room for improvement in how frequently we practice, the breadth of tools we use, and how successfully we share user research. We found two noteworthy macro-level points here.
First, we found a perceived gap of user understanding between product (e.g. designer, engineer) and non-product (e.g. sales, finance) people, even though both groups want to understand their users. This finding shouldn’t be surprising, as product people conduct the research and are closer to their findings. (There are exceptions here, such as customer support teams, who learn a lot about the subset of users that contacts customer support.)
Digging into this understanding gap further, we found that once organizations grow past 50 people, perceived user understanding diverges between product and non-product people. Perhaps smaller organizations are just better at sharing information? Perhaps smaller organizations are exposed to users in different ways? Perhaps product and non-product people work more closely together in smaller organizations? We didn’t find out why smaller organizations might be better at sharing user understanding, though that would be an interesting point to investigate with further research.
Second, we found that organizations already do a lot of user research sharing, especially through slideshow files and live presentations. The majority of organizations share their user research somehow. However, combined with the outcomes results above, it seems like this research sharing isn’t working as well as organizations expect.
Let’s connect these dots. We found that while organizations expect to understand their users well (3.8/5.0), product people only somewhat understand their users and not up to the target level (3.3/5.0, less that the desired level of 3.8/5.0). Those product people then try to share their research, which most organizations actually do especially through presentation files and talks, but even with this research sharing, product people perceive non product people only weakly understand their users (2.7/5.0). Let me repeat that — product people do the work to share user learnings, but don’t believe that non-product people actually understand their users. Something’s not working here in spite of work being done. Perhaps this knowledge sharing from product people to non-product people is the biggest opportunity to better drive user understanding inside an organization. To help our organizations become more user-centric, we may not need to do more or different user research, rather we may just need to get better at sharing the research we already do!
Note — we found an exception in smaller organizations, which seem to do a better job of sharing user knowledge across the whole organization.
Tips For Communicating User Research
Some great ways to improve the effectiveness of sharing user research are to be consistent and make it easy for everyone to access your research! At XO Group, we share the voice of the user regularly in several ways: quarterly research presentations and product demos, monthly user panels, weekly blog posts (we also share release notes here!), and daily messages in dedicated Slack channels.
Other helpful resources:
- Rank your findings, illustrate risks and benefits, and provide recommendations and next steps — get more tips for communicating with stakeholders from this UX Mastery article.
- Make user research collaborative: involve your team in research efforts to build empathy and to demonstrate the value of research. Here’s a great related article from Nielsen Norman Group.
- Host a user testing viewing party: observe and analyze your videos as a team. Usertesting knows how to make it fun!
To increase frequency of user research practice within our product organization, we put all new team members through a day of user research training, we regularly conduct XO Product School sessions to uplevel the team’s research skills, and we offer weekly user science office hours to the company. This way all product team members are empowered with tools, best practices, and support to conduct regular research.
User Type Impacts Research Practice
Organizations can serve different user types: consumers, enterprises, small businesses, etc. as well as multiple types of users at the same time. For example, marketplaces like Yelp and GrubHub typically connect consumers with small businesses, serving both of these user types. In practice, we’ve found that consumers are the most accessible user type to research because the individuals in an organization are consumers themselves, and we can talk to anyone as a representative consumer. Even organizations with both B2C and B2B users often start their user research practices with consumer study and then expand to business users. We therefore segmented organizations between consumer as a user (could be B2C or both B2C and B2B) and no consumer as a user (e.g. B2B only), and found some potentially meaningful differences.
We found that across the different user research methodologies, product people at organization with a B2C component are performing more research than product people at B2B only user organizations. (Interviewing appears to be the exception.) Given that our results are directional and not precise, we don’t want to read too much into this chart other than the rough headline of more user research being done across the board in B2C organizations.
Combining these results with the chart below around perceived user understanding, we see in non-consumer/B2B-only organizations both a reduced practice of user research as well as a lower perceived user understanding by product people. This relationship makes sense, as more user research contributes to better user understanding by those researchers.
We find it interesting that organizational interest in user understanding and perceived actual organizational user understanding are pretty similar across organizations with different user types, while it’s the B2B only product organization that’s falling behind its B2C peers in user research and understanding. We’ll dig deeper on this topic in future research.
Tips For Conducting B2B User Research
We conduct B2C and B2B research at XO Group and understand that recruiting your target user for B2B can be a challenge. Check out how data startup Segment, “designed an automated workflow to recruit participants for a UX research program…to coordinate testing and feedback sessions throughout the product development process”. This article explains Segment’s process which you can borrow and adapt. Another tip to keep in mind while recruiting: When you’re transparent about your goals, you’re likely to find users who are interested in having an impact on product direction.
User Research Practices From Coast To Coast
As mentioned earlier, the most frequently adopted user research methods appear to be interviews, usability testing, and surveys. We also found that relative usage of these user research methods stays consistent across geographies.
We did find some interesting differences across geographies though. Although user research practice did not appear different by geography, user research attitudes appear to vary. We uncovered that Bay Area product people believe user research is more important and they understand their users better than their product peers in other locations, and also perceive that their non-product colleagues understand users less (see chart below). Again, we acknowledge that findings are directional and look forward to digging into this next year
Some of these differences are slight, but directionally they suggest a greater awareness of and value for user research in the Bay Area compared to other geographies. We wouldn’t be surprised if this hypothesis were true given the depth and history of product-centric tech companies in the Bay Area. For example, organizations like LinkedIn, Google, Facebook, eBay, Intuit, Uber, etc. develop dozens or more of new user-researched trained product managers a year, and have been leading product schools of thought since the field of Internet product management emerged as a maturing discipline in the 2000s.
In this light, it’s actually surprising that we didn’t find more user research practiced by product people in the Bay Area given the region’s higher user interest and confidence. We will dig more into this topic in the future.
Final Thoughts on User Research
Keep talking to your users — the people you’re building products for — and spread your knowledge across your organizations! It’s our job as product people to be the voice of our users.
Other Helpful User Research Resources
- Use a mixed method approach! This Nielson Norman Group Cheat Sheet is a reminder of research methods and when to use them.
- Media Temple’s list of 12 Practical Ways to Become User Centric explains how to consolidate learnings, fill in the gaps, and work from design thinking methods e.g. empathy and journey maps.
- Find out how to use affinity mapping to organize learnings so that you can more effectively share them — Interaction Design Foundation.
What’s Next for Our User Research?
In preparation for the 2018 New York Product Conference, Krista and I will launch an expanded user research survey earlier and outside of our networks to gather a wider respondent base. We’ll again gather insights on how product builders interact with and learn about their users, targeting some unanswered questions raised above, and also research product development methodologies, quantitative testing practices, and user research goals. As product builders we all want to help our users. We were surprised and excited to learn that expectations of user-centricity and user research practice are generally high, and that there are big opportunities, especially with better knowledge sharing, to build strong user empathy across our organizations.
Your Thoughts on User Research
We’re curious what you, the product community, think about these directional findings.
- Have you found successful ways to reach and maintain your own goals for user understanding? For example, Jet.com keeps an NPS feedback Slack channel that the product (and many non-product people) subscribe to.
- Have you found successful ways to get your non-product colleagues educated about and empathetic towards your users? What tools work for you?
- Have you found ways to make B2B (or any non-B2C) user research easily executable at scale?
- Other than usability testing, interviews, and surveys, do you and your team use another user research tool regularly? With what type of software/setup?
We’d love to hear about your experiences — please share in the comments area!
