Should product managers do user research?
If product management literature agrees on something, it is that great products match customers’ needs. Fine.
Still, countless product managers don’t have a clear idea of their customers’ needs because they don’t study them.
Once a PM at ManoMano, I was no exception. Sure, when ManoMano was an early-stage startup everyone talked to users. Back then we even had a “phone-roulette” customer service and every employee including founders answered customers requests. As the company grew, engaging directly with customers became harder. We created a customer service in our offices, then it was relocated to different offices and then spread in different countries to extend our service hours. Customers’ voices became quieter as product managers got further and further away from users.
Today, we try to restore the bond between product builders (designers, product managers, developers) and customers through research.
After 10 months of doing research, we took a step back and today we share our main learnings:
- Product managers don’t make time for research
- Research is not quite plug-and-play
- Product managers need backup to do research
- Maybe product managers should not do research at all
Product managers don’t make time for research
Product managers at ManoMano had to deal with intense scale-up and growing pressure on delivery which kept them in their day-to-day fires. They were unable to make time for research, they lost the connexion with users and, therefore, weren’t sure they were prioritizing the right features to improve the user experience.
Intense scale-up
Between November 2017 and November 2018 we’ve hired 200 people, that is basically the two-thirds of our staff that got onboard in this period. It’s like hitting the restart button for each arrival; PM were challenged on their priorities by each and every new stakeholder. I am not saying that it’s a bad thing, it is sanitary, but it sure takes time and effort.
Apart from stakeholders, the product team grew 8x in the same period of time. You can read more about our learnings and tips to achieve such scaling here.
In short, product managers had zero time left to talk to and observe users since the company priority was to enlarge the team and make it work as the team grew. And you know what? It’s fine. First, build the army, then get the work done 💪.
Pressure on delivery
While scaling, PM still had to deliver their roadmaps and, as more money was invested on feature teams, the return on investment was even closely looked at. Scrum has great tools to monitor your delivery, it brings transparency with burndown charts and challenges the team on their velocity. We needed this shift to make sure we were on the right track.
Nevertheless, back then, our technical environments weren’t stable, it took sometimes hours to deploy code and hours to have a proper QA on features so velocity seemed like a harsh key performance indicator… Morale went down as uncertainty grew. At that time, I was still a product manager and it was hard to stay optimistic and inspired by a user vision when all you do it trying to fix problems to ensure delivery.
I wondered if I was working on the right things and, more globally, if we were focusing on the right problems as a company. Sometimes, I would just prioritize quick wins because they were easier to specify and to test. Sometimes, I would just add tickets to the next sprints just because they were already specified, not because they truly mattered to users.
And then, it hit me.
“I know I don’t deliver many points, my past sprints failed, but once our project is in production, I know we’ll have more impact on the user journey than ever.”
— John, product manager.
We had lost touch with our customers and frankly, we had no idea if we were making the experience any better for them.
How could we restore this connection with them in a long-lasting way? Turns out, if research is quite easy to kick-off, it is not that easy to scale…
Research is not quite plug-and-play
Sure, you can always grab your phone, dial the number of one of your customer and just listen, but let’s face it, research is much more powerful when you have a plan. As we wanted to get a clearer idea of what we should do to reconnect with users, I discovered the job of user researcher (UR).
It seems really trendy in the US… like product management was a decade ago.
What does a user researcher do?
- She recruits users by phone or email
- She writes test protocols, surveys, interview guides, etc…
- She gets out there and talks to users!
- She watches user sessions, listens to past interviews
- She analyses and synthesizes findings into a research recap
- She participates in meetings with designers and PM
As a PM, there was NO WAY I could add all of this to my agenda 😱. It was already filled up by tickets specifications, sprint preparation, meetings with stakeholders, functional tests, demo, prioritization and roadmap preparation.
Ok, and what about our UX designers? Should they do it? Well, yes they could if they weren’t outnumbered. We only had 3 designers in the team back then so they were focused on delivering workflows and screens to the 13 feature teams…😅
That’s when we decided to open a full-time position for a user researcher. I got the job and frankly, switching from product management to research came with a lot of erasing what I knew and making space for discovery. This is why I realized the following…
Product managers need backup to do research
If a PM is truly valuable for making sense of user problems and building a vision towards them, he is absolutely not the person who should orchestrate user research.
Instead of spending time profiling users, scheduling appointments, picking the right tool depending on the type of research or lead the interview; he’d rather take an hour to brief a user researcher to do all of this and sit quietly in the room while users are being interviewed.
How do interactions between a PM and a researcher work?
First, let’s set some ground rules:
- A researcher only works on what has a high priority to maximize the chances of his conclusions to be tested in production.
- Good research is always driven by goals and limited in time.
- Serendipity is NOT searching without a goal, it is surprisingly finding something unrelated to this prior goal.
- Time limits the type of research you can do. Just agree on the depth of the study at the beginning, knowing your constraints as a team.
Here is the framework, created by Facebook Research we use to frame our research:
Here is what is shared between the researcher and the product manager while filling-up the framework:
After the kick-off, we integrate this framework in a document called the Tracker, that will be shared with all the participants in the research and its outcome. The tracker has all the documentation, next steps, roles and information about the project.
From here, my role as a UR is to write the research protocol, dig for customer insights and prepare a findings and recommendations document. In order to be faster in the recruitment part, I’ve gathered a community of testers (+600 users, 50 top users), easy to reactivate once we have something to share with them.
Still, since there are 13 feature teams at ManoMano, I cannot help all the teams and we won’t wait to hire 12 user researchers to do research! So yes, product managers do research, they call customers, they ask questions, they write conclusions of what they learned. The strategy was: do research, in any possible way that you can.
Hmmm… 🤔 But what about biases?
“To say a team can be immune to bias is hopeful at best. To say that a team can be immune to bias whilst also being invested in the success of the product is myopic.” (see full article here)
When you control the output and the outcome, how can you keep intellectual integrity?
Maybe product managers should not do research at all
We had a big conversation with the CPO; as a former PM, it was tough for me to admit that the product team should not explore customers’ problems by talking directly to them. But, I had to admit, they were struggling with their biases and this could put the entire research in jeopardy.
To limit the risk of biases, we worked on a presentation to train our product managers to do research correctly, here are the main takeaways:
Research is not about confirming what you know, it’s about discovering what you don’t.
- Before starting the study, write down your intuition on the topic, this way you’ll separate from it.
- Write down your personal context and the biases it creates (find all cognitive biases here)
- Don’t take notes during the interview, record all of it. (ask for permission of course)
- While asking a question, JUST ask the question, DON’T give ANY possible answers. Ex: “How do you usually contact an e-commerce website?” DON’T ADD: “By email, by phone, or …” just to make it sound more like chitchat. It limits the possible answers.
- Always ask OPEN questions, not YES/NO ones
- Ask short questions
- Don’t use the negative form in question
- Embrace the silences, don’t break them
- Give neutral signs of understanding like “hm hm”, “OK”, “understood”, “very clear”
- Ask WHY or even if the answer sounds straightforward or that it is silly to ask. Or just say “tell me more about that” to dig deeper.
If a PM conforms to such rules, they can limit the damage of their biases. Nevertheless, they are probably the most biased people to conduct an interview.
Another option you should really consider if you plan to do research in your company is to hire a freelance (or agency) user researcher.
Here is a Pros&Cons list:
PROS
- No personal interest in developing feature A or B
- Fresh-eye on your product
- New ways of working and a new professional culture
- She is already out of the building, it’s time for your team to get out too!
CONS
- Risks of misalignment increased, make sure the mission is clear, don’t assume she knows.
- You need to have a dedicated budget and it’s hard to pay someone for “just spotting the problems” not solving them!
- A potential lack of understanding of the product and the user dynamic in the first place.
So, should PM do research?
We strongly recommend that people involved in the outcome don’t lead the research. The entire legitimacy of your product team depends on its ability to know its customers and to back-up their roadmaps with user data and verbatims. Don’t waste time on biased research, train PM to fight their biases at best but, if possible, just hire someone to do it!
Oh yes! One last thing, we are still working on how to best do research so if you have any innovative way to do it or any suggestions, feel absolutely free to leave comments or ping us on Twitter!
Thank you for reading, subscribe for more articles on user research!
