Industry user research
lessons learned from conducting user research in industry - the ones that cannot be taught in school
How can infrastructure be better managed, footwear be better designed and career coaching be better provided?
I have explored questions like these at VMware, Autodesk and Women Who Code. These problems have not only improved my user research skills and but have taught me invaluable lessons on empathy, creating impact, maintaining composure, building trust and relationships, making peace with smallest changes and ambiguity, persistence, learning, teaching, storytelling and many more. This article is an attempt to share the lessons learnt and tips that worked. Wherever possible, I’ve given real examples (obfuscating the confidential details).
Every research project is different but they more or less have had the following key phases. The phases aren’t always in this order, and there is usually a constant back and forth between certain phases. The phases are also tuned depending on the type of research project (generative, formative or summative) and the available time and resources.
Phase #1 Research request + Stakeholder meetings
A research request could come from multiple sources — product manager, designer, engineering backlog or may be it is a follow up on a new topic which was discovered in an earlier research session. I have 1-on-1 discussion with the requestor to understand their goals, questions, expectation from research, concerns and implementation timeline. I do make it a point to ask for good resources to help prep up on the domain knowledge and competitor products.
Phase #2 Desk research
The essence of this stage is to determine if research is needed / possible within the given timeframe or if existing knowledge could be leveraged.
If it is an unfamiliar product, I begin with understanding the high level picture — its users, mission, stakeholders, business goals, technical architecture and design language. To understand the product better, I use the product, read documentation and speak with engineers. It helps in asking the right questions.
Customers tend to use multiple VMware products to manage their infrastructure. Even though the focus of a research session would be on one particular product / workflow, users do talk / ask questions about other relevant touch points. It is crucial to be as knowledgeable as one can because A. it helps to understand their comments better and B. answering the user’s query gives off a good impression and the user becomes more engaged and open.
For example, two of VMware products (platform tool, management tools) exposes Containers technology. In a usability test on containers in the management tool, I was asked questions about the platform tool. I then got information about that part which helped me answer in the subsequent research sessions.
In case of existing products, I often draw an entity-relationship diagrams to better understand the system entities. In case of new product use cases, I study the competitor products if any.
It is essential to determine if research is needed (previous research, not enough time).
Phase #3 More stakeholder meetings + Prioritisation
Once I have basic understanding of the problem and question areas, I formulate questions and get those reviewed early by the relevant stakeholders (mostly PMs, engineers and UXers) to ensure correct understanding of their concerns and questions. I have found this to be a good time to ask product / engineering questions to think through the problem space and corner cases.
Most of the times, the number of research questions are more than that can be handled in the given session time. This turns out to be a good time to set realistic expectations about the available time.
Factors that could help prioritise:
Level of contention
In person / remote research opportunity
other research requests
For example, for a new tool in the infrastructure management spectrum, the product management and the engineering team had different views on the power dynamic amongst system user roles. We prioritised this because this topic was highly contentious and fundamental.
Especially when working as a consultant researcher, it is important to set the right expectations and be aware of every possible task. For Women Who Code project, I didn’t allocate any time for unanticipated items such as time required to reschedule sessions or housing keeping items such as anonymising data and moving files.
Phase #4 Secondary research
To understand the problem better and ask better questions, if time permits and if needed, I do some more ground work by speaking with internal users of the product (if any) or technical account managers (TAMs). TAMs have lot of information about customers and their pain points. Customer support tickets are an additional source of information on past issues faced by users.
I usually narrow down the ticket list through keywords and function areas.
There are also some non-traditional sources for doing secondary resources. In Autodesk, I went through job descriptions and interviews of footwear designers to get an idea of their work and better prepare the interview questions.
Phase #5 Creating the research protocol + Recruitment
Limited niche users, multiple user personas, sale teams being protective of their customers, buyers being different from users make finding and recruiting the right research participants for B2B products a really long process. Add no dedicated participant recruiter to that list and you will get a recipe for a challenging process. I usually create the research protocol and recruit users in parallel.
Once the higher level research questions and timeline are more or less finalised, I determine the appropriate methodology and the type / number of participants required.
After phrasing the research questions / activities, I make sure to get as much feedback as possible from the the involved stakeholders and other researchers. It is useful because — it helps build a better relation with stakeholders, create a better engagement, introduce more transparency. It ensures that the original intention of the questions is not lost or that the right success or failure metrics are being captured. It also helps to receive feedback from someone outside the project, because one might end up using project lingo with which the user is not familiar.
The most important check I do at this stage is to ensure that I’m not asking an unnecessary question. My trick for doing that is to imagine what kind of design / product decision would be enabled with the received answer. If it doesn’t, it is better to not ask that question.
During my initial days at VMware, I used to ask the users to list VMware products in their eco-system. I thought of it as a a good warm up question. Instead, I found it digressed the conversation with users talking about pain points in other products; leaving me with no actionable data.
Also important is to plan how to capture the required data and delete the unnecessary one.
For instance, in one of the usability studies at VMware, we had only one instance of the test system. We wanted to retain the system objects created by participants for later analysis. But the object names would have biased the next participant. I set aside some time in between sessions to rename the object names (noted them somewhere else.
Once I know what type of participants are needed, I go through a list of users from the past research sessions to find suitable candidates. This list has their contact information and key details such job role, location, VMware products used, last time contacted and so on. Knowing when they were contacted helps avoid contacting them frequently.
For finding new users, my go to sources are
Technical account managers (usually protective of their customers)
VMware social community open to user
I sometimes do a quick phone screen with the new users to understand their background, vet them if needed and schedule the research session. In case of time crunch, I have found Calendly to be super useful to help with offline scheduling.
Customer conferences are a good opportunity to conduct research and preparing for those is a different ballgame all together. But I’ll leave that for another article.
Phase #6 Arranging session logistics + Dry Run
I ensure that audio or video calls, recording work fine, non-disclosure agreements are in place, venues are booked and supplies required for in person activities are procured. Also, most importantly in case of usability tests, I ensure that the system (hardware/software) are working and have enough sample data. While doing research offsite, where I’m not sure of the internet connectivity, I keep an offline copy of resources required.
Not everything can be anticipated and a dry run helps smoothen out the kinks. I dry run the test script with internal users if possible to ensure that the questions are understandable and scripts fits the scheduled time. It helps in ironing out the kinks.
Phase #7 Conducting the research
There are quite a few learnings in this phase that I have acquired through hands-on projects and other researchers. The most important guideline would be to make every research session more of a conversation with help of an empathetic heart, keen ears, thoughtful mind and a poker face.
I read the past research notes of the users (if any). That information helps in forming the personal connection, avoids asking repetitive questions and gives a good start to the conversation. The users always feel better when you remember their past problems and it is even better if you are able to tell them that those problems were solved.
It is crucial to keep the discussion on track but there is also value in serendipity. My fellow researcher says, “the best insights come up organically”.
Another lesson I have learned is to go to the root cause of problems (techniques such as 5 Whys are helpful) and bringing the user’s mindset to “what should be” instead of “what is”.
Having enough time in between research session helps smoothen out the questions and do quick reflections. Reflections (either by self or in a group) ensure that important data is not left on the table, speeds up the analysis process later and helps validate what was heard.
I invite the stakeholders especially engineers to come observe the session. They come more close to the user than the usually customer support tickets, it adds more credibility to the process and they feel more involved. Plus, additional notes are always welcome. :)
Phase #8 Analysis
The time required for this phase depends upon how well the notes were captured in the session. I usually start by going through the top three takeaways from every session. They often turn out be the most crucial insights. For the rest of the notes, I use affinity mapping technique for clustering and deriving common patterns.
Once I didn’t have much time to capture the data from the notes onto the sticky notes. I just printed out all the notes and used different coloured marker to highlight patterns such pain points, good points, recommendations, concerns and so on.
If there other product team people in the research session, depending on their availability I try to involve them in the analysis phase to cross validate the findings.
Phase #9 Conveying the results and creating an impact
This would be the most crucial phase of research. After all, as my manager says “Any research is as good as the decisions made”.
Irrespective of the research findings, the goal at this phase should be creating empathy about the users in the product team.
It is important to figure out when / whom / how and what to tell and also who should tell
9.1 When to tell -
As my ex-manager used to say — “You should have told them yesterday”. Sprint planning doesn’t stop for anything and it is better to let everyone know the feedback as soon as possible. It then has a greater probability of being captured in sprint stories. After all, in agile, even one week can seem too long.
Reminding the team of the past unimplemented research findings (if it is an appropriate time to implement those) is as important as telling the new insights.
9. 2 Whom / What to tell -
Audience can be bucketed in two — strategists and the implementors. In the past, I have found engineers not being that interested in strategic research findings as in actionable insights. Since then, I tell more of the tactical findings to the engineers and strategic findings to the executive and product management team.
“What should we do” becomes more valuable than “What did we learn”. That is how I try to phrase the research findings. One useful advice (given by an executive) to come up with an actionable recommendation is to do a “so what” analysis of every insight. Another point to be remembered while phrasing insights is that it should sound that it came from the user and is not the researcher’s opinion. Phrases such as “It was observed from users that” or “The users said that” help achieve that.
If time permits, it is useful to convey the actionable insights through rough mockups rather than simple text.
9.3 How to tell -
Unlike, what I was used to in school, this part isn’t as clean or simple as giving a single presentation and report.
In school I was taught the persuasion technique of “Logos, Pathos and Ethos”. Simply put it means convincing through logic (logos), emotions (pathos) and credibility (ethos). I truly assimilated the ethos part when I started working in the industry. Until then, I had used quantitative and qualitative data to convey the insights. After working with a team for a long time, I realised that they have started trusting me more and became more likely to implement the recommendations. To build credibility, I talk about the process a bit in presentations.
Once I didn’t have space to do affinity mapping using sticky notes. I ended up my desk whiteboard. To my surprise, it generated curiosity amongst my teammates.
I have found impromptu conversations about insights to be more effective than presentation and reports. These conversations can happen in a break room or in a planning meeting. For that one has to be ready with insights to be told through short stories and without a slide deck.
Another way of delivering findings is through a brainstorming session. The insights act as a starting point of what could be done next. For the Women Who Code project, this method worked well in getting the team put their thinking caps on. The key is too have an optimum number of people to have a focussed discussion.
For later usage, I document the findings / recommendations in a medium that the team is comfortable with. (Wiki pages or JIRA tickets)
9.4 Who should tell
The most effective person to tell the user’s story is the user them self. Customer conferences and off sites are good opportunities to make direct connections between users and the product team.
If in person connections are not possible, in the past, the UX team at VMware has encouraged users to post feedback on online social forums accessible to the product team.
Sometimes no matter how dearly you feel about a recommendation, the engineering team just doesn’t have enough bandwidth to implement it. It hurts but then I remind myself that a product is never delivered in a big bang. It is always incremental and as long as every increment is a complete story in itself and is adding to the bigger story, it is okay to let go of a recommendation for some time. Keeping the big picture in mind, it is a skill to carefully guide the team in every increment towards it. Choosing battles is crucial but a difficult knack to acquire. I’m still learning.
Some final thoughts
Creating and measuring impact
I have always found measuring impact to be tricky. I’m still learning how to measure it. Super direct indicators like NPS are not always available and neither do they tell much. Some indicators that have given me satisfaction in the past (not sure if they are good measures of impact) are a connection made between an engineer and a user, a smile on the user’s face or a user pointing to something on the product screen and saying “Hell yeah”.
Finding opportunities to research
There multiple opportunities to learn about the users; even though they are not led by the UX team. For example, VMware UX team leverages on company conferences to conduct on-site research. In the past, I have asked product managers to invite me to their customer calls.
Building good relations with co-workers and customers
I cannot the importance of this enough. It hugely impacts all other phases.
Whether it is a TAM / product manager who would help you recruit the next participants, or the facilities co-ordinator who would help book the next venue, or customers who feel that they are valued and then reciprocate with their time, or UXers who would give invaluable feedback or engineers who are going to implement that much needed change, everything is people dependent. It helps to mindfully cultivate those relations through genuine interest, reciprocation and trust.
Learning from others
Hands-on projects have taught me a lot. And I have also learned a lot from researchers at work and outside through
Informal coffee chats / lunch time discussions to understand other products
Getting feedback on research plans
Shadowing them in their research sessions and helping with notes and analysis
Attending their presentations
Slack channels — mixedmethods, ethnography hangout, hexagon ux
Google group for design user research
And yes, dscout and medium articles!