Learning about the highs and lows in user research

Sharing my key takeaways from the User Research for Everyone conference

User research continues to be an underrated practice within organisations. Organisations want the latest in innovation or the shiniest user interface. It’s not bad to want these things or be motivated by progress. Yet, user research makes sure if organisations aren’t wasting their time building unnecessary things.

I can empathise with outsiders about their misconceptions about research. Maybe people don’t understand what research does. Maybe they can’t see the immediate financial benefit spending your budget for research. As such, as UX practitioners, we continually face challenges with people so that our impact can take effect.

A few weeks ago, I was fortunate to take part in Rosenfeld Media’s User Research for Everyone conference with the help of Ethnio. With 8 of the most respected user research experts, the conference was a learning experience in how to tackle the various challenges in embedding a research culture.

Each speaker had motivating and insightful thoughts. I wanted to share what struck me. Below is my summary and key takeaways from each speaker.

Erika Hall: Just Enough Research

“Be proven that you are wrong all the time. Not that you are right. Learn and be command of the facts”

People have a natural satisfaction in being right. It may be helpful when we solve things for ourselves, but it doesn’t apply when we solve problems for our users. Our determination to be right can alienate us from being empathetic to our users. It also prevents us from seeing if we are making an incorrect solution. As Hall argues, wanting to find the right answer is a good thing, but it is different to already having the right answer in your head.

My key takeaways:

1. The difference between a person with research mindset versus a non research mindset is their aptitude to being curious. In continually asking questions rather than settle for the assumptions, you use research to learn more about your target audience.
2. Knowledge is the bridge between your assumptions and the real world. Assumptions are always risky. Poor designers assuming that they know their user’s experiences and build what they know, a good designer instead works with other people and design for the real world.
3. Admitting that you are letting go of knowing the answers leads to a place of growth and learning. It means reminding oneself that come from a place of curiosity than thinking we know all the answers. A reminder that we are are not our users.
4. Start from a place of agreement with people you want to convince. In reminding that we all care about building a better product, it can start getting people on your side about research.
5. Good questions are specific, actionable and practical. Better yet, the best questions are the ones that explore the unknown and carry the most risk. The more times we explore our context, we are decreasing the risks within our product, our business or from our competition.

Hall’s talk serves as a reminder to listen with an open perspective and challenge our own assumptions to what users need. In having an iterative research process, it brings us closer to building knowledge as opposed to opinions.

Leah Buley: The Right Research method for any problem (and budget)

“Instead we can think of research as a series of questions that build on each other”

In most organisations, research is a practice that exclusive to a specific team. Buley argues that when research is only a department concern, it creates barriers for information to cross pollinate the organisation. It also inhibits people in other departments who may have interest in research to reach out. This is something that I have experienced. Some were open to put in place our insights, others weren’t. People from other departments would not share their own research, which contributed to later conflicts.

My key takeaways:

1. What is happening around us? Understanding the context frames the next opportunities in your existing product. By knowing the latest in competitors, technology and customers expectations, it informs us into how we can built better experiences.
2. What do people need? In observing behaviours and using our intuition, research informs for the problems with an experience and what they do to try to make it better.
3. What can we make that will help? In observing what people do, we learn where we can to minimise their pains or supplying a need. Buley recommends the Rapid Iterative Testing and Evaluation (RITE) method to quickly sense problem areas.
4. Does our solution really work? Conversion flows help measure if people move from an initial download to a continual use of the solution.
5. What is the impact of this solution? Does it help improve our user’s experience, our product or our business? It determines how worthwhile this solution is from a financial, customer and product perspective.

Bully’s 5 questions are building blocks for anyone to sell the value of research to anyone within your organisation. It also shows that anyone has the capabilities to do research even when there is a no budget. By adapting people’s perceptions and questions to these 5 takeaways, any data gathered will inform you towards a thoughtful solution.

nate bolt : How to find and recruit amazing participants

“The more someone that really needs to use your product, the more likely they will give you honest feedback”

Depending on what you are trying to find, there are many approaches to recruiting research participants. I have recruited random strangers for guerrilla research to screening targeted users for 1–1 testing sessions. I have found that preparation is key to make sure you get the best participants possible. Bolt demonstrates the pros and cons behind 8 methods of recruitment to show when each recruitment process suits you.

My key takeaways :

1. The realness, the level of attachment to your product, and the moment of use influences your recruiting spectrum. The more authentic, attached and in the moment a person is to your product; the more genuine qualitative data you get. It is because they have motivations to make changes to your product.
2. Have a initial recruiting checklist. It helps place a constraint and time box to your research to deliver worthwhile results. A checklist also informs important stakeholders of your process to gain initial buy in.
3. Screeners are helpful to filter out participants that do not suit your study. It allows the scope of your research to be practical and specific.
4. From sourcing a recruitment agency to ads on social media, there is no one way to recruit participants.
5. Be aware of the recruiting pitfalls. From no shows, bias, quiet users or different quantitive data, you have to focus on their behaviour and using your gut feeling to adapt your insights.

Bolt’s talk was a great introduction into Live User Intercept, a tool where you can observe remotely in real time for user research. Bolt also was a great reminder that remote research is possible.

Steve Krug & Laura Klein: DIY Usability Testing Discussion

“Usability Testing is a gateway drug to research”

In my experience, there is something immediately informative in observing a customer’s struggle in using a product, or a satisfaction in seeing their delight. Yet, in developing a constructive usability testing practice is still a learning process.

My key takeaways:

1. Decouple the testing schedule from the deliverable. In my earlier testing experiences, usability testing was not conducted until the deliverable was ready, with no bugs. This took 4 weeks between iterations which went against our ideal of 2 week iterations. In that time, the designers formed an attachment to the solution. It became time consuming to fix changes from customer feedback. In simply informing the designers to deliver what they have in 3 weeks, it motivated designers to deliver their best without accomplishing perfection.
2. If you decouple the recruitment from what we want to learn, it limits the type of users that come to testing sessions. From an efficiency standpoint, it makes sense to recruit as many people as possible. However, Krug and Klein discussed that it is better to try and test with anyone, not just your target audience. It allows fresh eyes to observe flaws that you may have unnoticed.
3. Doing a comparative usability testing session helps stakeholders understand the value of research on a competing product. Stakeholders are always interested in what competitors are doing. As you are testing others and not yourself, it does not affect your work. In learning the faults of other competitors, it opens opportunities to understand the issues with your own product.

Usability testing is a great opportunity to learn how people use your product and if it matches your expectations. However, it suited for whenever you are to investigate opportunities for improvements.

Julie Stanford: Creating a virtuous cycle: The Research and Design feedback loop

“Research and design need to work through a thoughtful loop”

In an ideal product world, user research would be equally as valued as the design of a solution. Yet in real life, the demand to think fast and solve quickly means that short scope for research. It creates reactive solutions that aren’t well thought out. The solution can be a patchwork fix, where additions are included but could be solved in a better way.

My key takeaways:

1. When user research is done without a distinct purpose in mind, research can hinder the project. Researchers may not go deep enough with their questioning to arrive at tangible results. Researchers are also privy to falling for assumptions of the user. It can leave the design and product team frustrated that the project is not moving forward.
2. Stanford suggests of a thoughtful research loop. It is where research and design work very closely with one another to understand the context of the problem. The ideal option would be for research and design is done by the same person. If this first option is unachievable, she recommendations the following formula:
   Question → Mind Meld → Attend → Evaluate findings →Brainstorm →Evaluate solution
3. The key to this formula is a joint participation of researchers and designers. In making a plan for researchers and designers to collaborate, it allows both researchers and designers to learn from one another. More importantly, it removes any gaps and misunderstandings in each other’s work.

Stanford’s talk was personally reassuring. Working fast may sound ideal, but if it means working fast in the wrong direction then the work is no longer beneficial. The virtuous cycle provides the building blocks to apply the right insights to deliver the right designs.

Abby Covert: Making sense of Research findings

“ For every minute you gather research, it takes 10 minutes to synthesis the research”

People often forget about the thinking time it takes to interpret and synthesise research. Gathering research can be a bit messy. You may get a quick sense from your debriefs but after collecting so much data, what is the best way to make sense of it all?

My key takeaways:

1. Having someone take notes for you without direction is like note taking for yourself. In my own note taking experiences, I have found it challenging capturing the notes for oneself and keeping a positive engagement with the interviewee. Having your notetaker is so handy, but without guidelines, your notetaker may take notes that may not be useful afterwards.
2. In grouping the data, you can quickly draw out qualities for your persona or even bring additional insights that shape next line of questioning.
3. Don’t over schedule your research sessions. Scheduling the right amount of sessions each day is important for your capabilities and also to maintain the quality of your research. It also allows time to collate your findings to synthesis.
4. How you synthesise your findings will depend upon your audience. It is like creating any other user experience, so understanding your audience mental model will help you understand what research insights that resonate most with them. A helpful table is like one below:

5. Develop a classification system to interpret and apply findings. It means planning the right purpose, the right format and the right elements to best convey your findings.

Covert’s talk serves as a reminder that research needs to be thought with the end in mind. Whether it is a feedback from your problem space or the solution space, the end goal of research informs the necessary next steps. As such synthesising research to affirm or question, those next steps proves useful in getting designers, developers and other key stakeholders on board.

Cindy Alvarez: Infectious Research

You can’t come in and convince an organisation to love research, without them already wanting to love research…You can only lead them to change their own minds

In my experience, research has not always been warmly received by all within the organisation. It is challenging to have continually enthusiastic about research when your stakeholders do not understand nor appreciate your cause. There are times where short term criticism can led to personal demotivation.

Here were my key takeaways:

1. First discuss with stakeholders where the gaps in our knowledge exist. It bring everybody on the same starting point to challenge all of our assumption. It also open everybody up to be willing to learn what users say.
2. Find and create the opportunities with stakeholders that allow you to lead by example. In researching on problems that are commonly known, unsexy and you have the magic touch to deliver, your results will demonstrate how valuable research is.
3. Lower your stakeholders resistance to research by speaking their language and doing what they do. In getting them thinking that it was their idea all long may not give you the initial credit, but it allows them to get familiar with the research process.
4. Win over your stakeholder with next steps and recommendations. Without it, your research dies. By hooking in your stakeholders with actions, it provides the strategies needed to starting solving your problem.
5. Amplify the impact by repeating the process and make people feel good about their involvement. Repeating the process will increase their exposure to be more familiar with how research works.

Alvarez’s talk reminds me that research is a continuous ever-changing investment where its value to solving the next problem will shine in the long term. Some results can come quickly but as research aims to know all about the customer, taking the time for research can deliver something that more meaningful or prevent a wasteful solution being built.

Thanks for reading! If you enjoyed it, click that heart button. Thanks again for Ethnio to allow me to participate and I hope others can utilise this post in their own research practices.

Subscribe to The UX Blog