Tips for Qualitative Interview/User Study/Testing
I've done 4 qualitative user testings and participated in quantitative one last year. There are some tips and tricks I want to share.
It is really important for the whole team to be on the same page about what we are trying to evaluate though the testing/user study. It sounds obvious but sometimes people are just rushing to see real users reaction in the actual environment as soon as the minimum viable product/prototype is ready. This would often result in a collection of feedback so that shines no light on where the development should go next.
Usually the product/project managers or the executives would have a very clear road map of the product/project. The team just need to set up a dedicated meeting before deciding the form of the user testing, to agree on the things we are trying to evaluate at this stage. Therefore we craft the user testing process to avoid the potential emphasis from the users on things that temporarily less important.
In one of the user study I experienced, our team wanted to know how several types of content would play out on different device form factors. we constructed the study process so that navigating within contents plays a minor rule, driving users attentions away from it. Our findings turned out to be very focused and directive.
Plan ahead and in details, make sure all the logistics and millions of other little things are covered (i.e. Batteries are fully charged, enough spare devices, pens and paper). Also make sure the facilitator, moderator and note taker are very familiar with the plan and the interview guide, and try always stick with the plan! I have experience of team members Who didn’t follow the guide during the study, asking leading questions. It made the user’s respond not reliable in terms of gathering findings.
Always have a plan B. So many thing can possibly go wrong: prototype not working, weather condition changed, weak network signals, etc. And that’s not yet taken into account things from participants side: absence or late show ups that cause sessions overlap.
We need to make sure we have alternatives if accidents happen, which means trying to cover as many scenarios as we can imagine while we are planning and be flexible during the study/testing.
Don’t forces yourself too hard if you happen to have discouraging participants. It might be because of the perfunctory behaviors or answers, unwillingness to cooperate or simply a bad mood striking them. Sometimes we need to give up on the session knowing that it won’t generate useful insights. Pivot the conversation and try to smoothly end it earlier. Save yourself more time for other sessions.
Be agile during the days of the study/test. Organize notes every day. We usually migrate them to a digital file (Word or Excel), putting them under the questions or sections that they belong to. Don’t leave everything to the end because things would start to get messy and it would be too much to handle.
Be mindful about sharing and discussing personal thoughts right after each participant, or at the end of everyday. The study is not complete yet and we are not seeing the whole picture. The thoughts are also too fresh and therefore usually very emotional. For example, if some feedbacks from participants happen to prove your assumption, you would get too excited and unintentionally exaggerated the importance of it among other insights and findings. It is not wise words to try to reach some conclusions so quickly. Let the organized notes sit for a day or two. Then gather the team again to go through notes and extract findings.
Keep the report short. For better or worse, most people outside of the team don’t have the time or patient for a long report. It is always good to make the first page of the report a bullet-points style summary.
Almost all the tips I mentioned above are requiring people to think twice before putting up the study/test. But the one last tip I want to provide is actually the opposite: don’t wait until everything is ready, because they will never be fully ready. It’s better to do quick and light-weight versions than long and complicated ones.