User testing for uncommon interfaces and conditions: concrete examples
Purpose
When you design a product interface, you need to test it often with users at different development milestones to check if it’s usable, satisfying, and how to improve it.
A lot of articles explain user testing, why doing it and how, so I won’t expand on the theory. I will relate concrete user tests we did on uncommon interfaces (Wii video game, humanoid robots and HoloLens software) or in uncommon conditions.
Participants recruitment
Whatever the interface is, recruiting participants is a tedious task: you need to find volunteers who fit your users targets (children/adults/elderly persons, novices/experts in a task, B2C/B2B, etc.), and available on the right time at the right place. It can be even more challenging depending on your product.
Confidentiality
Sometimes, and especially in emerging technologies, you can have confidentiality issues. They can be more or less constraining: you may just need to make participants sign a NDA or you may be completely unauthorized to show the product to people external to your company.
Usually you can’t do the user test outside the office, you have to make participants come.
Before Pepper announcement in June 2014, it was highly confidential, even inside the company. Windows were covered and robots had to be hidden under a big fabric to be moved inside the building. Our only solution was to user test with new employees under NDA on their first week; to make sure they didn’t already know the robot. Fortunately for us, we were hiring a lot by the time. Of course, during our user tests we kept in mind that the full targeted population of our product wasn’t represented by our participants, but it was better than nothing.
Sometimes you have to make hard choices in your methodology, the important thing is to be aware of their consequences on your results and conclusions for the product.
Rare target population
With Minsar on HoloLens, we faced another problem: one year ago, there weren’t a lot of companies owning a HoloLens. Since our software is designed for professionals using this device in their daily work, finding participants was nearly impossible. We chose to recruit participants in a neighbor company and to train them to use a HoloLens before the user test. It’s not perfect but it answered a lot of our questions. If noobs manage to use your product, you can predict that experts will too; still keeping in mind that you can’t be sure or predict anything for power-user features.
Now that HoloLens is more common, we can directly go to helpful other companies using this device, for participants to be in their daily work environment.
Natural environment
During user tests, participants are usually impressed or even stressed. They don’t behave naturally. You can try your best to make them comfortable, but one of the greatest solutions is to go directly into their life.
Just Dance (video game on Nintendo Wii)
While working on Just Dance video games, I asked participants to do a hard thing in front of unknown persons and without an appropriate context: to dance!
We tried to be welcoming with participants when they came to playtests in our dance room, but the Just Dance natural atmosphere wasn’t there. Therefore, with my colleague Alexandre Chamoy, we decided to go to participants’ home. Volunteers were asked to invite friends for a Just Dance party at their place, we brought and installed all the material (the Just Dance prototype, snacks, drinks, even a TV once) and let them do what they wanted; us sitting discreetly at the back of the living room to watch and take notes. We did this several times, with adults, children, families, it was very interesting. However, it’s very costly in terms of budget and time, we can’t afford to only do playtests like that, but from time to time it’s very relevant.
Go meet your users in their daily lives to get a more genuine user test environment. It’s obvious but hard to really do.
Pepper and NAO (humanoid robots)
At Aldebaran (now Softbank Robotics), we mainly worked on B2B robots to put in shops, schools, hospitals, public places, etc. Thanks to its size (58 cm / 1.9 ft) and weight (5.4 kg / 11.4-lb), NAO is easier to move than Pepper, so we were able to bring it in relevant places.
One of NAO’s goals is to be used with children for education. What is the best place to find children in this context?
We asked a primary school in the neighborhood if they were interested in having a NAO in two of their classes for an hour to help us. They happily agreed, so we went there: one of us with NAO and the pupils, and two of us at the back of the classroom to observe and take notes. At the end, we gave them and their teacher a form to fill in and ask them to draw the robot. We learnt a lot from this experience: how children naturally behave in their classroom with a robot in front of them, what were the differences depending on their age (there was one class of 6 years old and one of 9 years old), how they perceived the robot… The school staff and pupils were very excited by this experience too. A real Win Win!
Never hesitate to ask external organizations for help, it doesn’t cost anything and it can give your project a lot!
NAO can also be used in hospital or retirement homes. A study was run, with our partner Strate College, on the perception of robots by elderly persons. We went to a retirement home, beginning with individual and collective interviews, and forms. Then the session finished by bringing a real NAO. The differences between before and after this moment regarding their attitudes and opinions towards robots were amazing. Even the most aggressive ones about robotics were touched by NAO and wanted to interact with it. A picture or a video isn’t enough, showing your product is the key (if it already exists, at least a prototype).
Don’t rely on interviews and forms based on a concept to assess attitudes towards your product, always confront it to your target population.
Last example about robots, a supermarket wanted Peppers in one of their shops. It’s quite difficult to test specific features in an uncontrolled environment. Directly putting untested robots there was out of the question.
My colleague, Sandrine Tourcher, designed a user test on the field mimicking the real conditions of the supermarket:
- Five Peppers were put at different spots with a dedicated content.
- A list of specific items to get was prepared for participants to move around the place like real clients would do, to recreate supermarket environment flows.
- A quiz to answer thanks to the five robots was also given to make sure participants interacted with all Peppers.
External participants (families, couples, single persons…) came on Sundays, when the supermarket was closed. They only had the shopping list and the quiz, and were free to do what they wanted in there during a limited time; including interacting with the five Peppers. We were at least one observer by robot, and discreet video cameras were used (with participants’ agreement) to get maximum information. These user tests were very rich. Thanks to their results, the team was able to improve robots contents before the release in the supermarket, to offer a better experience.
If you can’t observe users in real conditions, try to recreate them for your test.
Different language
Your target population can be totally different from you. At Aldebaran / Softbank Robotics, I had to make a user test on the same feature both in France and in Japan. I don’t speak Japanese at all, but, since it was about the dialog with the robot, I wanted to make it in the participants mother-tong language. Fortunately, I had a fully bilingual colleague: Yufo Yukuda. I trained him to run the test: explaining the methodology chosen, the scenario, the material, the importance of the choice of each word he or participants will say, how to behave with participants to avoid biases, etc. Then we went to Tokyo together and thanks to him, we gathered all information we needed. With this solution, I’ve been able to clearly confront the results of the two user tests regarding the country, and Yufo was glad to learn more about user testing. It’s always a team work, each of us being more or less involved depending on the methodology and our job.
We can’t always run user tests by ourselves, you can rely on your colleagues if you offered them the right preparation.
Conclusion
To design user tests for uncommon interfaces or conditions, be creative and flexible. Your product can be confidential, too big to be moved, just a prototype, or require a rare population… There are always solutions. You need to find the best way to get the maximum relevant information with the means you have, respecting a correct methodology and being aware of biases.
Thank you for reading this article, I hope it will give you great ideas for your future user tests! Please 👏 👏 👏 if you enjoyed it and feel free to share your experiences or ask me questions.
If you want to know more about UX Design, you can read my previous articles:
