Language barriers in user testing?

When it comes to user testing our products, we have previously faced problems with language barriers. Over 68% of our users are international students. The majority speak English, but a high number are not fluent. Especially those who are in their first year and may have only come to the UK for the first time a few weeks / months ago.

If our users didn’t speak any English at all, there would be a very simple solution to hire a translator for user testing. But our users will be speaking English and we must get the most honest feedback we can, whilst the users potentially not being able to explain themselves fully.

Jakob Nielsen uncovered some interesting points in his 2001 article: ‘First Rule of Usability? Don’t listen to users’. Through research and analysis, he found that users will give you spoken feedback on what they think you want to hear, especially in moderated environments. Also, that users often think they know what they want from a product — they may pose solutions to problems which may not be accurate.

In summary:

“pay attention to what users do, not what they say.”
- Jakob Neilsen

It transpires that when it comes to user testing a product, spoken feedback is not always the most reliable source. Especially in a moderated environment.

The initial assumption of language barriers being a problem seems to not be as much of an issue as originally thought.

Observing body language and tone of voice will give us more qualitative feedback and a more genuine reflection as to how a user is feeling. Studying hand gestures, facial expressions and emotions are all subconscious reactions from the user and are therefore a completely accurate source of information.

Relevant emotions to observe during user testing include:

• Confidence — Smiling, hands in front of stomach, enthusiasm
• Confusion — Reserved, scratching top of head, squinting, leaning forward
• Frustration — Scratching back of head, scratching, tapping / shaking
• Nerves — Closed posture, touching / protecting neck

If users are content and confident, this indicates that they are having a good experience with the product. If users are demonstrating signs of confusion and frustration, we should either ask questions as to why this is occurring, or analysis their body language in more depth to get to the root of the problem.

When analysing body language, we need to ensure gestures, emotions are consistent across users. We can’t define one users feedback as a given as there are many factors that could be swaying feedback. For example, the user’s mood, moderated environment etc. It will be important to take note of initial perceptions of user reactions, but also to analyse video footage in more dept.

To conclude, user testing should be focused around the way the users use the app, and supported by spoken thoughts on usability — which may not be accurate. Fluent in English or not. However, initial conversation with users who are fluent or not is still necessary to allow the user to feel comfortable.