Bridging Language and Culture Gaps Across the Pacific — a User-Research Perspective

Below I am conveying the experiences I had when I conducted our first user research sessions with our Non-English speaking Japanese users.

I had the opportunity to organize and conduct user research as well as test sessions with our Panasonic users in Japan for Arimo’s artificial intelligence predictive maintenance project I am working on. In short, PM (Predictive Maintenance) helps determine the condition of in-service equipment in order to estimate when maintenance should optimally be performed. This approach promises cost savings or time-based preventive maintenance because tasks are performed only when warranted.

To conduct this research, we followed all parts of the research process: from identifying how many users we would be interviewing, choosing which locations we should visit, clarifying who all should be involved in reviewing and approving the test script, etc…

We had all the test scripts ready, the travel budget approved, our calendars were aligned with our users, however, the biggest logistical hurdle we encountered was linguistic in nature. Who would help us translate? 95% of the users whom we were interviewing spoke only Japanese. We did not have any extra room in the budget to have a translator travel with us to Japan or to hire one while we were in Japan.

We had hit a pretty big roadblock. Not one of us spoke Japanese and none of us had any prior experience interacting with Japanese culture from a research perspective. The question now became: ‘how could we conduct effective research sessions with our Japanese users so that we wouldn’t miss out on the cultural nuances?’

For the cultural aspect considerations, there was no way we could have learned everything we needed to learn in only a few days. Given this fact, it was obvious that we would have to experience the culture first hand and would probably be required to tweak the research process and questions in real-time. One of the things I did do to prepare was to talk to some of the stakeholders who have had prior experiences interacting with Japanese business folks, which gave me a good starting point.

The unsolved question still lingered though: who would do the translation? I started to look at all viable options with my product team and we brainstormed the following options:

· Have the user’s boss sit in during the session and help to translate.

· Use Google Translate to translate the test script and user sessions.

· Send the test script to the user in advance so that the users can prepare for the session in a manner better suited to them.

· Have one of the project stakeholders translate for us.

We spent time evaluating each of these options and we ended up using a combination of them during the interviews. After doing the sessions, we did a retrospective of our experiences during the interviews and determined that we should NOT do some of the following things during future research:

1. Have the User’s Boss Sit in During the Session to Help with Translation:

We learned that Japan is a very hierarchical society, which consequently means that sitting along with the boss is definitely not the most beneficial approach in this situation. We found that users always were concerned about what to say and what not to say and would be very hesitant to express themselves fully. We determined that the feedback we would get from sessions like these would definitely be skewed.

Experience: One of the junior users we were interviewing (a recent college graduate) was volunteered by one of the senior managers within a group in Panasonic. She came into the interview looking very tense. We started by giving her a run-down of what she should expect. During the interview I could see droplets of sweat trickle down her forehead. Things progressed to such a point that we had to stop the interview and take her out to get some coffee so that she could relax and also get to know us better. After some general chit chat, we had to explain our activity again and had to stress upon the fact that she had nothing to worry about and that whatever she said would be kept within the product team. When we got her back to the room, we observed that she was still a bit uncomfortable.

Lessons Learned: We later learned that her boss had asked her to do this interview and she understood it to mean that she had to ‘pass it’ and she was also under the impression that the results of the interview would be shared with her boss. This was a key takeaway for us. In the future, we discovered that we should be more proactive by sending out direct emails to the users explaining the research process to them ahead of time. Also, we learned that a better approach for research would be to conduct it in a neutral area like a hotel conference room or a coffee shop.

Experience: One of the main team leads (who usually refers candidates to us that report to him) took us for lunch along with some of the same candidates we had interviewed. During the lunch, the manager asked: “How did the interviews go and how did my users perform during these tests?”. The room suddenly got very tense.

In summary, this re-iterates that they look at interviews culturally as something they need to ‘pass or fail’. Our primary takeaway was to give the main team leads who are referring candidates to us an overview of the UX process as early as possible. Also, as with the experience highlighted above, we discovered that we should likely conduct it in a neutral space and completely avoid using the words ‘interview’ or ‘test’ and instead reword it to only ‘UX Research’ going forward.

2. Use Google Translate to Translate the Test Script and User Sessions:

Google Translate is good for small chit chat, where you have a few words or basic questions that need translation, but we learned that Google Translate is not sophisticated enough to help you translate complex languages like Japanese. We did find it was useful to create the Japanese version of the wireframes and we did get to use it during one of the sessions.

Experience: During the usability test we quickly learned that some of the words and sentences did not translate exactly to what we intended to convey because some users appeared to be confused when completing certain tasks. After the test, we learned that some of the words that were translated on the wireframes were not used in proper context.

Lessons Learned: Our primary takeaway from this experience was that we need to invest in hiring a professional to help us translate our work before we show the users our screens.

Experience: In another user research session, one of our user’s spoke very little English. For that particular session, we had to use Google Translate as a primary method of communication since a human translator was not available. As a result, we had a miserable session. Google Translate gave very erratic responses and gave us a very disjointed experience. It would turn on and off on its own and in some cases after speaking into it, it would hang (with the circular loop going on endlessly). The 1-hour scheduled test lasted for 2 hours.

Lessons Learned: Our primary takeaway from this experience was to never to use Google Translate again as a primary method of communication during interviews because it wastes everybody’s time. Additionally, the user during the session started to feel embarrassed and helpless as a result.

3. Send the Test Script to the User in Advance So that the Users Can Prepare for the Session:

In theory, this approach sounds like a great idea. Our product managers have been routinely sending all of their questions in advance before they have any call with the folks in Japan, which helps them prepare for the meeting. Because of the language barrier, this approach has been largely successful within the product management teams. From a user research perspective, however, sending them the test script in advance might defeat the purpose of doing the research in the first place for the following reasons:

· First: They would come to the session prepared with answers and biases and would likely discuss it with other colleagues who would also be interviewed.

· Second: The questions would also be sent to their managers, which would then add further bias to the test results.

· Third: During the interviews, we also conduct user tests, so sending them questions in advance would not make sense.

Experience: In preparation for the interviews, we sent pre-test questionnaires in English to the users using SurveyMonkey and found out that all of the users were confused with simple questions like “Job Title” and “Specializing in?”. After the test, we had to ask them to fill out those questions again and had to explain to them what each of those questions meant.

Lessons Learned: Our primary takeaways from this experience was that this step also needs to be placed into context with the culture and language and should to be translated by a professional translator.

4. Have One of the Project Stakeholders Translate for Us:

Predominantly, this seemed to be the most viable option for us because it was not only cheaper but was also the ONLY option which could move our research and test activity forward in a reasonable amount of time. I did the following things to ensure the activity would be as unbiased as possible:

· Explained to them the activity itself.

· Clarified why we intended to do the activity.

· Defined what their role as a translator would be (ie: they would need to be a pure translator, even if they have some background with the project, and should not bias the user with their own thoughts on the questions being asked).

· Sent them the test script in advance and requested that they do not share it with the users.

· Ran a mock session remotely before the real test for the purpose of practice.

Experience: Two senior users were puzzled when we asked them questions about their work. We asked basic questions such as: “What does your day-to-day at work look like?”, “What time do you get to work?”, and “How much time do you spend reading emails vs. performing other tasks”. They seemed a bit uncomfortable, which is when the project stakeholder informed us that asking questions like these to senior users is like questioning their work ethics. We learned that we had to give senior users a little more background of what we (as a UX team) would do with this kind of information. The project stakeholder not only explained to them about our processes but also went a step further by explaining to them about how he has been working with the personas that the UX team created and explained how the designs are kept in context.

In many other instances, this method worked well for us because we on-boarded the stakeholder with our processes and gave him some advance preparation. This also helped us rephrase certain questions which brought in cultural sensitivity. The stakeholders themselves also mentioned how they were learning many more things from these user sessions that they had not thought about earlier. This new perspective helped them to look at the project from a different angle, which in turn enabled them to redefine some of their business goals to better align with their new understanding.

I hope you liked reading about my experience! Please give me a ‘like’ if you learned something new. I would also like to hear about similar experiences or thoughts that you may have had and how you ended up resolving them.