Image for post
Image for post
Bangkok, a grand place to do user testing

Lessons learned from user testing 13,000km away

Vicki Anderson
Oct 15, 2019 · 8 min read

At the end of September three of my teammates in Liberty IT and I travelled to Bangkok to carry out user testing on the application we are building for LMG Insurance Thailand. Planning the trip we were under no illusions that this would be a normal user testing session. Jet-lag, a translator, and time restraints were just some of the ways that this differed from testing in Dublin. Being prepared for unexpected challenges, we were able to adapt during the week, take some of our learnings on board to tweak our methods and ensure our we got the best out of each session. Going forward we will aim to take an iterative approach to trips and build these learnings into our process.

Image for post
Image for post

One of the most important considerations when planning a trip like this is your translator. During a regular user testing session we would have Anthony introduce the session and ask the questions while I took notes. While a few of our users could read English, they didn’t speak English and we can’t speak Thai, meaning we had to adapt our regular interview setup. We were lucky to be given an excellent translator, Jouet, who kept the conversation going and was able to probe and ask for more detail where we would have.

At the beginning of the week we introduced user testing to the team and Jouet, explaining the importance and benefits as well as how the sessions would be conducted. Our first sessions involved Anthony asking the questions while Jouet translated them to the user and translated their feedback to us. After two sessions however, we realised that it would be easier for Jouet to ask the questions and so we gave him a script. Being able to recognise this quickly and adapt we were able to make the rest of the sessions quicker and flow better. Going forward we would make sure our translator was asking the questions from the start and we were focused on observing and taking notes.

During this trip we realised that it would have been useful to have some kind of onboarding session with Jouet to allow him to become more familiar with the process. Included in this would be giving him an opportunity to run through the tasks and prototype himself prior to carrying out the user test. This would allow the translator to become comfortable with the prototype and the tasks being tested and ask any questions prior to a user being in the room. While we had briefly covered why and how we do user testing with the whole team, going forward we would do a more in depth session onboarding the translator on how to conduct these interviews and what exactly we are looking from the test.

Luckily with Jouet, he was able to pick it up quickly but it would be useful for a future translator to have a better background when beginning the test. This could even be done along with introductions remotely the week before travel if pushed for time. We only met Jouet the day before our testing was to start so it would give a good opportunity to get to know each other before arriving.

When seeing your users involves 17 hours of travelling you want to be able to get as much out of the trip as possible. Being there for five days, we decided to run two rounds of user testing. The first round involved explaining the process to the users, asking them some background questions, and then asking them to work through four or five predefined tasks, depending on their role. We scheduled this round with five users for Tuesday and a second round for Thursday to verify any updates we had made after Tuesdays sessions.

The second session was somewhat of an experiment we wanted to show the users that we cared about their feedback by showing them changes we had made directly based on what they had said two days previously. While we were under pressure to get all the feedback consolidated and updates made between Tuesday and Thursday, it was worth it to gain the trust of the users who now know that we take their feedback seriously and care about their experience. The users reacted positively to seeing changes they had suggested and going forward we hope this will have given them trust in us and shown that we will listen and react if they find they have issues with the app. It is our hope that having that level of trust will make remote user testing on other parts of the app much easier.

Going forward we will try to leave more time between the sessions to allow more time to go through the initial feedback and make the resulting updates. Ideally we would try stay for an extra few days but if confined to a week it would be better to do your introduction to user testing on Monday morning and carry out the user testing Monday afternoon and Tuesday morning. This would give you Tuesday afternoon and all of Wednesday to decipher the feedback and implement changes.

Image for post
Image for post

User testing is usually carried out behind closed doors, ensuring the user doesn’t get overwhelmed by too many observers. Private user testing with just the user, facilitator, and note-taker (and a translator if needed) ensures that there is no manager in the room making the user feel like they are being tested. Some methods will involve a live stream to the rest of the team in another room but this is difficult if you are flying to another city and setting up in your customer’s office as you may lack the space and equipment.

During our sessions however, we allowed a business representative to observe one session, as long as they didn’t try interact with the user during the test. Even allowing a representative from the business to observe part of a session can be good as it allows the business to understand the process of user testing. This is especially useful when working with a team who are not familiar with design thinking or user testing. Allowing the business to observe during the first time builds a level of trust in what we are doing and should make it easier for us to run more session in the future, remotely or in person.

In a separate session Brian and Robbie, the senior engineers on the team, observed the tests. While it’s good for the business to understand what user testing is, it was good for the rest of our team to observe and see what problems the users were having. We are a product focused team and it’s important for the whole team to understand the impact the product that we are building will have for the end customers. As a result we no longer think about the end customers as being users but instead think of them as the five people we met who showed us what they do each day in work.

There’s a sweet spot when it comes to running sessions with users, too few and you risk missing out on vital information, too many and they start to be repetitive and you’re left with too much repeated feedback to go through. Having said that, there’s no magic number of tests that you can aim for, it depends on the project and the users. On the trip we aimed for five or six users for the first round of tests and we ended up carrying out the tests with five users. Five was a good number for us as we managed to test two different role types within that number and we were beginning to see patterns in the feedback.

For the second round of testing we left it quite open. We wanted to test four tasks but we were conscious that we might end up getting a lot of the same feedback. We ended up testing with three users as it was mainly positive feedback relating to the changes we had made along with one extra task.

Image for post
Image for post

Once we finished our user testing sessions we went through the feedback and notes we had taken during all of the sessions and populated a user testing insights map in sketch. Going forward it would be better to take 15 minutes between each user testing session and populate a map for each test. This way you would be populating the map while the insights are fresh in your mind and you would have everything segmented before you start grouping and assessing the feedback to make changes between the two rounds. You can then keep these individual maps as a reminder of what each user said and did but also take the findings from the user testing as a whole and create a matrix of the tasks, problems found, and roles of users for better filtering of the feedback if needed.

Image for post
Image for post

While we have learned a lot from this trip, it doesn’t mean we have discovered the perfect formula to a user testing trip. Every client is different, every project is different, and every trip is different. The six points above are just some of the most important aspects which we picked up on while on this trip. While we will implement them on our next user testing trip, it’s important that we keep the agile mindset that allowed us to pivot and change things easily on this trip. Going out with the mindset that there are going to be challenges and the team is going to have to adapt is the key to successful user testing abroad.

Found this mildly interesting? Follow me (Vicki Anderson) on Medium and check out some of my other articles! Please 👏 this article to share it!

Have some tips of your own to share or just want to chat about the ones I shared? Give me a shout on LinkedIn

LibertyIT

Liberty IT thoughts and tech stories on technical…

Vicki Anderson

Written by

Product designer, photographer, and frequent traveller. Most often found searching for flights on Skyscanner.

LibertyIT

LibertyIT

Liberty IT thoughts and tech stories on technical leadership, digital transformation and agile software delivery.

Vicki Anderson

Written by

Product designer, photographer, and frequent traveller. Most often found searching for flights on Skyscanner.

LibertyIT

LibertyIT

Liberty IT thoughts and tech stories on technical leadership, digital transformation and agile software delivery.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store