Conducting usability testing for LifeSG’s notifications feature
This usability study was conducted in September 2020, and adhered to the COVID-19 restrictions established by the Singapore government at that time.
One of the first tasks that I was handed when I joined LifeSG as a UX Designer was to plan and conduct a research study for a feature that we were revamping. As a budding designer fresh out of university, it was an exciting task to work on, since I was able to test out a prototype that I had designed, validate the hypotheses that I had, and dive deeper into the behaviours and opinions of how citizens engaged with government services.
But before I go on, what exactly is usability testing?
What is usability testing?
According to the Nielsen Norman Group (NN/g), a UX research and consulting firm,
In a usability-testing session, a researcher (called a “facilitator” or a “moderator”) asks a participant to perform tasks, usually using one or more specific user interfaces. While the participant completes each task, the researcher observes the participant’s behaviour and listens for feedback.
A large part of ensuring how well a usability testing session goes depends on the research plan, the participants you recruit and the user tasks you write.
Since participants are hard to recruit, you want to ensure that the time you have with them are not in vain when a prototype fails to load, if they are unable to understand the tasks they have, or if they experience technical difficulties during a remote session.
Here are some key takeaways that I learnt when planning for a research study during early COVID-19!
Choosing the appropriate test participants
When testing for a feature that involves several government services, scoping for a usability test becomes complicated as the sample group can become quite large (i.e. the entire Singaporean population).
We wanted to ensure that the sample criteria was not too restrictive, and that it needed to represent our users well. NN/g recommends 5 participants per sample group, with around 3–4 participants if there are multiple groups of people using the app or site. Since this feature was relatively large in terms of scope, we decided on 3 main target segments, with 5 participants in each segment group:
- Young Adults, aged 18–25 years old. We looked for a good mix of students from universities and tertiary institutes, and fresh graduates.
- Adults, aged 26–54 years old. We aimed to recruit a diverse group of individuals who were employed, unemployed, single, and married.
- Seniors, aged 55 years and above. We found participants who were mid-career, about to retire, or that had already retired.
The 3 target segments allowed us to test the feature across a wide range of people based on their usage of certain government services, but also kept it relatively contained if we needed to draw patterns from a specific subset of the target segment.
When we were relatively satisfied with the participant criteria, we started sending out recruitment forms to gather interested participants.
Preparing for both in-person and remote usability testing sessions
Next, we had to consider whether the usability test session was going to be held in-person, or remotely. While the COVID-19 situation in Singapore had relatively stabilised, we understood that participants may be uncomfortable meeting in-person.
It was a tricky situation, as our previous experiences had also shown that some user segments, such as seniors, faced technical difficulties during remote usability testing sessions, especially if they were not well-versed with using technology. Hence, those sessions would be easier to facilitate if they were performed in-person.
We decided to let participants choose the type of session they preferred — either in-person or remotely via Zoom. In the end, most of our participants had sessions conducted in-person at a location of their convenience (since our office was closed), and only a handful had sessions conducted online.
Planning for both in-person and remote usability sessions simultaneously had its own share of problems. Firstly, there was a lot of travelling back and forth from the in-person locations and home (since we were all working from home). We could have simply gone to a nearby cafe for the remote sessions, but there were two issues.
- Access to good Wifi. You don’t want to conduct the session with poor Wifi, or with one that requires you to login multiple times over an hour, like some coffee shops require!
- Noise levels. Cafés or shopping malls can get very noisy, which might make it difficult for you to hear your participant, and for your participant to hear you. This also affects your recording of the session as well, and during playback you may find that it’s hard for you to pick up what the participants were saying.
Of course, remote sessions came with their own fair share of issues. One of the participants struggled with sharing her screen when she was about to interact with the prototype, as she was not familiar with the technology. We took a good 15 minutes trying to help her with it before we finally managed to start on the first user task.
Creating the prototype and crafting user tasks
Writing good user tasks was also something that I had to learn. The Nielsen Norman Group notes the following:
A task needs to accurately and adequately reflect the researcher’s goal, as well as provide clear instructions about what participants need to do.
While this may sound easy on the surface, it took a long time drafting the tasks until I was satisfied with how they were. For one, we had to make sure that tasks were not leading, in a way that too clearly stated how the participants could complete the task.
On the other hand, the tasks could not be written vaguely without a clear goal for the user to work towards. Writing a deliberately ambiguous user task may lead to the participant asking you, “Sorry, so what am I supposed to do?”, which would have defeated the goals of the usability session.
- Avoid using words in the interface, especially if you want participants to find a particular feature. If you are testing a Notifications feature, where the tab is labelled “Notifications”, instead of writing “Find out what notifications you have”; you may rephrase the task to “Find out what messages you’ve received.”
- Refrain from being over-specific in your task. Writing the task as “Go to Notifications, look for the Action Items tab, and find out when you have to renew your passport by”, directs the user to exactly where they have to go, which defeats the qualitative nature of the tasks.
- Keep any scenarios that you may want to include clear and concise. Try not to provide too much unnecessary background context, as participants may lose track of what the actual user task is. A task such as “Assume that you got married two years ago, and are currently living in a 4-room BTO with your wife, and 3 year-old child. Your wife has recently just given birth to a pair of twins two days ago. Find out how you would apply for your newborn child’s birth certificate” is unnecessarily long-winded. Rephrasing it to “Your wife gave birth a few days ago. Find out how to register your child’s birth” helps the test participant focus on the task at hand, instead of trying to decipher what the task is.
Ending thoughts
I learnt a lot from conducting this usability test back when I’d first joined, and even now I’m still trying to figure out ways to refine the practices that we have on user testing and research.
Of course, whatever I’ve spelled out above are not hard and fast rules for conducting user research — what may work in one context would probably have to be tweaked for another!
Lastly, a lot of my learning came from watching how my fellow designers conducted research, and asking them for tips and advice on how I could improve. This research study would not have been possible without their help and guidance :)
