A tale of three languages: Usability testing our translated services
Service-oriented language… in three other languages
One of my favorite parts of this job is creating something that will make someone’s life easier. I’ve written about teaching city employees how to write people-oriented content via Funshops, and showing those same employees how to manage their content so they don’t lose track of what they have. What can I say? It’s exciting to teach people how they can save time and money by re-thinking how they write and organize content.
Part of my role as a Content Strategist is helping employees understand the impact that re-structuring their content can have on residents. As we continue to test and adapt alpha.austin.gov, it’s very important to reinforce good habits in our content authors, especially when we knew our next step was to translate our services into three other languages. If it’s hard to understand technical, department-oriented, and acronym-filled content in one language- you can imagine how challenging that content is to translate!
I know that Funshops and best practices can’t solve everything, and in a city that is growing as quickly as Austin is, there’s a lot we can do to reach the residents who read primarily in languages other than English, or help translate information for their friends and family members.
Translating into the first three languages
Based on analytics, people who speak languages other than English aren’t using the City of Austin’s website in non-English languages. This means non-English speakers or residents who read English less than well, underserved and growing refugee population have poor access to online city services.
There are eight other languages that are primarily spoken in Austin and we initially wanted to translate Alpha.austin.gov into all eight languages. However, focusing on just three languages allowed us to iterate on our translation workflow quickly.
So after we built a solid foundation of people-(and service)-oriented writing, we got to work on translating our content. We chose to translate our content into these three languages:
- Spanish,
- Vietnamese,
- And Arabic.
Spanish and Vietnamese are the top spoken languages in Austin, after English. We decided that Chinese- both simplified and traditional, which is the third most written language in Austin, will be addressed in the next round of translation. Arabic, the fourth most spoken language, presented an interesting challenge for both our design and development teams because it is read from right to left.
We used a translation service that already had a contract with our city. Using a translation service means humans translated the content, instead of a machine. Having a human translate the content meant that the message or meaning was translated as well, not just the words. We’re still iterating on the workflow, but don’t worry- I’ll share more about our translation process after we refine it.
As soon as we got the content translated it was time to head to Barton Springs for a swim and a nap under the sun. Just kidding! Now that we had the content translated, it was time to test!
Testing, testing!
What we tested
Thanks to our incredible team, we were able to test a high-fidelity prototype. Meaning the prototype we were using, Alpha.austin.gov, has a high level of details and functionality- it could be near a finished product… but we’re just getting started!
We had a few goals going into this test:
- First, we wanted to make sure that people using the services offered by the City of Austin in any language other than English get the information they need in order to complete a task on the City of Austin website.
- Second, we wanted to make sure residents were able to easily switch between languages using the language bar at the top of the screen.
- And third, we wanted to ensure the translations were accurate.
Who participated
We tested with eight residents here in Austin. Each participant was a native speaker in either Spanish, Vietnamese, or Arabic, and were comfortable speaking in English. We wanted to make sure that nothing was (forgive me) lost in translation during this round of testing. To prevent losing meaning as much as possible, we recruited bilingual residents. However we hope to test with monolingual residents in the future- with the help of interpreters.
How we tested
For our tests we asked residents to think aloud as they moved across the site and through the service. After we explained what think aloud testing was, they seemed comfortable telling us what was going through their minds, and they all asked us if we wanted them to share their thoughts in English or in their other language.
The qualitative data we collect from the participants, especially at this phase of the project, can validate our hypotheses about what a resident needs to know about a service in order to access it. That the data collected is in the residents’ own words adds even more value to it. We decided to test two service pages on Alpha.austin.gov.
Task 1: Pickup free paint
The first task was to have the resident find out how they could pickup free paint in a language other than English. We wanted to make sure they were able to select their language using the language bar at the top of the screen, so that they could access digital services in a language they are comfortable reading- if English is not that language for them.
Next, we asked them to read about the service. If anything was unclear or the translation was not quite right to them, they brought it to our attention. We would ask them what they thought the next steps would be or what they would suggest because we’re developing these pages to help them!
Finally, a quick game of role playing. We asked them to pretend that I was a co-worker and needed to paint my living room but had no money. With this task we were trying to learn if the information was simple enough for them to remember and then potentially tell one of their friends about it. I asked them if they knew where I could get free paint, and listened as they told me about the service they just read. Most participants were able to recall that it was free and that I had a few colors to choose from.
Task 2: Look up your trash, recycling, and compost pickup days
For the second task, we first asked if the participant lived in a house or in an apartment/condo and if they knew when their trash and recycling was picked up. Several residents knew their trash days because they either asked their neighbors or kept an eye out for when the bins started appearing on the curb.
We then asked them pretend that they were new to Austin, and use was to look up their trash, recycling, and compost pickup days. Again, we wanted to see if the content was simple enough to understand when the rules are slightly more complex. I say more complex because curbside pickup is different if people live in a single-family home, an apartment, or a condo.
Six out of our eight of participants told us that they asked their neighbors or leasing offices to find out when trash was collected instead of using the City website. This was not unique to one demographic, as at least one person per language preferred to find out via a neighbor instead of using the website to find out when their trash is picked up. I thought this was interesting because my hypothesis was that people would just “Google it” to find out- but instead, they reached out to their neighbors. However, in our conversations it seemed like this was one of the few instances the participants would go to their neighbors, and not the internet, for information about the City. That being said, it could be worth exploring with a larger test pool.
What we learned
Overall, this round of usability testing proved the value of human translation when it comes to City services. We observed all but two participants use Google translate for single words when they needed clarification. We asked one participant about that, and she told us that Google translate can be helpful in small doses, but she doesn’t trust it for paragraphs or larger blocks of text.
Each participant was pleased that the City was taking steps to improve the digital experience for bilingual or monolingual residents. When asked about seeing City services translated into Arabic, Anna, an Arabic speaker, said, “There was nothing, now there is something.” She elaborated that she was happy to see the City taking steps to make their content more accessible to all residents.
“My mom is a monolingual Spanish speaker… I try to find Spanish GED courses for her… this [alpha] is actually something I’d recommend to her.” -Zach, Spanish speaker
Hearing feedback like that validates the work we’re doing to make City services accessible to everyone in Austin, through using service-oriented language and human translation services. As always, it’s a work in progress- we’ll keep iterating, testing, and sharing our learnings.
