Usability in another language: Why we’re simplifying before translating

When my mom first moved from India, she barely spoke any English: “It’s hard when the language in your head is different from what people say outside of it.”

She recalls particularly frustrating moments at employment offices. “It felt isolating.”

Language barriers feel especially acute when trying to access essential benefits or other city services, like getting a pothole or broken street light fixed.

One way people in the city of San Jose can report neighborhood issues is via the City of San Jose’s issue reporting app, San José 311 (formally called My San Jose). And the city is diverse: about 57% of people speak a language other than English at home, with 14% over the age of 18 speaking English “not well” or “not at all.”

To better understand how we might improve the app to make it more accessible for non-English speakers, my colleague Julie Kim and I conducted user research via a six month Code for America fellowship.

19 discovery interviews, 16 stakeholder interviews and ten prototype-testing sessions later, it was clear: simplifying the content is an important first step to getting the translation right. (link to full report here)

A simplified English version of the app tested better than a translated version

We built simpler versions of the app in English that had less text, fewer points of friction, jargon-free language, clearer icons, and reduced the number of clicks so users can complete reports quicker.

We then asked seven Vietnamese-speaking residents and three Spanish-speaking residents to test our version compared to a translated version of the current app (note: these were translated by human Spanish and Vietnamese translators.)

Example screens of prototypes we tested with low-english proficient Spanish and Vietnamese users: Spanish version of current app(left), Vietnamese version of current app (middle), Simplified English version of app (right)

The majority of users (five Vietnamese and two Spanish-speakers) preferred the simplified English version of the app. One Spanish-speaking user tester explained: “…The [Simplified] English one made more sense to me even though this is in Spanish [my preferred language].”

We also learned how some labels didn’t fit with the user’s expectations.

We performed a card sorting exercise, which involved representing various content categories found on the San José 311 home page and left-hand menu. This includes a category called My Home Services, which offers information like garbage schedule, recycling information, yard trimmings, and street sweeping, water services.

Results from our card sorting exercise with 10 users (3 spanish speakers, 7 vietnamese speakers) revealed:

• all ten testers didn’t expect any of the services (apart from water) in this category. Instead, they viewed these services outside their home (unlike what’s directly inside their home, like water).
• further label testing confirmed that the label ‘Neighborhood Services’ better matched up to users expectations (which is in the newly released version of the app!)

Other labels needed more context.

Example of abandoned vehicle form in current app in (Spanish, left) and (Vietnamese, right): Users were unclear what a label like ‘substantial damage’ was referring to when filling out a report about an abandoned vehicle. Or a label like ‘multiple street lights out’ when reporting a broken streetlight. These labels seemed out of place vs a clearer (and conversational) version (is the vehicle damaged? or Is more than one streetlight out?) that flowed more contextually as a user filled out the form.

What we’re doing now

We took our user research findings to implement and release a more usable version of the 311 San Jose web and mobile app: clearer language, adding text only if it adds context, and improved icons. It also includes gentle nudges, and explanations on the benefits of creating an account.

Homepage previous version of San Jose 311 web app.

Homepage of newly released version of the web app

Example of abandoned vehicle form in newly released version of app

The new version of the app now displays a screen with clearer messaging after a resident submits a report, and better sets up expectations about turn-around times.

We expanded on our earlier simplified prototypes from our fellowship, continuing to test with local users to validate if our improvements were on the right track. Additional user testing with low-english proficient users showed that the majority (6 out of 9) rated the simplified versions as either “easy” or “very easy” compared to the previous version of the app.

We also user-tested and updated the email notifications so that they’re clearer and easier to understand.

User testing validated that residents prefer friendly and direct communication updates about the status of their reports.

An example of previous confirmation email; all user interviewees rated this version as “least preferred”.

An example of new confirmation message with plain language improvements in the subject header and body content.

Now with a more usable version of the app, it’ll be better set up for translation down the road.

Can a human-trained machine do it better?

We used the free Google Translate web service to translate labels for the existing San Jose 311 app during our Vietnamese card sorting exercise and usability testing. We quickly found that it was a poor experience (and based on multiple conversations I’ve had with other UX researchers, and gov-tech folks, this isn’t a big surprise). Subsequent interviews with Vietnamese and Spanish translators highlighted similar themes in the automatic translations: overly formal tone, incomplete sentences, some words still in English, and keywords missing from call to actions. It also struggles with translating common City terms (e.g. illegal dumping or streetlight outage).

Clearly, solely relying on free automated translation tools won’t work.

But how well would a more sophisticated machine learning model perform? This differs from the basic, free Google Translate since we’d train the model with City-specific phrases and keywords. It’d then learn the translations and improve with time. We’d need the help of professional translators and native speakers to source and vet the training data. Testing the quality and iterating based on usability testing can help us figure out what’s working, and what isn’t. It could still reveal the same issues (at worse) or provide a more enriching experience (at best).

If an automated translation model works well, we could scale to other city services within the City and, potentially, to other cities.

We’re starting to think through our language translation plan, so stay tuned for upcoming posts.