Meet the bilingual Google Assistant

Google Assistant is now the first multilingual virtual assistant.

Most people can speak two or more languages, but voice assistants have always forced them to pick and use just one. Multilingual households are becoming increasingly common, with several sources [1][2][3] indicating that multilingual speakers already outnumber monolingual counterparts, and that this number will continue to grow. Family members in bilingual homes often switch back and forth between languages, and now the Assistant can keep up. No need to fiddle with with the settings: Google Assistant and Home speaker are now bilingual(something no other digital assistant can do), allowing users to issue commands in two different languages interchangeably. While Google Assistant could already understand multiple languages, now you can speak in two default languages interchangeably with phones and smart speakers and it will be able to follow what you’re saying and reply back in either. Previously, users would have to manually switch back and forth between individual languages.

Bilingual support reflects the growing number of multilingual households and helps smart speakers become a device accessible to everyone and this new flexibility should help people feel more natural when talking to Google Home — particularly in these households where not everyone speaks the same language. At MWC 2018, Google framed the functionality as a way to help Assistant become a more international service with users able to essentially set two default languages.

Technologically, this is not an easy task, with Google setting out years ago to solve the problem and introduce this highly requested feature. There are three aspects in bilingual support, starting with identifying multiple languages.

Humans can recognize a language without having to understand it by paying attention to intonation, phonetic registry, and other acoustics. Since 2013, Google has developed LangID models capable of distinguishing “between pairs of languages in over 2000 alternative language pairs.” Google had to train its algorithms to reliably determine which language a person was speaking. As a first pass, the assistant tries to identify the language just by the sound of the person’s voice — similarly to how a person unable to speak German or French can still know when they hear those languages spoken. To double check, the system also runs the audio through the recognizers for both languages a user has activated. Examining the output to see which makes most sense helps the assistant decide in which language to respond.

Source: Google

In a nutshell, It’s a thorny AI problem that required multi-year effort from Google. To build a system like this, you have to be able to identify multiple languages, understand them and then make sure you present the right experience to the user. And you have to do all of this within a few seconds. Also, this requires Assistant to run multiple processes in parallel to identify language a user is speaking while simultaneously parsing their command. (Learn how Google made it work here.) Although different smart assistants like Alexa, Siri and Cortana can all be set up to understand and speak different languages,

Google Assistant is first to support multiple languages at once.

This new function may also be a welcome salve for users who’ve been frustrated by their smart assistant’s struggle to understand an accent. As per a recent survey, People who speak Spanish as a first language, for example, are understood 6 percent less frequently by smart speakers than West Coast English speakers. Letting users volley between languages may help thwart misunderstandings. In the video above, Google shows the feature helping young kids interact in whatever language is natural for them, as well as being used by those living abroad. Meanwhile, on phones, the feature can be leveraged by users who speak one language at work and another at home.

Users can specify that they want listening done in two languages in the app a new “Assistant languages” menu under Preferences in Assistant settings on their phone or Google Home speaker. Then, a person can call out requests or commands in either language. If you’re looking for an answer in English, ask, “Hey Google, what’s the weather like today?” If you’re craving tunes from your favorite German hip hop band, just ask “Hey Google, spiele die Fantastischen Vier.” Currently, the Assistant can understand any pair of languages within English, German, French, Spanish, Italian, and Japanese. Google will be expanding to more languages supported by the assistant — Hindi, Indonesian, Korean, Brazilian Portuguese, and Thai in the coming months. Google is also aiming to make Assistant trilingual in the future.

It would be interesting to see, how Google solves the problem of colloquial combinations of two languages, such as Hinglish, the mixture of Hindi and English spoken in India or Spanglish(Spanish & English). Maybe one day Google Assistant will be smart enough to join in those conversations and be able to recognize all the languages that you may speak.


If you enjoyed this article, things you can do from here:

  • I read every response on Medium or Google+, so keep the conversation going in the comments below.
  • Subscribe to this blog via RSS.
  • Follow Deepak(Author), on Google+ or Twitter
  • Do tap/click that little clap below to recommend it to other interested readers! Or feel free to pass it along.