Google needs to fix its notoriously bad bilingual speech recognition on Assistant and Gboard (Update


Google collects vast amounts of speech data across all of its products, and while it hasn't been too transparent about the practice, we as users profit from it for the most part. Speech recognition has consistently gotten better over the years, which has allowed impressive sci-fi tech like smart speakers to enter our homes. There's one department where Google needs to step up its game, though: multilingual speakers are having a hard time using more than one language on any Google product.


While there are a lot of people that live all of their lives speaking one language only, many countries around the world routinely juggle between two or even more all the time. These days, plenty of products are still developed by people with only one primary tongue, so bilingual support is often not as good as it could be (which is also true outside of speech and audio processing).


That's a problem for people who switch between more than one language regularly. Android Police founder Artem routinely struggles with Gboard's dictation feature while our editor Rita just straight up removed her secondary language, French, from Assistant. There are also reports on Reddit complaining about issues arising from a bilingual Assistant. Some Redditors have even noticed that their Google Home speakers need more processing time when they have to juggle two languages.




Gboard

Here are some examples from Gboard's bilingual woes for me, a native German speaker. These issues would certainly be amplified for people using more than two languages.

https://youtu.be/oya0PjsOkRM


As you can hear and see, there are also some instances where Google arbitrarily switches languages in the middle of a sentence while processing. That sometimes does make sense (I change languages mid-sentence depending on who I talk to), but it often happens when it's not supposed to happen. That's the case in these examples, where Google completely missed that I was talking English and created some nonsense German gibberish, translated for your convenience:

  • "I will now demonstrate how bad[ly] Google understands me" -> "Arena Damen straight Herbert Google landesgrenzen" -> "Arena ladies straight Herbert Google borders"

  • "If only there was a way I could automate this" -> "BVG ticketautomat" -> "BVG vending machine" (BVG is the public transport agency here in Berlin, so this combination surprisingly makes sense.)

  • "Hold on, I still have to finish the article" -> "AUTOMEISTER Haftbefehl nicht radeke" -> "CARMASTER arrest warrant not radeke" (that last word isn't even a word in German, and I have no idea why CARMASTER would show up in all caps.)

Note that it took a few tries to get this level of misunderstanding out of Gboard and in the snippets you hear here, my accent showed through the most, which could amplify the problem. Still, Gboard should be able to notice that what it's producing in German is pure gibberish and switch to English instead.