What if you could mouth what you want to say in your native language, but “speak” in an entirely different language?

A group of researchers at Carnegie Mellon University may be headed in the right direction:

Mr. Jou, a graduate student in language technologies at Carnegie Mellon University, was simply mouthing words in his native Mandarin Chinese. But 11 electrodes attached to his face and neck detected his muscle movements, enabling a computer program to figure out what he was trying to say and then translate his Mandarin into English.

The result boomed out of a loudspeaker a few seconds later:

“Let me introduce our new prototype,” a synthesized voice announced. “You can speak in Mandarin and it translates into English or Spanish.”

Source: Post Gazette

Obviously a ways out from being a “production” tool, but incredible nonetheless. I’d be keen to hear just how accurate it is once you get off “script”, I’d assume it only has a decent machine translation backing it up. I’d love to understand how it recognizes the muscle movements too – do they train it with a wide vocabulary or do they need to teach it just some basic sounds to recognize?

Imagine a day though when you can speak into your computer and it creates your marketing content etc. automatically in as many languages as you like, automatically performing text-to-speech and translation actions all at once.