The Genie Inside Your Smartphone:
Fifty years after we first met the sentient, soft-spoken computer Hal 9000 in 2001: A Space Odyssey, we’re finally able to command our computers with our voice. Now when we talk to our smartphones, we expect them not only to hear us but also to understand and, unlike Hal 9000, to obey us. But to truly improve our lives, our mobile devices must not only process what we say, and what we mean, but also our context. When you’re driving, your phone shouldn’t expect you to touch it. While you’re sleeping, it should know to send your calls straight to voicemail… unless there’s an emergency, something it should also be able to ascertain. If you want to take a selfie, you should be able to simply say “Take a selfie!” In short, your smartphone should live up to its name. That’s the goal with the Moto Voice and Moto Assist software integrated into the second generation Moto X smartphone.
“It needs to be adaptive and responsive,” says Mark Rose, senior director of product management at Motorola Mobility.
And to do that, the Moto X is always listening, for verbal commands from the user and also ambient cues of the context. That emergent behavior is spawned by complex interactions between the software and hardware. Myriad processes, running simultaneously, analyze inputs from the handset’s sensors and software, triggering immediate responses when called upon but fading into the background when not needed. The magic happens behind the interface’s curtain.
“All of the processes are divided and connected in a way that makes it seamless, so from the user’s point of view you just talk to it and it does things for you,” Rose says.
Enabling your smartphone to hear you was the first major challenge. Signals from the microphone must be filtered for background noise like nearby conversations, music, and TV shows. Just think of the brainpower you use, subconsciously, at a loud party to tune out all the chit-chat except the one you’re participating in. After minimal training in a quiet space, the Moto X can recognize your voice amidst a cacophony of others. And that involves computerized consideration of gender, accent, and dialects of a variety of languages.
“Testing and improving the voice capabilities was exceptionally difficult,” Rose says. “While humans are trained to understand voice, there are many nuances to making it work well on a mobile device. … We ran extensive internal testing, but the real work came after we pushed the software out to the public. Then we had a much larger sample size and diverse population to work from.”
The context awareness comes from the device’s array of infrared sensors, accelerometers, and microphones tuned with sample data from users. For example, when a user talks into the microphone or waves his or her hand over the display, those inputs flow through a software “context engine” that deduces what the user may be doing at that moment and then determines the best response. In many cases, the right reaction may be to leave the user alone. For example, the Moto X can turn on voice-only commands when driving and automatically silence the ringer at the time of a meeting scheduled on Google Calendar.
Indeed, while most people think about how their devices behave when they are using them, the Moto X software design team spent many cycles considering what they call “user-less interactions.” It’s a fascinating design question: What happens when you’re not interacting with your phone which is, well, most of the time? The Moto X, Rose says, is smart enough to figure that out, something that became abundantly clear one morning at 6am as he made his way to Motorola’s Chicago design studio. He used voice commands to send a message to his wife in California telling her how bone-chillingly cold it was outside.
“She responded back, ‘you know it’s 4 AM here, right?’” says Rose. “And I said, ‘I didn’t realize your phone would wake you up. Mine doesn’t do that.’ I take these capabilities for granted now.”
