Lights, Camera, Android!
Google’s interactive Spotlight Stories are moving from Moto to Android, and trying new forms — from hand-drawn animation to live action
--
The upcoming live action short called Help! features a scenario familiar to almost everyone who ever bought popcorn and a movie ticket. Location: The concrete ditch known as the Los Angeles River. Character: Damsel in distress — in this case a stylish young woman in contemporary garb. Action: Tracking shot of the woman as she runs—face often turning backwards in fear — from a hideous gargoyle.
And then, something unexpected. The sound of a helicopter, the glare of a spotlight beam from above. At this point a viewer might anticipate a cut to the chopper. But Help! (which couldn’t be more different than the similarly named Beatles movie) does not have cuts. And the director won’t determine what the viewer will look at on the screen. Oh, and by the way, the screen won’t be in a theatre or on a TV, but on an Android phone. When you watch Help, your phone is a window into a 360-degree world captured by a specially designed camera. You simply point the phone in the direction of what seems to be the most stuff happening in that world at that moment. So when you hear those churning blades, you might choose to lift the phone towards the sky, to see the helicopter hovering above. Or you could continue tracking the damsel. Or even look behind to see what’s chasing our terrified ingénue. There is footage to accommodate all of those choices. In any case, it’s a choice that must be taken.
Welcome to what might become the future of cinema. Or what might just become a weird footnote to that storied history. Google, the creator of this technology, is of course betting on the former. And for a week last October, using an exotic multi-lens’ed behemoth dubbed a spidercam, the nova-hot director Justin Lin — known for the last four Fast and the Furious epics — was supervising a shoot in the Southland stone basin that will go through months of post-production and eventually become a Spotlight Story.
Spotlight Stories are brief, interactive entertainments — “immersive shorts” is the preferred term—originally conceived by the Advanced Technology and Projects team at Motorola, during the company’s brief life as a Google subsidiary. The ATAP unit did not go to Lenovo when the Chinese PC maker bought Motorola last year, but was shifted to Google’s Android, Chrome and Apps division. ATAP has many projects in the works including a 3-D sensing system called Tango and a modular phone development kit called Ara.
But Spotlight Stories reach beyond the geeky advances into something that uses advanced tech to reframe storytelling itself. They are the embodiment of the vision that Steve Jobs always espoused about the nexus of art and technology.
Those few who have been exposed to them know that Spotlight Stories can take your breath away. (Only in part because you have to expend physical energy jerking the phone around to see everything.) They are also costly and have no obvious business model. Yet their promise is such that Sundar Pichai, who heads Android (as well as all Google products as of last’s month’s reorg), has blessed the project’s continued development. “The ATAP team has developed a new art form created specifically for mobile,” Pichai said in a statement to Backchannel. “We’re excited to see how the technology behind Spotlight Stories will be used by artists to bring stories and characters to life for people to enjoy.”
After three animation-based efforts—the most recent, a heart-tugging vignette called Duet, will be released “very soon”—ATAP is doubling down by expanding to live action and hiring a director from the top of the A list. But the plot will really thicken later on, when Google opens up production to outsiders. A cliffhanger awaits: will a new generation of interactive filmmakers begin churning out immersive shorts for Android phones? Or will ATAP’s advances be remembered as a test lab for the coming era of virtual reality?
Spotlight Stories came to life as a serendipitous offspring of the Google-Motorola marriage, which had the lifespan of a Kardashian nuptial. When Google paid its $12.5 billion dowry in 2012, Larry Page saw his new gem as more than just a high headcount afterthought accompanying the patent portfolio that motivated the purchase. He envisioned Motorola as a hotbed of invention that would supercharge the entire Android platform. The tip of the innovation spear would be the ATAP group, set up as a mobile-oriented cousin to Google X. To lead it, Google tapped a high-profile hire, Regina Dugan. She had been a surprisingly public (TED talks!) head of the traditionally circumspect government agency DARPA. Yes, the same DARPA that gave birth to the Internet, as well as everything from lasers to the artificial intelligence behind Siri. Dugan set up ATAP to conform to the DARPA system, which employs researchers for two-year periods and encourages them, in turn, to hire contractors to help develop their projects. That way a relatively small team has huge reach, and everyone strives to work quickly for maximum impact.
Spotlight Stories was born soon after the acquisition. It began with a question and an impulse. The question came when ATAP engineers, particularly one from a company Google had bought called Human Engines, wondered what could be done with the 60 percent of the time a phone’s high-powered graphics chips were idle. The impulse, coming from Dugan, was to capitalize on the personal nature of a device that people wanted within reach 24 hours a day. “Do something emotional,” she told a team headed by Human Engines co-founder Baback Elmieh. The mission of Project Avatar, as it was known then, was to use the specific technology of the chip-and-sensor packed Moto X in development to generate interactive graphics that would somehow tap the human side of its users.
The missing piece was storytellers. “We tried painting on this new canvas, and we pretty much sucked at it,” Dugan explains. “So we had to call in the professionals.”
ATAP snared a prize immediately: a Pixar director. Jan Pinkava had directed Ratatouille and had a robotics doctorate to boot. “We were able to sit in a room and brainstorm,” Pinkava told me last year, “and came up with the idea of giving the audience a camera, to make the technology work so a story would unfold before your eyes.” Pinkava brought in an all-star crew including former Pixar animator Doug Sweetland and Caldicott-winning illustrator Jon Klassen . Everything had to be reinvented, even the way a movie is scored — the breathless Django-esque melody evoking Parisian bistros was flexible enough to punctuate the pratfalls but also to vamp for a while if the viewer decided to turn away from Pepe the mouse and study treetops instead. The creation process was joyful: “Going into that room where Jan was, was like being at Pixar in the early days,” says Rob Cook — who was one of Pixar’s founders, now consulting on Spotlight Stories. “Just kind of buzz to it, and it made me happy to be around.”
Over the next few months the team created Windy Day, a raucous interactive delight starring a hapless mouse named Pepe who kept losing his red sombrero. On October 30, 2013, Moto X users were alerted to the debut by a hat icon mysteriously appearing on the screen. If they clicked on it, their phone instantly became a window into Pepe’s world.
Windy Day was a total hoot, a laugh-out-loud CG (computer-generated) cartoon that rewarded multiple viewings. But it did not foment anywhere near the splash that its technology or its artistry deserved. (Had it been released by Apple, there would have been ticker-tape parades from Cupertino to Burbank.) The problem was that so few people got to see it — the justification for the investment (a unique feature of the Moto phone) had severely dampened its profile, because sales of the Moto X were disappointing. When the second Spotlight Story, a thematically similar short called Buggy Night, appeared last March, it was equally innovative but lacking the novelty of the first effort, got even less attention.
So when Google sold off Motorola last year and took in ATAP, it was fair to speculate whether Spotlight Stories would continue. Though Google says its run rate is relatively modest, it almost certainly racks up millions of dollars in engineering costs and the salaries of directors, animators and composers with considerable IMDB credits and, in some cases, golden statuettes. Dugan, however, claims that it was not a tough sell. “All through Google, Spotlight Stories are just beloved,” she says. “Spotlight is not about a piece of content — it’s about a new format that’s unique to mobile. And the only way you can develop that new format is to create in it.” So ATAP was able to complete its next project, which embodied ambition in a more artistic realm. Instead of plucking the funnybone, Google went for the heart. And the auteur was a celebrated master of something one wouldn’t associate with an envelope-pushing new technology: hand-drawn animation.
While the first two Spotlight Stories came with the CG sheen of a Pixar production, Glen Keane’s participation marks an immersion into the delicate magic of classic Disney features — expanded gently into the interactive maelstrom of a Spotlight Story. Keane is an animation giant; cartoonist’s ink is in his blood (his father created the “Family Circle” comic strip), and when Keane joined the Disney studio in the early 70s, he learned directly from some of the remaining “Nine Old Men” responsible for Snow White, Bambi, Peter Pan and other iconic works. Among the characters Keane brought to life are the Little Mermaid (he pictured his wife as he drew Ariel), the Beast, and Pocahontas.
When Pinkava contacted Keane in March 2013 to create a Spotlight Story, it came at a propitious time. “After spending years doing traditional animated features in movie theaters, I was looking for some way to connect in a deeper, more personal, expressive way as an artist,” he says. “And I see this medium as a way of putting my animation directly, literally into the palms of people’s hands.”
The story is simple—we watch as a boy and a girl grow from infants to young adults, getting an impressionistic sense of their development until a dramatic reunion concludes this roughly three-and-a-half minute tale—so simple that it is entirely the execution that draws us to the concept. “There’s no dialogue—it works in any culture,” says Keane. “It connects very much in the way that the baton was passed on to me from Walt Disney’s nine old men, a way of telling stories through character.”
Not that there weren’t challenges. The Spotlight Stories team asked him to animate the short at 60 frames a second, a drastic change from the usual 24. “I was trying to interpolate twenty-four into sixty, and it’s not divisible!” he says. “It was ruining all my timing.” The answer was a metronome app that kept the quickened pace. The dense frame rate also dictated that Keane churn out a huge number of drawings — 10,055 original animation sketches were used in the final work. “I eventually realized that this was a positive,” he says, “I had many more opportunities to express myself.”
The most difficult part, he says now, was the very issue that makes Spotlight Stories unique: “At first, I really had a hard time giving up control, control of the camera,” he says “I wanted the audience to enjoy my composition as an artist. I felt like, ‘You’re going to enjoy this experience that much more if you just put yourself in my hands.’” Duet has perhaps the most difficult user choice of all the Spotlight Stories — within a few seconds, after we see both infants forming, the boy and the girl spin off in opposite directions. One must instantly decide which to follow. The ensuing action must be sufficiently captivating to overcome a nagging fear of missing out. (I’m following him — what’s she doing?) Keane overcomes this by his light touch — every so often, the arc of the two characters (three, including a dog that sometimes gets his own moment in the, um, spotlight) weaves back in sight, in a sort of elegant pirouette. And of course, the viewer knows he or she can always view Duet again, taking a different path through the story. Helping things along is the soundtrack, which has a theme for each character, including the dog. Sound designer and composer Scot Stafford (who has worked on every Spotlight Story) recorded a score where multiple layers of music all play at once — depending on what you’re looking at, only the appropriate tracks are turned up loud enough to hear. “A theme will gracefully bubble to the surface while another gracefully exits,” he says.
As an extra measure to assure a smooth experience, ATAP cheats a bit, especially when the phone senses that the viewer is getting restless looking at one character. “We do actually detect if you’re trying to look at the other character,” says Rachid El Guerrab, who took over from Elmieh as Spotlight Stories tech lead early this year (he was the other co-founder of Human Engines). “We bring the other character over. Just to help out.”
Keane came to see the process more like a stage play than a movie. In the theatre, especially if you are close up, you may choose to focus on one actor—even if the main action is occurring somewhere else. Sometimes it’s fascinating to see how the actor continues to express, even if some other character is moving the action forward. “I saw it like visual poetry, with the freedom to bring ideas in and leave them behind,” he says.
“There was still a beginning, a middle and an end but I found that I was just not locked into the kind of structure that I had known in the traditional long-form feature animation.”
Duet is slated for release very soon, certainly before the end of year. (It has already been shortlisted for an Oscar.) At first, it will be only viewable on the phone that has been home base for Spotlight Stories, the Moto X and G. But since ATAP belongs to Android now, it’s clearly time to expand the platform to the millions of phones that run standard versions of the Google mobile OS. This isn’t trivial. “With the Moto X, we had a controlled environment,” says El Guerrab. “Now we are in a different space where we don’t know what the target phone will be. And these will also be played on phones that haven’t been invented yet. Our challenge is introducing this to a broader audience in a way that keeps the magic of the Moto X.” Dugan estimates that of the billion-plus Android devices in the field, about 100 million of them will be able to run Duet when Google releases it (and the other Spotlight Stories) in a post-Moto version next year.
“One of our goals this year and early next year is scale,” says Dugan. “In terms of both the audience, meaning who can see it, and creatively. So we started with CG, then we went to hand-drawn. And then to live action.”
When Google reached out to Justin Lin, the director who reaped billions in ticket sales for the Fast and Furious franchise was ripe for an experiment. “When ATAP was still part of Motorola, I had seen Windy Day,” he says. “I was blown away by how interactive it was. I kept thinking, can we do this in live action?” The question was so compelling that he made room in a busy schedule — he’ll direct the second season of HBO’s True Detective — to work on the first immersive short with real sets and actors.
While Keane and Pinkava had seen the Spotlight process as similar to live theater, shooting an action short in 360 degrees from an viewer’s shifting point of view recalls something quite different: a sophisticated video game. Lin knew it would be important to distinguish Help! from that experience. The challenge is providing choices in how the viewer takes in the narrative while maintaining the continuity of the narrative itself. “I’m not a gamer,” he says. “It’s still a subjective narrative.”
“It throws everything out the window of being a director,” he says. “What I loved in filmmakers like Altman and Kubrick is their subjectivity and POV [point of view]. In this construct the POV is up to the viewers. If they want to see the performance, they can stick with the actor. And they can see what the actor is reacting to, too.” (In this case, a Godzilla-like monster — there’s some CG in Help!) But in a Spotlight Story, even an actor’s reactions are different. “You have to allow the viewer to have time to frame the actor before he or she sees the reaction,” says Lin’s production partner, Todd Makurath. “In that sense it is more like a stage play than a traditional movie, which is punchier.”
Before Lin even got actors on set, the team had to resolve some heady technical challenges. No camera rig in the world was able to capture the world in 360. At first the team attempted a construct that bound together 16 cameras, called a sphere cam. It moved like a Mars rover. But that had to be discarded because “stitching” the images together was too complicated. The solution was a rig with four Red Epic Dragon cameras with fish-eye lenses. Since it moves on sinewy cables dangled from a crane it’s called the “spidercam.” Each camera collects so much information, the finished product will be too big to be stored on the phone — Help! will have to be streamed.
“This is a movie in 360 — not like a gimmick we put together,” says El Guerrab. “When we were on set, everyone’s first question was, ‘What’s in frame?’ And we told them — everything is in frame.” Spotlight Stories producer Karen Dufilho-Rosen had a piece of advice for the movie-makers: “Think of it as shooting inside a snow globe,” she said,
Interviewed soon after the shoot was completed, Lin was totally bullish on the future of the form: “I’ve been privy to other ways to present storytelling in a different way, like 3D, but haven’t seen the way that it adds significantly to the experience. But this exceeded my expectations. As a storyteller myself, I can’t wait to have another opportunity.”
There would be a possible irony if Lin is successful with Help! The machinery he and Google have created for live action seems to be useful not only for Spotlight Stories, but a different version of interactivity: the kind of virtual reality soon to hit a broad market with technology like Oculus Rift.
It’s an open question whether Spotlight Stories may wind up by becoming a key enabler of the machinery that will generate complex virtual reality, or if phones themselves may indeed wind up becoming the portals into increasingly ambitious interactive narratives. Lin feels that the phone-based approach can persist. “On the phone, I have more control,” he says. “It allows the viewer to interact and frame their own story.”
Google knows that for Spotlight Stories to really catch on, it has to open up the system to creators beyond its closely curated experiments. And that is precisely what’s next on the roadmap — a Story Development Kit to allow others to create immersive shorts. It’s already been released to a few select partners. “Already, we’re seeing things that originally took two weeks to do, done in hours with the SDK,” says Dugan. “We want to package this so that others can create without going through that learning process.” Beginning next year, Google will carefully release the SDK to more creators, with the idea of seeing the circle eventually expand. Producer Dufilho-Rosen and tech lead El Guerrab expect production to ramp up to perhaps a new story a week by 2016. El Guerrab even dangles the possiblity that eventually Spotlight Stories will be available in non-Android platforms, including iOS. “Nothing is off the table,” he says.
“ATAP will continue to do a few each year that will be at the edge of what’s possible,” says Dugan. “And we’ll have a middle layer where we curate things, because the creative leaps are as important as the technical leaps. We’ll start to develop a community of people who understand that. And we’ll see where it goes from there.”
Rob Cook, the former Pixar executive, thinks it could go in amazing places. “There’s no telling what artists will do with it,” he says. “They can build whole worlds you can go around in, have sideline things you could go watch. Or you could have stories with updates—worlds that build over time. It’s a new medium.”
There’s quite a lot that needs to happen before anything like that. For one thing, the technology has yet to reach an audience beyond the small population of Moto X users, not to mention capturing the attention and commitment of movie-makers. But Dugan exudes confidence. “Those who have seen Spotlight Stories have rated it higher than some of most beloved apps,” she says. “When you do something at the edge of what feels possible, and execute it not at 90 percent, but 110 percent, it feels beautiful and kind of badass. And people love it.”
Watch for Duet soon on Moto, and next year on an Android phone near you. And if all goes well, by mid-2015, you’ll be able to view Help!, and find yourself forced into a split-second crossroads decision, thanks to Google money and a lot of Hollywood innovation:
The lady? Or the helicopter?
