The Future of Visual Storytelling
The future of visual storytelling has changed quickly over the last decade, from new interactive technologies, like Amazon’s Alexa device, to the rise of cord cutters and streaming video on demand, to virtual reality. For this project, we wanted to explore what the future looks like in this rapidly evolving field. We focused on two major trends: (1) audience interaction, and (2) point of view shifting. For each, we analyzed the available technology and applicable media formats. We selected examples of relevant technology available currently, and we surveyed future trends. We also asked experts in film, media, technology, and gaming to tell us about the trends that most excite them . Join us as we enter the future of visual storytelling.
A key trend in visual storytelling is increased audience interaction. Until recently, immersive storytelling experiences were limited to live shows, such as Disney’s groundbreaking 1998 4-D show It’s Tough to Be a Bug!, and the cult interactive film Rocky Horror Picture Show. In these examples, audience members are immersed in the story both actively and passively. Actively, audiences join the story by yelling and singing. Passively, the audience physically experiences the scene through the use physical special effects, such as squirting water or throwing rice.
However, in these experiences, the story is predetermined. In interactive storytelling, the plot shifts based on audience interaction. This can be as simple as a digital version of text-based “Choose Your Own Adventure” stories, to a more complex artificial intelligence approach, such as FIFA’s new “Journey” mode, where the player follows a soccer star’s career progression.
Going forward, new technologies, like Amazon’s Alexa voice control or Affectiva, an emotion recognition software, are bringing immersive interactive experiences into the home. Applicable media formats include advertising, public media and documentary, and gaming.
Blippar is an augmented-reality mobile app that enables users to unlock virtual content in the physical world around them. Brands use Blippar to target and connect with highly-specific consumer groups. As a contribution to the interactive storytelling world, Blippar users scan physical products and print advertisements to unlock coupons, content, video clips, games, and more. We are especially excited about Blippar’s potential because the business model is already proven. In particular, we see advertising technologies capitalizing on emotional recognition software in the future. If the software detected that a consumer was sad, for instance, Blippar-like companies might serve up more upbeat and comedic video clips, or even a coupon for a massage or other wellness benefit.
We also envision advertisers targeting specific users using augmented reality billboards or “magic mirror” technology. For example, in Rebecca Minkoff’s New York store, fitting room mirrors allow shoppers to adjust lighting for day or night and to scan their current products to unlock curated suggestions.
Public Media and Documentary
Virtual reality has been used to great effect in traditional news media. One example of this is the New York Times feature, Fractured Lands, a unique project that enables readers to empathize with the people most affected by the Arab Spring through personal interviews and a virtual reality experience that places readers in the center of battle in Fallujah. We envision the future of news media will include more of these multi-media longform pieces with increased interactivity . For example, readers might adopt the perspective of striking workers in a story on labor rights and choose actions to see how those actions shape different outcomes.
Alternatively, in a current example, Before It’s Too Late is a virtual reality experience that highlights climate change stories, allowing viewers to interact with the media and access unique content. We believe this kind of interactive storytelling would give readers greater context and empathy in understanding the world around them.
As mentioned previously, videogames are a huge area of innovation in interactive storytelling. FIFA’s release of “Journey” mode allows players to track one character throughout his career, controlling his play, choosing which teams he decides to join, and even managing post-game interviews. We also want to name-check Pokémon GO, the wildly popular free-to-play augmented reality app that lets users discover virtual Pokémon characters in the physical world around them. Finally, Urban Archive NYC is a new location-based augmented reality project that overlays historic photographs of New York City with their real life counterparts.
While we envision augmented reality storytelling to keep growing, we want to caution some of the hype around virtual reality. Many virtual reality users report motion sickness when using the technology for longer than a few minutes. Significant improvements are needed before virtual reality experiences will catch up to traditional video games that enable hours of daily play.
In technology, a future trend we envision is the expansion of voice control, like Amazon’s Alexa, into augmented reality and projected augmented reality control. A user might, for example, verbally request instructions to build a piece of furniture, but projected augmented reality might also help her lay out the parts in the correct arrangement, and to detect if a piece were missing. Voice is only one way to interact with digital experiences; gesture, touch, and pressure are all fertile areas for innovation.
In industries, we believe health and fitness is an exciting growth opportunity. For example, new augmented reality apps enable users to track their caloric intake by taking pictures of their food. The technology that exists now is not very accurate and needs improvement. In addition, based on your food intake and exercise plan, the software could suggest actions you might take to achieve your health and fitness goals.
Point of view shifting
The other emerging trend to engage audience in visual storytelling is through point of view shifting. Since the beginning of digital visual storytelling, majority of the stories have been told through third person point of view. Specifically, most visual stories, particularly documentaries and films, are presented by taking the audience through sequences of events through camera’s lenses.
Despite of the fact that some narratives are in first or second person point of view, the audience observe the characters as outsiders as the story develops. In many cases, how much the audience resonate with a character depends on the narrator’s ability to tell that character’s feelings and thoughts, and such ability is often limited due to various constraints. In addition, story makers decide on the information each character reveals to the audience, who interpret limited information given in the show differently. While these limitations contribute to the artistic effect of visual stories, they also hinder audience engagement. As the results from the research Advanced Identify Representation at MIT CSAIL indicates, audience show different level of engagement when they are assigned different roles during interactive narrative experiment . In some cases, the stories would be much more powerful if the audience could experience events with their own eyes.
The recent development in Virtual Reality (VR) technology makes it possible for audience to experience events by telling stories with first person point of view with the audience completely immersed in a computer-generated 3D world. First started in late 80s and early 90s, VR technology was experimented at a time when personal computer was exploding . Not until recently, VR technology gained significant industry traction and tremendous consumer attention until Facebook acquired Oculus. Today, Oculus Rift, Samsung Gear VR, HTC VIVE account for majority of VR hardwares on the market. As a result, the rapid advancement in VR hardware has accelerated VR content development. Among various visual storytelling formats, films, public media and documentary, and gaming have seen a significant amount of experiments of first person point of view storytelling.
Films, specifically artistic or fictional one, also have been experimenting on shifting the narrative’s point of view. This point of view shifting has taken place in multiple ways. The first way is putting the audience in first person point of view directly in 2D. Hardcore Henry, the most notable example in this category, is shot entirely from the first-person perspective. Its production was made possible by GoPro .
The second way is to enhance audience view in 3D by shifting point of view occasionally as the story develops. Film director Ang Lee, a two-time Oscar winner, experimented this approach in his latest film Billy Lynn’s Long Halftime Walk. In some scenes when the major character has memory flash-backs or put on a spot, the point of view shifts to first person through which audience sees what the character sees so that they can connect to what the character feels and why he feels it. Pieces of first person point of view storytelling was enabled by film’s production in 120fps/3D/4K (‘hyper-real’ technology) . One other approach to take the film to an entirely whole new level by making the story completely immersive in VR. This enthusiasm among film-making community has been well demonstrated by Tribeca Film Festival’s VR programming, whose Virtual Arcade features films that are made in VR, through which audience experience the story from first person point of view .
Public Media and Documentary
Documentaries, including full-length documentary films and journalism videos, have also been increasingly adopting VR technology. Documentaries are made by following specific subjects with a clearly-defined purpose. Documentary makers aim at connecting the audience to the stories and relate to the subject matters, either people or events, on a human level. For the past decades, documentary community has been looking for a more powerful storytelling vessel. As one filmmaker Chris Milk points out, documenting is about “conveying a place and time that existed, to someone at a later date: giving a person the context and trying to make them feel as informed as if they were actually there”. With the VR technology, audience are able to experience as if they were there and connect without having the middleman standing in between through first-person lenses .
Currently, a number of full-length documentary films have been in VR. For example, Zero One, the first documentary shot in 3D, 360 degree about virtual reality specifically for Oculus Rift, attracted huge industry attention . Since then, many other VR documentaries were produced, including Capture Everest, a first-person documentary made by Sports Illustrated .
Similarly, journalism has adopted the technology to take the audience to experience the events through their own eyes. A number of major news companies have expressed their enthusiasm. New York Time developed their own VR news application in partnership with Google by offering their content through google Cardboard and Daydream View. CBS also launched their own VR studios and produced VR stories including a VR tour on Mars .
Gaming community is the most important crowd that has been pushing for VR technology. Among different categories of video games, Role-playing game (RPG) has always been an important one. Many populars games, such as World of Warcraft and Final Fantasy, all belong to this category. In these games, players are assigned a character and complete mission controlling the characters already through first person point of view on a flat screen; such experience is partially immersive. Therefore, it is a natural transition that RPG games adopted VR technology which enhances players experience by providing a fully immersive experience.
Established firms, such as Insomniac Games, Ubisoft, Resolution Games, expanded their existing gaming product portfolio while smaller players, most notably Owlchemy Labs which was recently acquired by Google, are all trying to get into this space by fully immersing players with an enhanced experience .
There are a number of challenges in adopting VR technology in first person point of view storytelling for a large audience. First of all, filmmakers need to adopt completely new ways in producing the VR content from hardware technology to film shooting techniques . It takes a lot of experiments and time to figure out best ways to shot stories in VR and for storytellers to master skills in telling stories in first person point of view through VR. The second challenge is from the content consumption hardware perspective. Currently, VR goggles are only purchased by early adopters. Even for these early adopters, they have to compromise between video quality and hardware affordability. Hardware affordability hinders the speed of adoption while the video quality largely influences audience’s experience. One other major challenge is audience’s motion sickness with VR due to various reasons. Storytellers will have to overcome this through finding the balance between content length and richness or hope future VR technology advancement could tackle this problem.
If all problems stated resolved in some extent, there could many more possibilities. One possible path is to create stories from multiple characters’ perspective in first person point of view. Specifically, different characters in the same story view and experience different things, and these various events provide different perspectives on the same subject; audience can consume the story from different characters’ eyes and then relate to the subjects on different levels. Storytelling, especially documentaries, will have many use cases for this approach. One other possibility is to develop multiple storylines for the same character based on audience response. This approach could make stories ‘adaptive’ and very similar to what many RPGs are today — the storyline develops based on choices game character makes. For example, films can be shot in multiple plots in smaller segments; audience will be directed to different next sections of the story based on their preferences and responses from the previous part. Some projects have been done in adaptive storytelling in journalism, and new technology can push this into the next new level .
We would like to thank our professors, especially Joost Bonsen for meeting with us personally to discuss ideas and trends, Cynthia Breazeal, and Sandy Pentland.
— Jaida Yang and Paige Boehmcke