The Future of Storytelling Media: Changing the Game and the Implications

Across societies and cultures, stories have always fascinated us. Not only do they help spread and develop our culture, but they also provide an escape from the sometimes mundane world in which we live. Harry Potter, Star Wars, and The Hunger Games have seen roaring success because they transport us to Hogwarts, a galaxy far far away, and District 12. While media experiences have entertained us thus far, they have been standardized and passive ones. The future of storytelling lies in making the experience more personalized and interactive, bringing us even deeper into storytelling worlds.

While movies & TV shows have offered tremendous amounts of entertainment value, as Gavin Baker points out, it is difficult to make a movie significantly better than those of the past. New, interesting stories may emerge, but entertainment improvements are marginal. While improvements in picture quality and CGI capabilities provide better viewing experiences, they do not change the game.

In this piece, we will cover the path to changing the game, through three levels, and the implications it will have. We will dive into:

1. how data will be used to understand the user experience;
2. artificial intelligence’s role in developing narratives;
3. what interactive media experiences look like;
4. why storytelling as we know it will still play a role in the future;
5. why producers like Netflix and HBO might not be best set up for success; and
6. what this means beyond storytelling.

This version is a long-form piece, and while I encourage you to read this, I do recognize that this probably covers more material than some may be interested in. Accordingly, I’ve also put together a short-form version with just the most important points.

For a tremendous review of the history of storytelling, how we’ve gotten to this moment, and how creative IP will complement this paradigm shift, please check out The Next Frontier in Storytelling Universes and the Never Ending Desire for More by Matthew Ball.

Level One: Better Monitoring and Predicting Consumer Experience

Currently, digital media producers rely on viewership counts, box office results, and critic ratings to obtain feedback on a movie or TV series. However, these feedback mechanisms aggregate reality.

Publishing firms don’t know how different individuals reacted to the film. Streaming video on demand (SVOD) providers like Netflix and HBO may know if a subscriber was engaged based on how many hours they watched and how they rated it, but think about all those times you put on The Office and had it running in the background, but were not paying attention to it. This reveals a flaw in the current system: companies don’t have a deeper sense of exactly how viewers reacted and how that drives overall viewer satisfaction. Furthermore, they only know how satisfied the viewer is with the overall end product — the entire movie or TV episode. They might fail to see that a subscriber enjoyed the main storyline but was completely disengaged from the side plot or that despite watching three straight hours of a show, they were mainly just texting their friends or scrolling through social media.

In order for SVOD providers to discern how their subscriber enjoyed different parts of the storyline and better understand what types of storylines deliver the best results for their individual viewers, they would need to optimize consumer experience by obtaining frequent and high fidelity data. Without this, changes made to optimize experience are simply shots in the dark and feedback loops informing future improvements are slow and not as valuable.

Collection of data could include monitoring internet response to content or tracking user eye movements, heart rates, and facial expressions. With more data to understand how audiences will react to story lines, content producers will be able to:

1. better predict what stories will be successful with artificial intelligence; and
2. employ artificial intelligence to deliver better content.

Better data will provide tremendous insights on content response, but privacy concerns will be the main driver of how this information is obtained and to what level this is possible.

Predicting successful story lines

Currently, for pre-release feedback, content creators use brain trusts and test screens, where the movie is shown to a few viewers to solicit feedback before audience release. However, test screenings only capture the feedback of the people selected and may not be leveraged due to concerns of leaking the plotline.

When provided with the plethora of data needed to understand storylines that work well, SVOD producers can develop artificial intelligence tools that will predict audience reaction to a story line with significantly increased accuracy. Thus, they can produce content with a much more informed view of how audiences will respond. For example, a director may identify that a character does not have its intended effect on the audience and alter the story.

Employing artificial intelligence as a storyteller

As its capability develops, artificial intelligence can play a larger role in not just anticipating box office results, but also by actually crafting narratives that it knows will likely deliver good results. Rather than serving as a validator of a story’s likelihood of success, it can start to make recommendations to tweak story lines or even develop narratives of its own.

This technology in its required capability does not exist yet, but it is not far from what is possible today. Artificial intelligence has already produced a movie trailer and a full script for a car commercial.

Level Two: Leveraging Dynamically Changing Content

With the data fidelity needed to support frequent and reliable feedback loops, the next step is to optimize by making content changes, in real time. By combining historical knowledge of successful narratives with current audience reaction, narratives will change in real time to optimize for user outcomes. This difference is a fundamental paradigm shift, diminishing a foundational part of storytelling of the past — the role of the storyteller in crafting the narrative. Instead, the stories of the future will overwhelmingly focus on the consumer of the story and be delivered by artificial intelligence. It will determine what happens next virtually as it becomes visible on the screen. This means that while we are excited and on the edge of our seats to find out what happens next, so is the narrative!

Dynamically changing content through traditional methods is not feasible as filmmakers simply cannot produce enough content to keep up with the provided feedback loops. To fully take advantage of the engagement data, content must be produced quickly enough to keep up with the feedback loops. Thus, artificial intelligence technology, as described above, would need to advance considerably to continue the narrative and update the visual and auditory components accordingly.

This does not necessarily mean all TV shows and movies of the future will be animated. Some might, but photorealism is possible too. Initially, content will be created using photo & film references to develop augmented frames, where individual pixels are exactly as they are in real life. Today, Unity is able to take film references, model the scene, and augment the footage, producing video indistinguishable from reality.

For people, Soul Machines is using reference film and photos to create standalone artificial humans. Among their pursuits, Soul Machines is developing a virtual version of Black Eyed Peas star will.i.am. There could even be a future where we could conjure up characters and actors of the past, as Unity CEO John Riccitiello suggests. As these technologies advance we could develop completely new, photorealistic frames of humans in real time.

While this technology is inspiring, we are a far ways away from developing content that dynamically responds to feedback loops. For reference, it takes Pixar about 24 hours to produce one frame. Using supercomputers, it took two years just to render Monsters University! (For an otherwise fascinating look into animation and content production, check out how Pixar has evolved its capabilities over the past 24 years and the end-to-end process for animating Toy Story 4).

Producing new content on the spot, without sacrificing quality, might prove challenging. Just as difficult might be managing technology’s ability to use its in depth understanding of people to influence what they view. We are already seeing today that faster feedback loops and engagement optimization can unfortunately start to develop echo chambers, confirming political and social beliefs, and increase political strain as sides more deeply hold onto their beliefs. Therefore, this must be managed properly or dynamic content will be met with strong resistance.

Level Three: Immersing the User

Beyond just developing narratives that look to optimize for the viewer, the audience will also have the ability to intentionally influence the narrative. Before, they had no control over what they view, but in Level Three, the viewer takes control.

To help understand the impact of being an observer versus a participant, consider a key component driving the success of Disney’s theme parks. Disney World is about YOU, not Disney. As Matthew Ball writes in Digital Theme Park Platforms: The Most Important Media Businesses of the Future:

“Disneyland is an experience involving many moving parts in harmony, like an orchestra. Everything has to be tuned, what you hear, what you smell, what you see,” how you see it, the speed at which you assimilate all of that, just like a film, is choreographed.”

While an obvious difference between a film and theme park is that you are physically in the theme park, the key difference maker is that:

“[Disney doesn’t] control the camera, because the camera is you — it’s you when you come to Disneyland”.

In both films and theme parks, the characters and the scene are meticulously choreographed and perfected, but in films, the storyteller keeps control of the camera, and thus the narrative and potential experience. However, when you are the camera, you can optimize for the experience that is best for you. In its parks, Disney provides all of the important infrastructure for its visitors, but then the visitors can see what they want, allocate their time how they want, and immerse themselves as deeply as they want. TV shows and films of the future will offer what Disney does in its parks.

Optional story lines

Translating this to digital media content, instead of just being an observer of the dynamically changing world, the viewer will have direct influence over what content they are consuming. In a rudimentary way, media companies are already realizing the value of this and are acting on it by providing optional story lines for the user to dive into. For example, Star Wars has its main storyline in Episodes I-IX, but it also has a few side plots — Disney has produced the films Rogue One: A Star Wars Story, Solo: A Star Wars Story and the TV series Star Wars: The Clone Wars and The Mandalorian to name a few. They have also announced three new series to launch in the coming years. Fans can keep up with it only if interested, focusing their time and energy where they like.

A seamless choose your own adventure

In the future, the user will be able to materially impact storylines. Viewers will transform from passive observers to impactful agents.

In doing so, choose your own adventure books will effectively be brought to the digital world. Though premature, this exists today too. Netflix released, through the Black Mirror series, Bandersnatch, a story with decision points that produce different outcomes in the story.

Though certainly novel, the movie did not grip the media world as the new format for storytelling because it is still lacks full empowerment of the viewer as many decisions were trivial. There were instances where decisions would yield a “wrong choice” response, show the same video, regardless of which option you chose, and impact something relatively meaningless like the soundtrack. This prompts the question — were these really impactful decision points? To engage viewers, it must feel like they have meaningful influence on the storyline.

The main limitation here is content production — Bandersnatch needed to have a limited number of true decision points because they could only produce content for so many scenarios. Producing for more scenarios would have linearly increased their budget, rendering it unfeasibly large. Developing artificial intelligence technologies to produce video in an instant, as discussed in Level Two, will lower the marginal production cost of new content. The number of choices could become limitless. You can even make a decision whenever you want, rather than just when prompted, like in Bandersnatch.

Though extremely rudimentary and limited, technologies representative of a similar idea are being built. Talk to Transformer, based on the Open AI GPT-2 algorithm, shows how a modern neural network can create text. It can recognize inputs from news articles, stories, song lyrics, poems, recipes, code, and HTML — for example, it can know the characters from stories like Harry Potter and The Lord of the Rings. Using the same base algorithm, AI Dungeon is a text generation program that can take a user action and further the story line in a coherent way (though, it certainly has its limitations).

Image courtesy of Towards Data Science, AI Dungeon

As these technologies begin to realize their full potential, imagine not rooting for a character in Game of Thrones to end up on the Iron Throne, but actually helping one of the characters rule the seven kingdoms. Better yet, take them all down and end up on the throne yourself! For those who want it, this can increase story engagement immensely. This has the opportunity to truly make media a first party experience — one where the viewers are as much a part of the story as they want to be.

This should not be confused with getting everything that you want — that is boring! Being challenged and surprised yields a much more rewarding and entertaining experience. Because of that, viewers will only be able to influence narratives, not dictate them.

The Reality of Level Three

However, it is important to note that more immersion is not always preferred— a lot of media is consumed passively, sometimes as a background activity to another task. Put simply, consumers will choose what content they consume based on the amount of attention they are able to give to it. Though a fluid spectrum is likely to exist, for simplicity’s sake, viewing options can be categorized into (1) passive viewing, (2) dynamic choose your own adventures, and (3) a hybrid of the two. Viewers can choose their option based on how focused they can be. Within a particular story, the platform can look to judge the attention capacity of the user and make minor adjustments accordingly to engage the consumer more or less.

Passive viewing, incorporating the ideas from Level One and Two, and a seamless, digital choose your own adventure, discussed in Level Three, have been covered in depth. But, what would a hybrid look like? One possibility is participating in massively interactive live events (MILEs).

MILEs

Discussed in detail by Matthew Ball and Jacob Navok, one of the ways to think about them is taking Netflix’s Bandersnatch example, except every decision is made by the collective viewers live in real time, changing what everyone sees and permanently impacting the story. A notable example is Reddit’s place. Reddit’s place was a large blank canvas that allowed users to color pixels from a choice of 16 colors for 72 hours. However, they were limited by being allowed to change just one pixel every few minutes. Check out a phenomenal overview of the creation, protection, and destruction that played out here.

Image courtesy of Sudoscript

MILEs create a blend of the passive viewing and choose your own adventure experiences by offering immersion optionality. If looking for an immersive experience with a sense of ownership and participation that comes with it, users can contribute to making changes for the narrative. To get a sense of this, imagine the sense of camaraderie that comes with the below attack and defense of the American flag in the Reddit’s Place where teamwork was crucial.

Another example is Twitch Plays Pokemon, an attempt to crowdsource the playing of Pokemon Red. To do so, Pokemon was streamed on Twitch and users could control the game by typing instructions into the chat. The game reached an average concurrent viewership of 80,000 viewers and holds the Guinness World Record for “the most participants on a single-player online videogame” with 1,165,140.

In shows and movies, there are so many moments where key decisions are made that influence the trajectory of the story line. In The Dark Knight, imagine if the audience gets to choose whether Batman should save Harvey Dent or Rachel or decide whether one of the ferries will blow up the other. In Finding Nemo, imagine if you could decide whether Merlin lets Nemo save Dory by telling the tuna to “swim down”. How great would that feel if the audience had voted to let him save Dory and they were able to!

Making decisions together is the key ingredient of MILEs. To facilitate them, there will be a “live” showing where everyone gets together to watch the show or movie. Some people can interact and participate, but those that want to sit back and relax can do so, following the action passively. In either case, the anticipation that comes from waiting to find out what will happen next will be immense. Those that are not able to tune in for the live event can watch at a later time, but will lose the ability to interact. This will increase the hype and FOMO, much like the phenomenon of watching sporting events live, versus a replay, further ingraining the TV show or movie into everyday culture and society.

Sociability will influence the popularity of choose your own adventure

One aspect that makes MILEs so intriguing is that it is an inherently social experience. Social experiences around media fundamentally enhance it. Discussing the latest happenings on Game of Thrones or Westworld and laughing over a scene of The Office increases the value of the story. To this end, choose your own adventure narratives need not be individual experiences. Join storylines with your friends and work together to accomplish a goal. In Narcos, play Steve Murphy and Javier Pena to hunt down Pablo Escobar. Work with your spouse to take over 1600 Pennsylvania Avenue as Frank and Claire Underwood in House of Cards. Gather your friend group and figure out how to survive on the island in Lost. In these worlds, people can come together as a team and engage in bonding moments that the narrative creates for everyone.

Inherently, however, there are some social aspects that won’t be replicable with this type of storytelling. Given that a group’s experience with the same story is unique, water cooler talk about the latest show is no longer possible. However, usually just the main characters and biggest moments are discussed. To this end, it is possible that while everyone has a unique storyline, there are standardized elements, such as character development and curveball events, that are consistent across all story lines, enabling discussion around how people are reacting to them.

Another possibility is that rather than having an infinite number of possible scenarios, producers could strike a balance between minimizing the number of potential story lines, while still offering the empowering sense of agency. This would enable people who experience the same story line to have discussions with each other. This can even be taken a step further where people are introduced to each other based on having made similar decisions in the plot, as Tinder has done with Swipe Night. Stories can be a way to make new friends — it doesn’t get more social than that!

Perhaps, the nature of how we share the experience will evolve. Like with vacations, where people share their experience through Instagram even though other people weren’t on the same trip with them, people may share their unique story experiences with others as a status update of sorts. Even more radically, the benefit from increased interactivity may outweigh the interest in general social discussion. Participating in storylines with a close circle of friends might be a more socially satisfying experience than casually talking about a show with acquaintances.

Though the above are a few possibilities, it would not be surprising if completely new social experiences evolve around choose your own adventure narratives. However, the extent to which choose your own adventure stories prove to have unsolvable social disadvantages, passive forms of media will continue to play a significant role.

Storytelling as we know it will not disappear

Though artificial intelligence will be able to deeply understand what can entertain us, it will not be able to completely replace human-made stories. Although it is ridiculously intelligent, after all, it is artificial — it does not know what it means to be human. It replicates humanity to the best of its abilities, and perhaps can do so quite well, but truly understanding what it means to be a human may prove to be beyond its abilities. This understanding is the value that human storytellers will be able to continue to provide.

In the movie Click, Adam Sandler’s character is given a universal remote that allows him to do whatever wants — fast forward, rewind, and pause his life. He eventually learns that being present in every day moments is what life is about. Could artificial intelligence understand the concept of time and the value of having cherished moments and meaningful relationships to produce such a novel storyline? Artificial intelligence can replicate emotion, but it cannot feel it — only a human can do that. This art of storytelling cannot be reduced down to a science.

Artificial intelligence is also confined to the data that it is fed, informing what will work. It does not have an intuition or imagination, gut feelings or visions to make logical leaps. Consider this quote from Henry Ford*:

“If I had asked people what they wanted, they would have said faster horses.”

Horses were all they knew. What they really wanted was a car. There was no data or evidence that people could benefit from this innovative mode of transportation, yet it saw incredible success. The parallel here is that it would be difficult for artificial intelligence to develop novel and visionary story lines because their algorithms will show uncertainty of success and move forward with a narrative with greater chance of success. Artificial intelligence will provide faster horses, not cars.

*There is no evidence Henry Ford actually said this quote, but it is illustrative of what thinking beyond what is currently known can bring.

Traditional Media Will Be Disrupted

The future of media will collect tremendous amount of data on user engagement, optimize for the user experience, involve the user as part of the story, and adapt story lines according to user input. However, traditional media firms are not the best set up to lead us into the future. Video games are closest to the future of storytelling media. The video game sector has seen tremendous growth and people are spending more time playing video games instead of watching TV or other media. As Reed Hastings has alluded, Netflix’s biggest competitor was not another SVOD platform with hit titles like HBO, but instead the most popular video game — Fortnite.

The video gaming industry is poised to see continued success in the future as it is developing its technology and capabilities along the same trajectory that the future of storytelling media requires. Take the 24 hours it takes Pixar to render an animated frame, referenced earlier. While it produces a spectacular end product, new content will need to be developed in real time. When you do something (e.g., move your character, shoot a gun, etc.) in a video game, you are creating a new frame, one that has never been created anywhere before in 1/60th of a second. Solving for this constraint is already baked into the technological development of the video game industry, giving them a head start over traditional media firms.

In fact, video game companies are already encroaching the film making space. Epic Games’ Unreal Engine is being used to produce scenes for Disney’s The Mandalorian. Currently Epic is just creating backdrops, but when video game companies can seamlessly drop in photorealistic characters (like the Unity CEO referenced) and autonomously develop narratives (like the work of AI Dungeon), is there anything left for producers to bring to the table?

This head start will prove crucial and it continues to grow. Traditional media has long viewed the video gaming industry as a separate customer base. But, over the past several years, the definition of a “gamer” has broadened and is becoming more mainstream. As Matthew Ball discusses, Hollywood has a blind spot to — and perhaps a reluctance towards — the video gaming industry. CEO of Disney Bob Iger has said:

“We’re obviously mindful of the size of the [video gaming] business. But over the years, as you know, we’ve tried our hand at self-publishing. We’ve bought [video game] companies. We’ve sold [video game] companies. We’ve bought [video game] developers. We’ve closed [video game] developers.

We’ve found over the years that we haven’t been particularly good at the self-publishing side…We’re good at making movies and television shows and theme park attractions and cruise ships and the like. We’ve just never managed to demonstrate much skill on the publishing side of games.”

By extension, they may also have a blind spot and reluctance towards the future of media. If they want to play a significant role in the future of storytelling, it might be time for the leadership team to become particularly good at the self-publishing side.

Traditional media is starting to take notice, highlighted by Reed Hasting’s comment on Fortnite, but it is possible the gap is too large to catch up. Even if they begin focusing their attention on making stories more about user outcomes, the video gaming industry has several years of a head start. The traditional media industry is a well-oiled machine to predict what story lines will be well received by audiences, but while they’re tremendously good at that, well-oiled machines are not very good at doing what they’re not programmed to do. When the objective switches focus from developing great narratives to producing technology that seeks to engage and entertain users by autonomously generating content (i.e., the jump from level 1 to 2), traditional media firms may struggle as many incumbents do with disruption. To be successful, traditional media companies like NBC, HBO, and even Netflix must begin to build competencies in deeper immersion like those of their video game focused peers.

Yet, traditional media may not yet have a death sentence. Video games are still lacking in two key components: dynamic narratives and minimum engagement requirements.

The dynamic narrative gap

Video games do have dynamic narratives with somewhat meaningful impact, such as those in Grand Theft Auto V and Assassin’s Creed Odyssey. However, video games need to improve them in both how often they trigger narrative changes and how they produce content for them. In this regard, they share the same challenge as traditional media. Currently, narrative changes are triggered by a decision point in the game. The story only changes when the game developer has set up a fork in the narrative. This means that content is not updated automatically based on user engagement, just as it is in Bandersnatch. Like traditional media, how they generate content is the key blocker for providing more moments when narrative changes can be triggered. Content is currently pre-written, and as such, expanding the amount of ways a narrative can diverge is still time and resource intensive. Offloading that effort to computing power, when the technological capabilities exist, will unlock this potential.

The minimum attention required gap

Secondly, video games are attention-demanding. Nearly all, if not all, of the most successful AAA, high-end video games demand 100% of the user’s attention. Video games must expand their attention requirements to also include less intensive options to be able to capture a larger audience.

Similar to the optional storylines discussed earlier, games are beginning to address this by having a core story (15–45 hours), side stories (60–80 hours), and exploration activities (100+ hours), such as in Assassin’s Creed Odyssey. This way, the game can cater to those that want to have a deep experience and those that want to stick to the basics. Although this does help reduce time commitments, this does not solve for requiring the full attention of players. Multitasking or passive consumption is difficult if not impossible — for example, it is impractical to enjoy a video game over a Sunday morning cup of coffee.

The industry is also making efforts to bridge this gap by making it possible to watch video games being played through YouTube and Twitch. This helps viewers “play” the game without the attention requirement. They get an increased sense of immersion, but someone else is bearing the attention burden, giving an effect similar to watching MILEs.

Given these two gaps, however, video games still have a head start on traditional media and look poised to keep their lead.

The Impact Beyond Storytelling

These changes in storytelling media may very well have implications beyond its own domain. As increased personalization and interactivity are brought into stories, people may begin to crave similar experiences everywhere.

Other types of media and television that are not story-based, such as sports and news, could change as well. With news media, for people who are more interested and engaged with different topics — what the viewer sees may be altered to cater to their interests.

This idea is already playing out in esports. During an event, a viewer can often choose to watch the main broadcast through the publisher’s stream, or they can choose to watch an individual stream of a player. With the main broadcast, a viewer can get overall commentary and a diversity of action, but following a streamer allows you to see their specific gameplay in much more detail. Depending on preference, viewers can choose accordingly. In fantasy football, many tune into NFL Red Zone to see if their player is going to score. What if Red Zone synced to the players on your fantasy football team (and any other players/teams you want to follow), instead of showing everything? Esports is taking this a step further by allowing the audience to interact with and materially impact the game. For example, when a video game player’s character dies, you may be able to complete a quick time event to resurrect their character. The ways in which this can be expanded are endless. Personalizing content and immersing the user can offer much better experiences.

In social media, Tik Tok, which has reached the same number of users as Instagram in half the time, has adopted leveraging artificial intelligence to determine what content its users see wholeheartedly. Unlike other social media platforms, where content is primarily sourced from your social graph — people that you are friends with or follow —Tik Tok uses engagement optimizing algorithms as the primary way to deliver content. People love it. Tik Tok’s average session durations are 1.5x Facebook’s, 2x Instagram’s, 2.5x Twitter’s, 3.5x Snapchat’s.

This will accelerate the use of AI to deliver content and experiences in other areas, and proof of value in those areas will yet again inspire others.

What does this all mean?

We have gone over a lot of ideas, a high-level roadmap to the future, and the implications. But what does this mean for you?

• If you work in the media business, rally and lead the troops to the future. Leverage some of the ideas discussed in this piece to refocus your company’s vision.
• If you work in the video gaming industry, think about whether your strategy is aligned and how you may close the two gaps highlighted earlier.
• If you are an investor, consider taking a closer look at the video game industry to deliver the returns you desire. Furthermore, think through some of the second-order effects of how other areas might be affected.
• If you’re an entrepreneur, find out an area you can stake out and contribute to this evolution.
• If you are a fan of TV shows and movies, get excited! The future of entertainment is going to be awesome. Get ready to dive into highly engaging story lines with your friends and family, or by yourself!

The details of exactly how this will play out are unclear, but what is certain is that users will both crave and expect personalized, engagement-oriented experiences. Because of that, the impact of the future of storytelling will have far-reaching implications both within and outside the industry. The media industry, the video game industry, investors, entrepreneurs, and fans alike should take note.

If you have any questions, thoughts, or would like to connect, please don’t hesitate to reach out via LinkedIn or Twitter!

Disclaimer: All thoughts and opinions are my own, do not reflect the views of my employer, are for informational purposes only, and should not be relied upon to make any investment decisions.