Data Science and AI in Film Production

Enabling Agile Storytelling with AI

Creativity, Inc. is one of the most insightful and thought-provoking books I have read. Ed Catmull recounts his personal experiences, and talks in depth about the processes he created that made Pixar one of the most successful companies in the business. I strongly encourage you to see the video interview below, where he describes the “Brain Trust.”

The key takeaway is that great stories are not written in a flash of inspiration, but are a result of an agile, iterative process. I couldn’t help but notice the similarities to writing great software code within an agile framework. Write, Evaluate your product, Rewrite, Repeat until you make it great.

In filmmaking, writing and pre-production are the stages where the highest leverage decisions that affect the final product are made. But when we look at the way most people approach pre-production, it resembles a Waterfall Process.

Even if we can’t be as assiduously iterative as Ed Catmull and his Brain Trust, being agile helps in adapting and responding to unforeseen events in production. One would have to just browse the Reddit channel to see how Murphy’s Law is in full force in most productions.

Most filmmakers would agree that the most tedious and time-consuming parts of pre-production are: a) script breakdowns, b) storyboards and shot list generation, c) optimizing schedules, and d) creating budgets.

If AI and Data Science are able to automate most of that process, then filmmakers can focus their time on the more creative and human-centric aspects. It makes filmmaking more fun, and generating these parts automatically reduces pre-production time.

Script Breakdown

The root of pre-production is the Script Breakdown. The script is the idea, which needs to be turned into a physical product. This is the stage where all the elements that are necessary to turn the script into to an actual physical manifestation are extracted.

Using Natural Language Processing (NLP) methods, End Cue’s Agile Producer platform automatically extract elements in different categories, like the main characters (and which characters interact in a scene), the dialog for each character, the type of scene, props required, visual or audio effects, and we can estimate the length of the scene and the number of shots.

Off-the-shelf NLP entity-extraction methods won’t do, which is why we trained ours with a human-annotated dataset to learn which entities need to physically actualized and which ones don’t. For example, the word “bees” might be classified as an entity in the sentence “A flock of drones buzzes like bees.” Our AI learns that “bees” in this context is a metaphor, and does not need to be depicted on film.

Story Boarding

Each scene may need to be filmed in multiple shots. From the script text, we infer the number of shots. Then using a text-to-image classifier, we classify the type of shot according to shot size (close up / medium shot etc.), shot type (single focus, eye level etc.), the type of camera movement and equipment. We search for images in an archival database for the one that best matches the shot. This generates the storyboard.

To make this happen, End Cue created a training corpus of scenes and corresponding shots from a library of movies. Computer vision, trained initially on human-annotated data, ran through movies frame by frame and classified each frame by shot type and scene description.

This is where studios and production companies can utilize their archival footage to create training data for AI. If the storyboard is licensed by the production, then there is an opportunity to monetize the archival footage.

These shot lists and storyboards are meant to be recommendations. It is the Director’s end decision to accept the suggestions, or change it entirely. However, it has been our experience that it is easier to criticize a suggestion, and choose something different, than it is to start from a blank slate. Multiple options for the images can also be generated, giving the directors a chance to pick the most appropriate one. This greatly speeds up the tedious process of shot list generation.

Scheduling

Shooting a film in the linear of order of scenes rarely makes sense since we can stitch and fix things during editing. Schedules are thus created to optimize the use of locations, actors, props, and other variables.

The Agile Producer includes a scheduling tool to input the variables that have an impact on the optimal schedule:

• Cast schedules and calendars
• Location dates
• Availability of equipment and props (for e.g. vehicles)
• Weather (as best as we can)
• Cost and budget constraints

This creates a combinatorial optimization problem. If there are 100 scenes, there are 2¹⁰⁰ possible ways of ordering them (which is a number with 31 zeros after it). There are many schedulers that have been developed for other industries, like airlines. Given the type of constraints, and considering we have a dataset of solved schedules, we developed our own AI techniques based on Mixed Integer Linear Problems to solve these more efficiently.

Compared to our past methods, where the scheduling optimization was done by hand or using rudimentary tools, we can expect to realize cost and time efficiencies in double digit percentages. I’d assume that for a studio production with a much bigger budget and more variables, this number could be very high. We can run the optimization on scheduling data retroactively. By that, I mean, we could take previous productions, run them through the optimization process and see how much we could have saved. That is obviously a regret-inducing process, but shows us where we could have done better.

From the schedule, it is straightforward to create the calendar. Then we have a common calendar that can be shared with the whole crew. And for each day, we can generate a pdf of the call sheets that can be easily shared and printed.

Budgeting

The Agile Producer platform represents the budget in a tree structure, across multiple levels. The lowest level is generated automatically from the elements from the script. Based on training data from previous budgets, we can estimate the costs for each line item. Some parameters, which involve costs that are negotiated have to be entered manually. These parameters will of course be different for independent production companies or studios.

The items at the lowest level are then grouped into higher categories. For example, our algorithms estimate the costs of the “Camera Car” item. And all these elements are categorized into the Transportation Captain and Drivers line item. These categories then automatically fold into higher level categories.

Pre-production data for greenlighting decisions

What does this mean for the most fundamental question: which projects to take on? We looked at our previous decision-making process for greenlighting projects, and compared it to how it has evolved after bringing AI into the picture.

Previously, we would analyze the script and look at whether it is a good story. We would run the script, as well as other variables through our forecasting models, such as genre, main casting, the current trends, etc. We would greenlight the project based on that. Then we would create pre-production estimates. This separation existed because creating production estimates required a different group of people, with different skill sets.

The Agile Producer platform allows us to immediately obtain production estimates from the script. Even if it isn’t the final production schedule or budget, these are new data that can impact the greenlighting decision. So now we have the script and production estimates to run through our forecasting model and inform our greenlighting decisions.

This is very impactful for development. There is greater confidence to take on “risky” projects, because we can lower our uncertainty with data.

Story Writer Visualization

Where can AI augment script writing? After all, the script makes the biggest impact on the content that we are creating. End Cue developed the Agile StoryWriter platform, to help writers write better stories using Deep Learning. Writers can use the tool to visualize their stories to see if there are any plot holes, see the effects of changing character interactions, and generate dialogues in different styles. I aim to dive into the story writing platform in a later post (I have a discussion on the Data Skeptic podcast). The view that I’ll present here is how producers can use the Agile StoryWriter to look at the effects of changes to the script.

The program ingests a script in pdf or Final Draft format, and automatically generates a scene-by-scene visual representation of the script as a graph. We can see the key character interactions from the graph. Multiple concentric circles arounds a node represent different characters in the scenes. The edges represent which interactions affect each other.

The platform can be used to compute “Butterfly Effects.” How does adding, removing or changing a node affect other nodes, i.e. scenes, in the script?

Murphy’s Law can strike at any time during filming. A critical scene can no longer be filmed: actor gets the flu, location gets flooded, equipment breaks down, all at once.

With the Agile Storywriter, we can remove the node, and find the other scenes that get affected and need to be rewritten. We may realize that there are a couple of later scenes that no longer need to be shot. The new script can be regenerated and improved efficiently in a way that the writer and director are happy with. The Agile Producer can be used again, to regenerate the new schedules, calendars and budgets.

All of this can be done in the span of a day, something that would have otherwise thrown the entire production off course.

Agile Storytelling

In concert, End Cue’s AI-based tools enable Agile Content Creation. Because it greatly compresses the time for script rewriting and pre-production planning, we can use it to explore What If scenarios in a data driven way.

We can run the modified script through the Agile Producer to generate production data. Coupled with a Forecasting model, and especially if there are data on distribution predictions, the system can be used to iterate through different versions of the story and find the most impactful one.

Studios could look at the effects of removing certain high-budget spectacles to see how that affects predicted outcomes. Independent filmmakers can see the effect of adding a female lead to the story, and how that impacts the chances of signing up distributors.

A platform like this could take a linear process, and make it iterative. You would still need to augment the human component, the culture of discipline, and experimentation through iteration to be like the “Brain Trust,” but AI can certainly help you on your way!