Taking Turns to Plot

Collaborating with the computer to create storylines.

Examples of storyline visualizations created through collaborative design between the AI agent and the designers using PlotThread.

PlotThread allows designers and AI to work iteratively to create expressive storyline visualizations. The tool uses a novel reinforcement learning framework to train the AI in generating high-quality storylines while collaborating with the designers. Using this tool, designers can create and customize storylines easily and the trained AI recommends possible improvements to make the visualization more aesthetic and compelling.

Storyline visualizations are a powerful method to reveal the evolution of stories and present important narrative elements. Their expressive nature and their ability to summarize events in a cognizant manner make them a popular tool for visualizing books, movies, and even meeting minutes! However, creating storylines is an extremely tedious task that requires a balance of narration and aesthetics. Although existing tools such as StoryFlow and iStoryline have been developed to completely automate this process, they are not comparable to the expressiveness of hand-drawn storylines and can often result in unpredictable outcomes.

So how does one make this tiresome task of creating storylines easier and more user-friendly where even a novice user can create expressive storylines? Tang et al. (2020) propose a mixed-initiative approach to creating storylines where the AI and the designers can collaborate to create storylines that are aesthetic as well as expressive. Overcoming the challenges of a limited dataset and the temporal nature of data, they developed a novel reinforcement learning framework that trains the AI to generate high-quality storylines and supports collaborative design. Using this mixed-initiative approach and novel reinforcement learning framework, they develop PlotThread that allows users to easily create expressive storylines through flexible, user-friendly interactions as well as possible recommendations from the trained AI.

PlotThread consists of a Menubar for (a) loading scripts, setting canvas, and exporting storylines; A Toolbar for interactions (b) to (i); Buttons (j) and Suggestion Planel (k) activate/stop the AI agent and present the AI-generated layouts respectively; A Setting panel (m) and an Embellishing panel (n) are used to change storyline parameters and inserting icons and images. The red lines in the interactions change the black lines (original layouts) into blue lines (desired layout).

Interaction Design

PlotThread interface consists of a menubar, toolbar, setting panel, Embellishing panel, AI trigger button, and suggestion panel to provide a balance between fine-grain and high-level interaction. Interactions like shifting, bending, and scaling allow the users to optimize models and control storyline layouts whereas interactions like transforming, attracting/repelling, relating, and stylishing allow the users to design the storylines at four levels: character, relationship, plot, and structure levels.

PlotThread in Action

Let’s understand how you and the AI agent can work together to achieve your design requirements using PlotThread. Here is one way of creating a storyline visualization for a simplified version of Justice League.

(i) Authoring process and (ii) initial script used to create storyline visualization of Justice League using PlotThread.

Using the initial script created, you first generate an initial layout using PlotThread. You then change the initial layout by (a) shifting the order of characters separating Steppenwolf and (b) repelling the part where Superman leaves the other superheroes. Then you (c and d) transform the layout as seen above to show increment in superheroes’ recruitment. The dashed ellipse represents the transform area and the solid line shows the transformation shape. You then (e) trigger the AI agent to look for inspiration and (f) select a storyline from the suggested layouts. You work on detailing and refining using (g) scaling and bending to emphasize closeness. You also use (h) Relating and (i) embellishing to enrich narration and reveal the relationship among the characters. You embellish twinned lines for intense fights, add pictures and text annotations, and change line colors to finalize your visualization.

The Future of PlotThread

While PlotThread supports easy designing of expressive storylines, the authors have plans for its future development:

• Diverse Sketch Styles: Future versions of PlotThread will explore various sketch styles that can be used to enrich the narration of storylines.
• Faster AI agent: While the AI agent of the current version is time-efficient for the authoring process. Further work will be done on employing GPUs and parallel programming to improve its time performance.
• Creating Visualizations from Scratch: PlotThread aims to make the creation of storylines as effortless as possible. To alleviate the labor-intensive task of preparing an input story script, future study is being conducted to create visualizations from scratch and improve the collaborative design workflow.

Currently PlotThread is implemented on a personal computer as a proof of concept and authoring process of the AI agent.

Research Paper

• Tan Tang, Renzhong LI, Xinke Wu, Shuhan Liu, Johannes Knittel, Steffen Koch, Thomas Ertl, Lingyun Yu, Peiran Ren, Yingcai Wu. PlotThread: Creating Expressive Storyline Visualizations using Reinforcement Learning. IEEE Transactions on Visualization and Computer Graphics 2020.