Immersive Production: mixing theory and practice at Columbia University School of the Arts

As I prepare for a course I’m teaching this fall, I’ll be documenting and sharing what I learn along the way.

Welcome to an experiment. Throughout the semester, myself, the class, and a number of special guests will work together to design, prototype and test the next iteration of Frankenstein AI, a ground-breaking work developed and produced in collaboration with the Columbia University School of the Arts’ Digital Storytelling Lab. Frankenstein AI mixes storytelling, performance, play, code and social impact to create a contemporary reimaging of Mary Shelley’s seminal work.

A dancer interacts with Frankenstein AI at the Sundance Film Festival
Participants share stories of isolation and connection in an effort to educate Frankenstein AI about humanity at the Sundance Film Festival

Launched as part of the Sundance Film Festival’s 2018 New Frontier program, Frankenstein AI envisions Mary Shelley’s “monster” as a naive, emotional, and highly intelligent life form, an artificial intelligence. The project’s connection to Shelley’s text is meant to provide a familiar metaphor through which to examine Artificial Intelligence and its impact on humanity. Over the next two years, Frankenstein AI will travel to festivals, museums, conferences and universities around the world. This is a unique opportunity to help shape a global project that is working at the edge of machine intelligence and human creativity.

A look at Frankenstein AI at Sundance

Digital Storytelling III: Immersive Production is a mix of theory and practice. Teams of students work to design, build and deploy a digital storytelling experience which is staged for the public at the end of the semester. The course combines project work, mentors, emerging technologies and collaborative methods to create a dynamic hands-on immersive environment that mixes story and code.

Learning Goals
In this course students will:

Gain an understanding of team-based new media production through lecture content, mentoring, and hands-on experience in the creation of a significant digital storytelling work.

Practice critiquing digital storyworlds through regular peer review during the design, implementation and deployment of team based works.

Develop deeper design research and thinking methods through ideation
processes in the creation of their specific piece.

Practice digital production processes that exercise iterative testing loops, minimal viable product creation, and agile methodologies.

Learn emergent business models and skills in order to prepare them to be
adaptive leaders and team players within a constantly shifting digital media

How the course works
1. Students are broken into teams 
2. An RFP (request for proposals) is presented to the teams
3. The RFP details a theme, location and time of a live event that will be staged on campus at the end of the semester.
4. Teams work together and with mentor/guest speakers to design, build and deploy a digital storytelling experience

Mentors/Guest Speakers
The course has mentors/guest speakers who are accessible to the students
throughout the semester who cover the following subject areas…

Experience Design 
Creative technology
Machine Learning
UX strategy & design
Service design/product development
Data Science

Frankenstein AI is a copyleft project that is released under a creative commons license and is designed to be remixed and shared openly.

  • All skill levels are welcome and the course is open to students across the university.
  • Students who seek to earn credit may register for independent study.
  • Please note that this class is limited to 12 students and a full syllabus will be provided in advance of class.

Syllabus (subject to change)

The following is a living breathing document that will change over the course of the semester.

Week 1 — September 10th
Welcome, Intros & Goals
- Playtest
- Lecture: An introduction to Story Thinking
- Exercise: World Cafe — surfacing the collective intelligence of a group
- Check out: Reflection and next steps
- Please read — The Strange and Twisted Life of Frankenstein — Jill Lepore

Week 2 — September 17th
- Check-in
- Lecture: It’s Alive: an overview of Frankenstein AI
- Playtest
- Check out: Reflection and next steps
- IFP week + Story I/O collaboration opportunity

Week 3 — September 24th
- Check-in 
- Lecture: A Short Conversation about Conversations with Romy Nehme
- RFP (request for proposals) is shared and explained 
- Prototype an immersive dinner party 
- Check out: Reflection and next steps

Week 4 — October 1st 
- Check-in
- Lecture: Designing for an Aesthetic + play test session 
- Fire Side chat: Sensory Storytelling 
- Check out: Reflection and next steps

Week 5 — October 8th
- Check-in
- Lecture: The Art & Craft of Finding the Core
- Teams form and begin design process
- Design Constraint #1 — Surface memories, emotions, fears and hopes
- Check out: Reflection and next steps
- NYFF collaboration opportunity

Week 6 — October 15th
- Check-in
- Case Study: Pulling back the curtain on an immersive production
- Class discussion
- Review current class prototype
- Design Constraint #2 — TBD
- Teams iterate their prototype 
- Check out: Reflection and next steps

Week 7 — October 22nd
- Check-in
- Lecture: Bridging the analog and digital divide
- Exercise: Building experiences that work for One & Many 
- Playtest: the class presents their work for a small group of working practitioners 
- Our guests work with the class to improve the prototypes 
- Global collaboration opportunity 
- Check out: Reflection and next steps

Week 8 — October 29th 
- Check-in
- Lecture: MDA 
- Prototyping Cafe — teams continue prototyping
- Check out: next steps

*Weeks 9–13 (November 12th, 19th, 26th + makeup class TBD)
- Check-in
- Mentors/Guest Speakers based on class needs
- The class continues to prototype and test their digital storytelling experience that they’ll stage at the end of the semester 
- Check out: Reflection and next steps

Week 14 — December 10th
- Class setups up the experience
- Guests arrive and are briefed
- Digital storytelling experience is stage
- Guest feedback
- Open discussion on next steps and possibilities 
- Check out: Final Reflections

Please note that due to an academic holiday falling on Monday, November 5th we’ll need to schedule a make up class. We’ll determine the best date and time for this once the semester starts.

Grades are based on
50% collaboration and teamwork
30% on final project
20% based on documentation

Office hours are available upon request. They can be scheduled on Monday’s before or after class.

Mondays 6 pm — 9 pm

Tuesday, September18th 
IFP Week time TBD *optional opportunity

Saturday, September 22nd 
Story I/O 10 am to 6 pm (Lenfest Center for the Arts) *optional opportunity

Saturday, October 13th 
NYFF time TBD (Film Society of Lincoln Center) *optional opportunity

Special European Showing (details will be shared in class) *optional opportunity

Monday, December 10th 
Run prototype of immersive experience that class creates for a small group of participants in 101 Prentis

Class will make use of the following…

Courseworks — course related communication

Medium — for documenting our prototyping

Google Drive — for teamwork

Columbia DSL prototyping network — to collaborate with creatives and partners around the world

What people are saying about Frankenstein AI

“Frankenstein AI is truly the New Frontier of storytelling and media. As smart objects and machines become a critical part of our societal infrastructure, artists hacking, inventing and defining the meaningfulness of these new capabilities is essential for designing a human-centered version of our future. Audiences of Frankenstein AI commented about how powerful the emotional experience was as they participated in the project. This gives us a glimpse into how humans and artificial intelligence might interact for cathartic experiences as these systems advance.”

Kamal Sinclair
Director, New Frontier Lab Programs at Sundance Institute

The 8 weirdest things on-screen at Sundance 2018

“Frankenstein AI: A Monster Made by Many intelligently updates Mary Shelley’s tale based on modern fears about artificial intelligence. The main installation was one of my favorite parts of Sundance’s New Frontier lineup; it culminates in a large, glowing screen — the titular monster — asking AI-generated questions to audience members, then using their responses to evolve. (One of my session’s: “Why do humans like having sex, even though they can see in color?”) But a one-time performance also added a live dancer (Jacinda Ratcliffe) who worked in tandem with the machine. Her movements were choreographed beforehand, and whenever the AI analyzed a set of answers, it would formulate a “mood” that affected her dance, delivering the information via earpiece. I still have no idea how intelligent or monstrous the AI actually is, and how much it’s just a device for heightening our awareness of physical human interactions — which is arguably one of the most interesting things about the project.” — AR

A.I. as Auteur: Interactive ‘Frankenstein’ Experience Shows How Artificial Intelligence Could Be Storytelling’s Future

“The projects in this year’s New Frontier section at the Sundance Film Festival ranged from the latest cutting-edge virtual reality to decidedly low-tech dance and theater performances. The most thought-provoking of these pieces incorporated media forms across the technological spectrum to explore the leading scientific development and debate of the day: the ascendance of artificial intelligence (A.I.).

As economists and social scientists debate the ramifications of A.I. on their respective disciplines, artists, unsurprisingly, are feeling compelled to do the same. Accordingly, the pieces at Sundance chose not to question issues like the future of work but rather cut to the philosophical core of the matter: in the era of AI, what does it mean to be human? As a recent Medium piece asked, riffing on critical theorist Walter Benjamin: What is the work of art in the age of algorithmic reproduction?

Perhaps the most ambitious piece at New Frontier, A.I. or otherwise, was the multifaceted transmedia work “Frankenstein AI: A Monster Made by Many” by Rachel Ginsberg, Nick Fortugno and Lance Weiler. Billed as a “participatory installation and performance,” the total experience, which will tour this fall, spanned over two hours and involved elements of theater, dance, collaborative storytelling, philosophy, Mary Shelley and, of course, artificial intelligence.” — READ MORE