A New Dimension in Filmmaking
What is possible when filmmakers begin to engage with the bleeding edge of volumetric imaging technology.
Filmmaking has entered the spatial dimension. Breakthrough research in computation and photography is creating images and videos that are no longer flat, but truly 3D representations of reality. These techniques are currently in research labs and large tech companies and have not yet found their way into creative tools. But when spatial imaging becomes accessible to storytellers, it will enable an entirely new dimension in filmmaking.
The easiest place to peak into this new world of volumetric photography is in the latest version of mapping apps created by Google and Apple. These maps have become so detailed that they’ve spilled from the flat screen and into space. These three-dimensional cartographies cease to resemble maps on a page and begin to look more like virtual worlds from video games.
This screen recording of Apple Maps — a program that comes bundled on every Mac computer — zooms in to see a 3D view the Lower Manhattan. Thousands of aerial photographs have been combined using computer vision algorithms to synthesize geometric models of real world buildings. Flying over them is just a click away.
As I observe these imaging technologies pop up in software applications that millions of people use everyday, it sparks my imagination. Will virtual versions of our physical world become a parallel reality? Could such a vast, spatially connected mirror-world contain all of our memories, the way home movies can only do in snippets? Or maybe these spaces will become a stage for a new forms of fiction—movies that viewers can literally move through!
Whatever form it takes, volumetric photographic worlds are dripping with creative potential.
But I’ve noticed a troubling trend in these mapping applications that suggest my inclinations aren’t the direction these technologies are headed…
The above screenshot is taken from Google Earth. It’s a summer day in lower Manhattan and the street is oddly empty of vehicles. What’s more uncanny is the blur of bustling traffic that can be seen in the glass reflection on the building to the right. It’s apparent that the creators of Google Earth have systematically scrubbed all traces of individuals. Almost — you missed a spot, robot!
Despite the photographic detail, the intent of these maps is still to be an informational document and not a photographic record of any specific moments or story.
If filmmakers and photographers begin to work creatively with volumetric photography, what new kinds of stories can be told that couldn’t be told any other way? My hunch is that because these forms are freely explorable and large scale, they are uniquely suited to telling stories that aren’t linear. Volumetric films will incorporate open worlds where viewers discover the stories within.
For example, there is no lack of human drama taking place online. Check out this visualization of Facebook’s friend graph. Each line in Paul Butler’s visualization below represents a friendship between two people who live in different cities. That’s a lot of long distance relationships!
The image tells the story of a globally connected world. En masse these friendships outline the geography of continents. It’s as if our relationships are lights visible from space. The visualization is awe inspiring, yet zoomed out to this scale no specifics of any of these individual can be made out.
Is it possible to use the volumetric filmmaking technique to tell stories that capture the vast humanity playing out within these globally networked communities?
This question was one of the driving creative ideas behind the CLOUDS documentary. In collaboration with filmmaker Jonathan Minard, we set out to create an interactive documentary that depicts a globally distributed community of artists, hackers, and designers who collaborate through networks to create new software tools for artistic expression with code. The documentary traces the origin of open source toolkits for artists, specifically Processing and openFrameworks, back to their roots at the MIT Media Lab, and into a future where algorithms play an increasingly important role in our lives.
CLOUDS was created using the tools of the community it documents. Taking the form of an interactive application, much like a video game, the documentary allows viewers to follow their curiosity through the network of ideas in the film by navigating it in three-dimensional space. The forty interviews were conducted using a bespoke 3D scanning video system — which Alexander Porter and I packaged and released into the DepthKit — to depict the artists as dimensional forms as if they were constructed by code.
The documentary fuses volumetric video and social network visualization to transport viewers into an imaginary world inhabited by this community — a world of code and creativity.
You can download CLOUDS and explore over ten hours of interviews and seventy generative visualizations contributed by the artists in the film. For deeper immersion, it features a virtual reality mode using Oculus Rift.
The documentary is the first step in my creative journey to deeply explore the potential of interactive storytelling using volumetric filmmaking.
I had the (mis)fortune of being captured on Google Street view while working on CLOUDS in London. Although my identity was algorithmically erased through automatic face blurring, I managed to swipe this image back and call it my own. It is a memento of an important time in my life. I cherish it.
Even though the Google Streetview car happened to take a photo that is important to me, let’s not leave it up to them to make this medium meaningful. As the most advanced technologies ever conceived are used to create maps of the entire world, we need artists and filmmakers to use these same tools to ensure that the important stories of our time are told at rivaling scales.
I’m reminded of the Jorge Luis Borges short story “On Exactitude in Science”
…In that Empire, the Art of Cartography attained such Perfection that the map of a single Province occupied the entirety of a City, and the map of the Empire, the entirety of a Province. In time, those Unconscionable Maps no longer satisfied, and the Cartographers Guilds struck a Map of the Empire whose size was that of the Empire, and which coincided point for point with it. The following Generations, who were not so fond of the Study of Cartography as their Forebears had been, saw that that vast Map was Useless, and not without some Pitilessness was it, that they delivered it up to the Inclemencies of Sun and Winters. In the Deserts of the West, still today, there are Tattered Ruins of that Map, inhabited by Animals and Beggars; in all the Land there is no other Relic of the Disciplines of Geography.
— Suarez Miranda,Viajes devarones prudentes, Libro IV,Cap. XLV, Lerida, 1658
Thanks to my collaborators at Scatter & DepthKit, Alexander Porter, Yasmin Elayat, Mei-Ling Wong, and Kyle Kukshtel, for being a brain trust to incubate these ideas. More soon…
