Whose Medium is it Anyway? or: If Film is a director’s medium, what is AR? What is VR?
“Find your character’s story. Then work through how they would tell it.” In other words, a character’s story lay between the written lines. That’s what the director told the actors. At the time I wrote plays for a small repertory company, a way to remain creative outside of design. The director interpreted the work for realization on stage. But it was up to the actors to breathe life into the characters. And inhabit that world. That’s why Theater is called “an actor’s medium.”
But what about XR? Does the presence of actors make it an actor’s medium, when programmers play important roles? And if characters are CGI, what then? Whose medium is it? The animators?
I think the answer starts with two related questions that double as tests. First, what feature defines the medium? Secondly, how is the feature exploited to convey the narrative and keep us engaged? Before looking at XR, let’s review two other tech-based media, Cinema and Television.
Cinema is visual. That’s what defines it. How do we know? It was silent for nearly its first three decades. The camera exploits the visual nature to present the story and keep us engaged. And what about actors? Don’t we experience the story through them? In any given movie, yes. But in terms of the medium, no. How many times have we read about last-minute cast changes? The new actor will give a different performance based on how they understand the character. But the film will keep the original feeling ― unless the replacement gives a truly awful performance. So, this test tells us that Cinema is not an actor’s medium. But replace a director far into a production, and you change what winds up onscreen. That’s because they must shoot and arrange the script into a compelling visual story. They make the camera-related decisions. Actors’ performances fit into the story the director wants to tell. Ideally, every emotional nuance aligns with the visual language. As a result, we don’t see the story, but the director’s interpretation.
What about Television? It too is visual. But it’s a long-form medium. Stories run for episodes, and often seasons. A single narrative will have several directors. And we don’t notice.
A television episode shot by a well-known director must fit within stylistic conventions. Without those, viewers get confused. One can only imagine Game of Thrones shot by Tarantino in one episode, complete with the type of dialogue for which he’s famous; followed by an episode done in B&W with lots of handheld shots, in an homage to Truffaut. A television series designed to allow directors leeway will embrace, rather than suppress, stylistic differences. American Horror Story is a prime example.
A new actor may replace a longtime favorite. But, outside of telenovelas and soap operas, the character they play will have the same story arc. And this gives us a clue about why Television is a writer’s medium. It requires plots, character arcs and dialogue strong enough to hold the viewer over numerous episodes.
The defining feature of narrative XR is that it’s spatial. Physical proximity to characters, whether they’re directly in your line of sight, etc., affect emotional engagement, which influences what you understand of the story. Confronting an angry character from across a room will lead you to one conclusion. But you’ll draw quite another if that same character provocatively “gets in your face.”
Spatial also means the environment does more than, say, reflect characters’ personalities. Movement through the environment affords understanding about the narrative. This can be as simple as gathering clues at a crime scene. Or discovering that the POV from each clue location reveals as much if not more information than the clue itself. Simply put, in spatial storytelling you “get what’s going on” by exploring. And you explore differently in each medium, which determines how you stay engaged.
Let’s start with the media that comprise VR: Panoramic, and Free-motion. The former has gained some currency as a term. Free-motion VR has not. I use it for the sake of discussion. Both terms represent the main experience types.
Panoramic VR
In 360° and VR180 videos, you watch the experience happen around you. They’re presentational in that regard. And you see only what the camera has shot; or if CGI, what was rendered as viewed through a virtual camera. You can have limited interactivity, such as moving from scene-to-scene. But it’s what you do afterwards that matters, namely looking around to figure out what’s going on. It’s all heavily visual. And this has an interesting effect. The narrative has less ambiguity than otherwise, expresses a sharper opinion, because visual exploration leaves wide latitude for interpretation. Too much latitude could potentially cause you to “misread” what’s going on.” So, we have a “camera-based” experience that affords limited options to interpret the story, as in film. Camera-related decisions, and production model, also follow film. I think it’s safe to say that Panoramic VR is a director’s medium.
Free-motion VR
Free-motion VR is what we generically think of as “VR.” You can move among objects and characters. And they can react to you programmatically. The environment can dynamically suggest mood through visuals and sound. It can create and reinforce certain perceptions about the story. And affect your reactions to it, which can influence decisions on how you interact with it. And any characters, though probably voiced by humans, may nonetheless rely on AI. An analogy to this, a singular artistic experience created from disparate elements, is music. Sir George Martin took simple pop tunes, the odd philosophical bit (e.g. ideas from the book The Psychedelic Experience), added baroque and classical elements, or created effects from scratch; from which he produced The Beatles’ masterpieces. Produced. That’s the key word. He guided artists and engineers rather than directed, since the process was collaborative.
Can we conclude that free-motion VR is a producer’s medium? Broadly speaking, yes. In my opinion, the medium favors creatives whose thinking has shifted away from the camera-centric. Or who’ve come to the medium from those where interactivity is key, notably gaming; or performing arts that rely on understanding movement. That said, other production models can result in compelling narratives.
AR
Narrative AR is the opposite of VR. The experience transforms part of your physical space into a different world. And you’re always aware of it. So, the CGI elements most important to the story, whether characters or objects, must carry it. And pull you in till the outer world fades to a kind of white noise. But in deciding “what” will suggest “where,” the creative can choose if a given prop will illustrate some aspect of a character or place. This is a contextual approach because props, aside from those important to the story (e.g. a weapon), do little more than frame the alternate reality. The result can feel theatrical. We understand the story through characters whose actions happen to take place on “a set.” However, an environmental element can push the story alongside the character, rather than merely serve them. And do so by eliciting reactions through shape, color or texture ― any of which can be dynamic. Perceptions about the narrative change in response. And they in turn influence decisions on how you interact with it. I think of this second approach as surrealist, a strong term to be sure. But it refers to AR’s potential to put the fantastic within the everyday. A talking unicorn on your kitchen table is an example.
So, who’s most responsible for the above? The writer? Possibly. But I think that’s true only for episodic series. Like television, the story must be strong enough to keep you coming back. For non-episodic content, I believe narrative AR is the narrative designer’s medium. That’s because where actions “physically” take place relative to the person who experiences them, when, and why in respect to the overall narrative, are fundamental to how the medium works. However, a more accurate conclusion might be the narrative designer as writer, who uses the raw story, sound and spatial elements the way a traditional writer uses parts of speech. Maybe this is a kind of XR Auteur Theory? Neither conclusion suggests that AR narratives produced along a film model, director-based, are wrong. But that the medium seems to favor narrative designers.
“Whose medium is it?” I’ve given you my thoughts. But, what do you think?
