Using Film Language in Narrative Handheld AR: Why the camera is and is not a camera
Handheld AR is the only XR medium that affords a framed experience. Screen boundaries lay within your central FOV. So, the experience is also “cinematic.” And can therefore benefit from film language. Here’s the standard example: Action is arranged so when you turn toward, say, characters in conflict, you “pan the camera” into the composition. The composition lacks the precision you’d see in a film. But you have a similar psychological response, in part because your presence creates an immediacy that no close-up can match. This requires a deeper dive to have a better understanding. We’ll abandon for a moment the “camera.” It’s a concept too ingrained with long held, and in this case erroneous, assumptions. The proper metaphor involves reflections rather than direct gaze. That metaphor is the buried mirror.
In his book The Buried Mirror: Reflections on Spain and the New World, Carlos Fuentes describes a mirror that reflects the distant Mediterranean shores, and the viewer’s immediate reality rooted in the Americas. It reveals “old worlds” clashing: The Amerindian, the African, and the Iberian. And the viewer sees themselves in each, though from the perspective of the “new world” created, literally and figuratively, from the clash. And this leads inevitably to self-reflection.
Likewise, in AR we see the distant shore reflected in CGI, a literal other world. And our mundane reality is there too, just outside the screen. We experience a “new world,” which is their synthesis. Yet, their distinctions remain. And this causes creative tension, in a well-designed experience. The result is suggestive presence.
What’s commonly known as “presence” is “immersive presence.” And specific to VR. I’ll skip the technical aspects because what matters is what they do. They reinforce the perception that you’re inside another world. AR can only suggest that world because you’re always aware of your own. So, your presence is partial. And relies heavily on engagement that’s psychological ― an intellectual, emotional and cognitive focus much deeper than “user engagement.”
Psychological engagement resembles what happens when we read riveting fiction. Except images, sound, etc., do in physical space what language does on the page. I’ll discuss that further in a future article about AR as literature.
And self-reflection? This is where film-style composition comes in. It encapsulates the essence of the moment; every facet from the emotional, to commentary on the wider plot. It acts like a real-time poster in that it communicates at a glance. And you reflect on the total effect rather than the parts. The composition harkens back to one of the oldest traditions in art, the allegory.
Consider Kawanabe Kyōsai’s painting, Hell Courtesan and Ikkyu, shown above. We see the courtesan’s moment of enlightenment. She realizes that life is fleeting, represented by a monk dancing with skeletons. Images in her kimono reinforce the message. (IMAGE) Kawanabe’s Hell Courtesan and Ikkyu.
You can plan compositions in VR and in wearable AR (e.g. Magic Leap). However, you get muted effects because you don’t have the hard boundaries that, in framing the image, create dramatic tension. Caravaggio’s Conversion of St. Paul illustrates this perfectly. It works because Carravagio leaves us no “free” space to contemplate a man’s religious awakening. We see the drama, a silent opera, on a too-small stage. Widen the canvas to a landscape painting, and the power would dissipate.
Compositions don’t have to align with a character’s story arc. That’s because they exist for you. They offer insight into the story by highlighting key points. Intensity amps up when compositions coincide with critical moment in a character’s struggle (e.g. proclaiming innocence to a skeptical courtroom). What about stories built on the discovery model, where you piece together the plot from environmental clues? Well, film language still applies. You choose objects whose relative arrangement, their layout as seen from a given position, evokes a response. Such compositions work cumulatively rather than individually. But they do have impact.
One More Thing
Film techniques come with a caveat: Scale. Let’s say you watch small figures in conflict on a coffee table. If they’re 5 cm tall (approx. 2 inches), your POV will mimic a drone or crane shot. That’s great for things epic or chaotic. But you cannot see detail, so might miss a plot point. A film director would cut or zoom to show that a character is thinking about committing a crime. Handheld AR can accomplish the same through other methods. Or a narrative designer might use scale to “rephrase” the plot point. Whereas a zoom would effectively say, “that’s the perpetrator,” an AR approach would be, “the perpetrator is there, somewhere, and that’s the problem.” However, in a room-scale narrative, the situation would change. If you’re thinking, “Given circumstances that are visible around me, who do I think will commit the crime?”, you can read body language to find out.
Generally, the closer characters are to you in size, the more film language techniques are available. One reason is film history. Movies have developed so that the medium shot, showing an actor from the wait up, has becomes the norm. Action and dialogue convey the story. But we also carefully read body language, such as facial expressions. We can only do that in handheld AR one of two ways. Either the character is small with exaggerated features; a head, arms. Or they’re bigger and within human proportions. Design and intent dictate what constitute big and small in an experience. For example, for a sit-down experience ― meaning characters moves around you instead of you around them ― a 1.1 m adult character will appear to be nearly 2 m. And that has an important implication. A character can tell much of their story through body language instead of through big actions that take up space. The more nuance you can see in body language, the more you’re likely to speculate about a character’s intentions. And the deeper your psychological engagement.
Your thoughts?
