Augmented reality is a medium
As Augmented Reality expands beyond its original domain — of research facilities and academic articles — into the purview of artists, creators, and popular culture, it is important to take a step back and analyze both the nature of this seemingly new concept and the assumptions applied to its limited formulation. As it stands today, it looks as if augmented reality will significantly change how we perceive the world and subsequently relate to it. With such potential, it is important for us to scrutinize augmented reality and how it tends to be conceptualized.
Let us first consider a simple definition of AR:
augmented reality adds or removes sensory data from our perception of the world in order to modify our experience of reality
As an example we can look at the augmented reality game Pokemon Go. Pokemon Go adds sensory data — virtual creatures and virtual locations — to our perception. In so doing, our experience of reality is modified by the integration of a fictional world. Playing Pokemon Go is then the performance of interacting with this fictional world.
However, the above definition of AR leaves much to be desired. For example, how do we construct successful AR? How can we conceptualize AR as artists and designers?
Instead of a definition, let us focus on a theoretical statement about augmented reality: augmented reality is a medium separate and distinct from the technology and content of its production. AR is not a feature, product, or piece of hardware. AR is a channel of communication, a membrane added unto one’s world. A medium.
Box with the sound of its own making
In order to better understand the medium of AR as more than a mere product or technology, let’s look at an example decoupled from contemporary technology: the sculpture “Box with the sound of its own making”.
“Box with the sound of its own making” is a conceptual sculpture developed by Robert Morris in 1961. The work itself is a nondescript wooden box of quality craftsmanship, that sits atop a pedestal. Inside the box is a speaker that plays the recorded audio from the production of the box, complete with the sound of hammering nails and sawing wood.
While this work predates the term Augmented Reality by 29 years, when we think of AR as a medium this work fits right in. In Morris’s sculpture, an audio recording of the work’s construction plays from inside itself, augmenting our perception of the sculpture by integrating its history as part of the present. As viewers, when we understand the recorded audio as the past we then experience the box as it is and as it was, two phases of its existence simultaneously present.
augmented — the present, by integrating sound from the past
reality — the wooden box being
Evaluating “Box with the sound of its own making” presents us AR as medium. The augmentation of reality re-contextualizes our perception of the environment, and this re-contextualization itself communicates with us.
The necessity of registration for AR as medium
“Box with the sound of its own making” presents its past as part of its present through the process of registration. Registration is the integration of data so that the data is perceived as part of our real world. Consequently, the medium of augmented reality can not be reduced to a state without registration.
In fact, anything we experience is registered[grounded]. Registration is merely the perception/conception of how an object exists in the world. A television show is registered on a television screen. A simple text overlay on glasses is registered on the surface of the glasses as a virtual overlay or HUD. What we are concerned with here is meaningful registration.
Meaningful Registration: Grounding multiple realities
Meaningful Registration occurs when the method of augmentation is meaningfully related to the thing that is being augmented, i.e. the “how” of augmentation relates to the “what”. For example, in “Box with the sound of its own making”, the “how” was achieved by using a speaker playing a recording of the past, and the “what” was the perceptual/conceptual relationship between the sculpture in the present and the sculpture in the past.
Through meaningful registration, “Box with the sound of its own making” breaks its own physicality of time and place. This phenomenon occurs through the experience of an object existing in two phases of its being simultaneously. This simultaneity occurs because the two phases are grounded in the same shared reality — that of a wooden box emanating audio, and this grounding occurs through registration — the wooden box is emanating its audio. Registration is present via the experience of wooden box and audio as parts of the same object, and it is meaningful because it conceptualizes the sculpture in an impossible state, hence: Meaningful Registration.
Meaningful registration then allows us to establish a shared relationship between the real and virtual realms that is the grounding[foundation] with which we construct our augmented world. More importantly, this shared relationship affords us a conceptual material through which we can communicate.
Thinking of AR as a medium
Thinking of AR as a medium expands our mental model of AR. We only need to look back at film as an example of an emerging technology that is a medium in its own, separate from the technology and content of its production. Film is not photography, although it involves a series of images. Film is not theater, although actors, sets, and props are integral to its production. Nor is film the camera or screen, although both are necessary. With hindsight, we see clearly both the development of film as a technology and as a medium. Through analogy, we can see a similar path for augmented reality.
Technologies are tools for production
Augmented reality arose out of technological developments in computer vision and related fields. This led to a dominant technology oriented perspective as to the nature of AR. The technology oriented perspective is a conflation of the technology with AR itself as a medium. To avoid such conflations, we should understand these technologies as tools of production within the conceptual framework of AR as a medium.
Such separation bears fruit. Via the separation of technology and medium, technologies[tools] of production are democratized, bringing to light interesting and unorthodox tools.
GPS — many technologies focus on human scale registration, for example placing a digital character in the room with you. With GPS we can register objects on a global scale. Such registration can expand our world view through world centric vantage points.
Emotion Analysis — grounding virtual content does not have to be through believable physicality. We can ground virtuality through emotion. What is obfuscated can come to make sense as we navigate it in our various emotional states.
The Social Graph — For better or worse, there now exists tremendous data about all of us. Using such data we can create ways of integrating our social lives with fictional worlds. Such blurred narrative realities have the potential to highlight and explore many strange themes concerning our lives in the virtual age.
Exploring the medium: subtractive AR
The medium-oriented perspective democratizes methods of AR as well as the tools. We will focus on one such method, subtractive AR. Subtractive AR is the removal of some sensory data about the world one would normally experience. Through this subtraction, the world is experienced in an augmented way.
As an example, think of an AR app that garbles one’s native language, giving one an experience similar to that of the tourist, immigrant, or refugee in a foreign land with a foreign language. Such an app could help foster tolerance and compassion for others.
Or consider an app that offers the phenomenological experience of Prosopagnosia. Through face recognition and AR modification of familiar faces, the app would afford doctors, family, and others a better understanding of the experience of those living with Prosopagnosia through the temporary embodied experience of the condition.
As with additive AR, subtractive AR grants us augmented experiences from the modification of our senses. Through a combination of additive and subtractive AR, one can experience the world as others do. Such experiences can build empathy generation machines, generating understanding and receptivity to the experiences, views, and beliefs of others.
Exploring the medium: non-visual AR
Another compelling method of AR is utilizing our senses other than vision. As with subtractive AR, we can contemplate how non-visual AR poses significant opportunities through the description of two potential applications.
Consider the current limitations of tools meant to help visual impairment. If we could synthesize facial detection, object detection, and text detection into a data stream of audio and tactile input, those with vision impairment could understand spaces, as well as navigate the world, with ever more clarity and ease.
Or, returning to our previous example of an AR app that simulates Prosopagnosia, through non-visual AR means, coupled with modifications of visual data, an AR application that recognizes faces can overlay audio, tactile, and other data onto the sensory data of someone with Prosopagnosia, indicating and identifying familiar faces. From subtractive AR those without the condition can become more empathetic to those with; while through non-visual AR those with the condition have tools to help overcome the obstacles presented by the condition.
Further, beyond single or combined sense experiences, there is also the relationships between senses. Exploring these relationships introduces new questions and opportunities. For example, via sensory transformation, i.e. the mapping of sense data to and from one another like a controllable synaesthesia, comes embodiment hacking, experiences that expand our conceptual framework of what reality is and how we exist within it.
Closing thoughts
Through meaningful registration we ground realities — present to us through sense data — together in a shared world. The shared world itself is an augmentation through sensory addition/subtraction on a spectrum from real to virtual. As we exist in such shared worlds through sense data, we are embodied in these worlds. As such, AR is well suited for embodiment hacking and empathy generation.
AR emerged from technological development. Understanding the technologies of AR as tools of production is both necessary to create and to conceptualize AR. This path of technology birthing a medium is followed by all artistic mediums that utilize tools. Therefore, mediums such as games, photography, and film can be scrutinized in order to understand the likely future developments of AR.
To close, most writing on augmented reality focuses on enhancing our reality through an ‘AR as always on’ state. However, contrary to this prevailing thought, many strengths of augmented realty lie in temporal experiences that re-conceptualize our reality post-experience. The medium of augmented reality is what the information moves through, and the augmentation is the communication device of the artist.
More Info
This article is part of a project found at oeoe.club
