Hyperrealism in the Age of Ubiquitous Spatial Computing

1/x — This is a series of exploratory essays where I will be recording my thoughts on a topic I’ve been thinking about in the AR/MR space. Let me know if this resonates with you or if you dispute these claims — I’d love to chat about it!

Special thanks to Mikal Khoso for proofreading this piece.

Currently, the majority of the elements being used for augmented or mixed reality applications are cartoonish or of low-fidelity. They’re easier to make, pleasing to view, and their low polygon-count makes them quick to load.

Static hyperrealistic elements have also been created that are detailed enough for them to fool the average viewer. Their use in mass-market products have been constrained by the computational power that is required of them.

Dynamic hyper realistic elements have also been created but are still stuck within the uncanny valley — the gray area where the object is too real to be an a pleasing animation but also too fake to make our subconscious believe its real (Ex: Saya and Mica). Elements in the uncanny valley feel disturbing to us.

Building off of Moore’s Law and the developmental speed of technologies which support the AR/MR ecosystem such as 5G, computer vision, and AI, there seems to be a good chance that we will be able to create cost-effective, digitally rendered elements that are indiscernible from their physical counterparts in the relatively near future. Many industry leaders predict this to be within the next 5–10 years. When that happens, that will be a major inflection point for AR/MR.

I’ve been toying around with the idea of what we should do once we reach this inflection point.

If we were to create linear scale, I see two extremes.

One is to create all digital elements in an aesthetic style that create an obvious delineation between elements in the physical world and elements in the digital. I’ll refer to this as “digital distinction” from now on.

The other, is to strive for the hyperrealistic renderings that imitate the physical world. I’ll refer to this as “digital integration” from now on.

Digital distinction creates a world where the digital and physical are two distinct entities that exists and interact on the same plane. The two would form a symbiotic relationship and creation in each of these spaces would be different in both process and aesthetics.

Digital integration creates a world where the digital and physical are perceived as one and the same. Humans would not know the difference between the two.

In both cases, certain elements would exist in a shared reality, accessible to everyone and others would exist in individual realities, accessible to the specific viewer.

However, it seem to me that the AR/MR industry is currently in blind pursuit of the digital integration reality and that concerns me. While it’s the sexiest and arguably the most technically interesting, it also brings along some moral hazards to consider.

New construction occurring near exclusively in the digital space. Physical construction only serves as structural elements

• Near complete reduction in demand for craftsmen of physical objects
• Displacement of roles traditionally in physical construction towards computer science
• Lack of opportunity for impoverished people without access to the mediums we use to interact with the digital reality -> lowering socioeconomic mobility and opportunity of the poor
• A bleak world for those who cannot afford headsets. Information asymmetry that has been supercharged by the internet, will now be supercharged again by AR/MR leading to greater inequality.

Confusion between designer-constructed interactions for digital elements and the collectively-constructed interactions of physical elements

• Treatment of digital elements carrying over to our treatment of physical elements (people). May lead to a dwindling of empathy due to the constant interaction with human-like, yet ultimately mechanical, object that exist only to serve. See this blog post by Hunter Walk
• Interactions models built by a homogenous population will only lead to people being left behind. Gestures differ by generation, culture, and gender at a minimum. See IDEO Gestural Research
• People will be left behind as gestures change from generation to generation.
• Different operational models of elements which are visually and functionally the same. Ex: Information kiosks — spatial computing would allow for an improved interaction model compared to physical kiosks

Unrealistic beauty standards for all things, especially human beings

• All that is created digitally will be aesthetically perfect.
• People can choose how others perceive them, leading to them ignoring their physical traits. Potentially good for democratizing recruiting and hiding visual bias. Negative if used to ignore physical conditions with underlying symptoms or if to hide/misportray body language.

Psychological condition of mutability. Digital elements can be returned to their previous states. Physical elements, upon destruction, cannot.

• May lead to a cultural shift towards one that revolves around mutability and ephemeralness. Save/Load.

Variation/inconsistency between designer-made interactions for digital elements and physical elements.

• Conditioning of danger negates bodily impulses to flee in actually dangerous situations.

Proliferation of Thought Bubbles

• Hyper-personalized feedback loops based on user-set interpretations of the world. Imagine your Facebook feed, but now in the physical world.

Of course, these are only potential effects of the digital integration reality — hypotheses. As noted earlier, second and third-order effects are notoriously difficult to foresee but there is value in contemplating the negative ones so that we can take preventative measures to make sure they don’t come to pass.

How can we protect ourselves against the ones who cannot foresee? On a structural level, we can become more diverse as a community, to make sure a homogenous group of similar minds aren’t the only ones defining the future for the greater whole. We can include ethicists in our discussion and give workshops on ethics and empathy to our leaders. We can consistently bring in outside voices and allow them to influence product decisions, not just those of our immediate users but also those of marginalized identities in tech, not only across traditional delineations such a gender, race, and class, but also of geographical location, age, and job industry.

I know how easy it is to lose sight of our goals, and to build solely for the sake of intellectual or social fulfillment without ever considering the ramifications of our actions. I’ve made that mistake multiple times. But I’m just one man with his own moments of weakness. What happens if we, as a collective, make this same mistake together?

Is the current state of AR/MR bringing us towards a world which loves humanity? It’s not certain yet, but at this moment, when we are building the infrastructure of and formulating the rationale behind this new technology, we need to make sure that we are leading the world towards a path which does.