Beat Saber is the Flappy Bird of VR
Answering the Golden Question
“When and how will the future arrive?”
When will VR be ready to take over the world?
One school of thought is that: immersive interactivity leads to engagement, and that’s how popular applications arise, popularity spells adoption, and the future arrives that way.
There was an XR training programme in early 2020 that was promoting a guest-lecture workshop on hand-tracking for “immersive interaction” (which was pretty impressive tech itself). The organisers seemed totally convinced that object manipulation in VR is the path to the future. The proposition was so starkly lacklustre that the promotion video had to paste in screenshots of “testimonial” comments like “I don’t think everyone can appreciate what’s happening here” to stress how significant it was, along with dramatic stock soundtrack for parody action movie trailers.
You can probably tell from the title already, this article will talk about why playability is not the answer.
Capturing a painting
If you search for a photo of a known painting, you might find many instances of these photos. Many of them can look quite different from one another. The results vary with lighting, exposure, angle, and so on.
There is likely a set of technical answers that address the question:
“How do we document a painting with photographs, for illustration?”
We can discover best-practice setups to get reliable results and establish sensible standards. Practical questions have practical answers.
The answers are subjective, if there are sensible answers at all, when you ask:
“Is it possible to faithfully represent a painting as a photograph?”
“Is it possible to accurately capture an image of a visible thing?
Even if the object in question is conceptually a flat surface (whereas dried paint is not… nor is iconography), the question is no longer about optics.
The “paining” as a work of art is also not just the physical thing that hangs on the wall. There are often multiple attempts at the same concept of a painting. There are sketches, studies, designs buried underneath the paint.
The painting is an artefact of reaching for an idea.
A painting as an instance
For some titles or subjects or ideas, there must have been hundreds if not thousands of paintings attributed to each. Henner did it, more than once. Titian could also do it, also more than once.
See? Sometimes the versions look very, very different. The one in the middle is “plump and numb”, while the one on the right projects palpable emotion. Painting emotional states is a great art…
The different versions illustrate at least two things about paintings:
- A “painter” could be actually a whole studio and crew — a production line with a spectrum of skill levels, sometimes even operated like a franchise. One concept is sold in many copies and variations, in response to various commercial demands, from murals to “postcards”.
- Painters work in iterations. For masterpieces, they “fix” their work over time, to get closer to the original concept.
When we see the distinction between a concept and one of its incarnations, it becomes clearer why a piece of installation artwork can be unmounted, taken apart, moved to another location on loan, reassembled, and still remain the very piece of authentic work.
The object you can see is a construction, an attempt to show you the concept — and that concept, at its best, is the expression of some poetic truth.
Editing everyday reality
A thought experiment:
What if we do the inverse, and try to turn a photograph into a painting?
We can apply some filters to a photo, then analyse it in the style of “accidental renaissance” as if it were a famous work of art.
The calm, imperceptibly mischievous facial expression of both the flame-throwing man (while directly staring at the viewer) and his accomplice are truly precious, depicted by no more than a few simple contour lines. The light from the phone screen held in the woman’s hand illuminates and reflects off one side of her cheek as she turns to observe the flame, unusually nonchalant with a mild hint of amusement.
Behind, several sleepy-looking additional characters sit around a drinking table, with feet up on the window sill, or with a facepalming hand over the forehead, like a chorus adding emphasis to the lack of excitement. In the far background, a balcony looks out to the dim city.
Let’s take the thought experiment even one step further. Let’s imagine we had commissioned someone (maybe from a village specialised in mass-production of oil paintings) to make an oil painting based on this photo.
Elements can then be edited. For example, if the overhead lighting is too bright, we can make it duller. If the patterns of the chequered shirt or of the scarf are too distracting, we can change them into something else. We can tweak the angle of the facepalming character so the posture is more obvious. We can make the bottles more distinct. We can make the flame more structurally vivid, more colourful. We re-coordinate colours, fine-tune composition ratios… We can omit objects that don’t add any meaning or impact, and mute unimportant minutiae. We can make more prominent what we want to draw attention to.
More real than reality
The point is, being able to paint, or more precisely, being able to have something painted in oil, gives us the ability to “edit reality”.
More crucially, this “editing” does not detract from the truth, but on the contrary, gets us closer to the truth we want to express.
Even photographs are not simply photons to pixels. Every photograph a composite of light over time through a lens, either causing reactions on film, or these days all in software. We can manipulate the settings, we can use long exposure to capture a dim background and then a second flash to capture the foreground with pre-calculated focus. We can do post-production.
What you see is the result of aggregation and computation, and editing upon editing, to show a sunset, or to show the northern light — not as the camera sees it, but as the artist sees it. That is art: the skilled editing of a medium to express an idea more true than what you see day-to-day without it.
Installing upon a scene
Art also means the mastering of techniques and the medium — to create what may or may not have been there. If you can paint, you can create a scene, a whole environment, with settings and characters and expressions (so skilfully done that the gaze is at times shown by no more than a speck of paint).
When you are that skilled, you gain the superpower of doctoring reality to elevate a truth in your imagination — you can create things that would not otherwise exist — you can overcome the shackles of reality.
In modern times, tech and special effects and teamwork will do as well.
Answering the question “What is the point of art?” simultaneously solves an array of paradoxical riddles, such as “How can an artwork be restored and remain authentic?” The artwork is not just there to “entertain the eyes and amuse the mind”, it’s not just “Ecce homo” to attract tourism or “banality of banality” to make a statement about mass mentality. The value of art is not set by auctions, it transcends popularity or how much reaction it provokes. The point of art is not to display ego or to reflect what’s in vogue.
The point of art is to show timeless truths about humanity.
Breaking the glass jar
I look forward to the day when VR can render convincing fantasy or stylistic environments (e.g. you could be in a world that is made of watercolour). Entire immersive worlds can now be the medium.
VR is also different from video games — and it’s not about the graphics. The sensory response that feeds back to your motor control creates an instant sense of altered presence, even without fancy graphics. Unlike video games which (partly due to the legacy of lacking in immersion) are too distracted by gamification, VR is a world in which you can simply be.
It’s more than creating or being in a scene that truly sparks joy, like a quiet stream in the forest or a sunset over the ocean’s horizon. It’s more than a state of aesthetic wellbeing achieved through an immersed experience and appreciation of great art or simulated beauty. At a more profound level, VR is a stage for “experienced allegories”, a story where its idea is felt instead of told, and the audience achieves “redemptive eudaemonia” by living through it.
If we think of the real world as an auditorium for consciousness — your skull and body carrying your brain and experiential self around, connected to sensory input — then it’s the audience, and not the auditorium, that is going through a lived life. VR is the auditorium that just got upgraded.
Appeal derived from interactivity at the level of object manipulation is crude. It reduces the mind into lever-pressing short-circuits (a rat’s brain connected to cocaine paddles) and tricks the ego into further indulging itself by cheap illusions of extension (cf. the rubber hand experiment).
VR will one day surpass R (reality), because R for most of us is just inane. We are trapped in the glass jar of cruel and petty gods. The point of VR is not to entertain. It is VR’s duty to transcend R.
Answering the Golden Question
What could bring about the age of VR? It’s the worlds that have to be good enough. A few things could help:
- Automated world building, visual and speculative — worlds built as working software, not just assets or a wiki.
- Solving the “Hill of Crosses Problem”: going beyond texture assets, adaptively generated details as you zoom in to each level of detail.
- The construction of “stylistic worlds” (think of Henryk Siemiradzki’s paintings, the incredible details from architecture to flora to landscape to lighting) has to move away from “manual and craft-intensive” to machine-powered, so that adequate mise-en-scène is abundantly available.
- Digital humans?
- Mild psychedelics?
- Next-generation glasses.
