The State Of Play In Spatial Computing/XR In 2024
The early Q1 2024 launch of the Apple Vision Pro firmly cements 2024 as the center of the Gemini (Mercury-Gemini-Apollo) Phase of Spatial Computing/XR (the middle of the middle).
A set of special devices, all interesting and well engineered, define our current Gemini Spatial Computing/XR era: the Apple Vision Pro, the Meta Quest 3, the Magic Leap 2, the Varjo XR-4, and the Brilliant Labs Frame.
Each of these devices is grasping at different aspects of what I will call XR Infinity (the ultimate Spatial Computing/XR design) which does not yet exist.
The team at Argo Design (who worked with me at Magic Leap for many years) have a conceptual design which embodies aspects of this ideal:
https://argodesign.com/work/mixed-reality-x
In my embodiment of XR Infinity, we will have a device that has the weight of a good pair of Oakley sunglasses (<< 100g), a FOV of 90–120 deg, voxel based segmented occlusion, a very refined natual digital light-field signal rendering capability, all day battery, human level retina resolution with a natural color spectrum, incredible AI and graphics capability, an animal like surface of brilliant, intelligent sensors, and more cool bits that I can not disclose (right now).
I do believe that we will see XR Infinity type systems in the Apollo phase of Spatial Computing/XR, which in my estimation begins in 2028 and will continue until 2035. When we hit 2036 and beyond, we will enter the Mars Exploration and Outer Galaxies Phase of Spatial Computing/XR (with BCI integrations) — and that is when things will get really weird, super cool, and very, very strange.
So where we are now?
The Big Guys (Or Drinking SeaWater But Loving It):
This group contains the Apple Vision Pro (600 to 680 grams), the Meta Quest 3 (515 grams), and the Varjo XR-4 (665 grams + headband 365 grams). They go from big to bigger to biggest — with a price spectrum that ranges from $499 USD to well over $4000 USD. By sticking to optimized VR displays, a number of things come more easily: colors, black, and field of view. They all offer a pass-through video capability to see the world, with the Apple and Varjo systems being at the top-end of visual quality.
I do not agree with everything in this article by the Verge (Nilay Patel), but here is a good summary of where The Big Guys are (it is specifically about Apple, but applies well to the the class of VR headsets simulating AR through pass-through video):
“There are a lot of ideas in the Vision Pro, and they’re all executed with the kind of thoughtful intention that few other companies can ever deliver at all, let alone on the first iteration. But the shocking thing is that Apple may have inadvertently revealed that some of these core ideas are actually dead ends — that they can’t ever be executed well enough to become mainstream. This is the best video passthrough headset ever made, and that might mean camera-based mixed reality passthrough could just be a road to nowhere. This is the best hand- and eye-tracking ever, and it feels like the mouse, keyboard, and touchscreen are going to remain undefeated for years to come. There is so much technology in this thing that feels like magic when it works and frustrates you completely when it doesn’t.”
and
“The other way to look at the Vision Pro is that Apple knows all of this, but the technology to build the true AR glasses it has long hinted at is simply not there — so the Vision Pro represents something like a simulator or a developer kit. A dream factory for people to build apps and meaningful use cases for the imagined hardware yet to come: true optical AR glasses that let you share digital experiences with other people. In that framework, this Vision Pro is the hardware Apple can ship right now to get everyone thinking about these ideas while it pours all those resources into the hardware it wants to build.”
The teams at Apple, Varjo, and Meta have all accomplished incedible feats of engineering and system design. Meta deserves a lot of credit for packing so much in at such a low price point-even with whatever subsidies and losses they carry per sale. They have all pushed and pulled at our current understanding of physics, materials, computing, and manufacturing to create these amazing systems that are far beyond the complexity of PCs, laptops, and mobile phones. Designing these systems is closer to the complexity of designing high-peformance aircraft or satellites — there are endless trade-off and impossible asks of disparate teams. I believe that they all know that we are all midway up the mountain, and that there is a lot more to climb. Design critiques should not be aimed to diminish the work of these teams (they all have amazing engineers). It should serve to push the teams and the designs in the direction of optimal good, to the endstate designs that enable the Spatial Computing/XR sector to soar.
As an example, we know that weight on the head is not good or comfortable, and that it needs to be reduced drastically. The best way to do that now is to move a large portion of the weight off the head into the pocket or belt (Magic Leap does this, Apple kind of does this). We know that we are not there yet in terms of all the tech and system designs, but specific design choices can make devices much more useable. A cable is a nuisance, but too much weight on the head and cervical spine creates fatigue and unnecessary biomechanical loading. There will be endless debates as to what is “too much”, but getting between 200–300 grams is a nice sweet spot for now, and pushing hard to get below 100 grams on the head is a goal for all. At 200–300 grams with intelligent load distribution on the skull, many will be able to enjoy Spatial Computing/XR for hours. At under 100 grams with intelligent load distribution, you open the door for a very wide market-but that must be coupled with the amazing visuals and capabilities that you see in devices like the Apple Vision Pro and Varjo.
The True AR Guys (Heading At The North Star, Icebergs & Yetis Do Not Scare Us):
There are a number of companies playing here, but I will focus here on the two (Magic Leap and Brilliant Labs) which I think represent current ideals in the pursuit of True AR Spatial Computing — where you can see the world directly and blend digital + physical in a more natural way.
The Magic Leap 2 represents an arc towards XR Infinity — and it is also represents a more maximum philosophy of True AR Spatial Computing/XR. It has a 70 degree diagonal FOV, which is the current world’s best in True AR Spatial Computing. It has a powerful AMD CPU/GPU, an extensive sensor suite, and it is also the first True AR Spatial Computing to demonstrate the ability to create black on demand, with its segmented occulusion technology. It weighs 260 grams, which is not at the < 100 gram ideal, but it is as comfortable as wearing a pair of good headphones for hours (due to its weight distribution).
If you follow the arc from the Magic Leap 1 to the Magic Leap 2, you can begin to see the arc that leads to the XR Infinity. The number of generations to get there is a function of sustained funding and the sustained innovation and passion of the team — but it is a design arc pointed at what many believe is the North Star Target: True AR that can become the device that integrates our mobile phones, tablets, TVs, movie screens, and an array of new spatial presence experiences and games that have only begun to be imagined. The Magic Leap 2 was also designed with AI capability in mind-it has the processing power, sensors, and connectivity to develop Jarvis like capabilities (from what I have seen to date, much of the system’s power is vastly untapped). It is trying to find a balance point between systems like the Apple Vision Pro and super minimal systems like the Brilliant Labs Frame.
The Brilliant Labs Frame represents the far end of the spectrum, and I love very much what it is trying to be-the thing we will all wear, everyday, most of the day, and use most of the time. XR Infinity represents a design ideal where using it is thoughtless and frictionless (like breathing and good magic) — but when we take it off we will feel immensely diminished (our IQ will drop, our understanding of the world will fade, we return to Kansas from Oz).
The Frame weighs less than 40 grams(!) — which is where we want these things to be. That is an A+. It also has a FOV of 20 degree (ughh) diagonal- which is where things are today in the Gemini era. From endless experiments, I think that most users will find that 70 degrees FOV is at the threshold of good for many True AR tasks, but what you really want is 90–110 degrees (what you may experience in a good pair of sunglasses). FOV beyond 110 degrees is also amazing-but you can imagine those systems for more extreme or professional experiences (high-end games, flight simulators, etc.) There is also the advantage True AR systems have in terms of the real world. You see the real world in a wider FOV than the digital FOV, and you have reasonably good peripheral vision. In a VR system you get a SCUBA mask effect, which can feel quite different in many ways than True AR.
More importantly, it integrates multi-modal AI systems-so it will act to augment our intelligence through our vision. I believe that AI will (and should) become a fundamental aspect of all good XR/spatial computing systems-so it is wonderful to see such AI centric design thinking here (privacy and other matters require immense efforts to get right, so let’s hope Brilliant Labs nails it).
I remain very hopeful and optimistic that we will see very cool design and engineering improvements through the Gemini phase. Apple has made a big splash in the market, which means that investment and supply chain capabilities are flowing and improving. There is enough goodness represented in the current best systems that we can all now understand and glimpse a spectrum of the Spatial Computing/XR Future.
I also hope that the best players and teams aim for the ideal of XR Infinity (and its variants). Within XR Infinity there is the philosophy of Neurologically True Reality, which represents a Zen pathway of deeply respecting our body and the world, and of developing technologies that bend to meet the world and our biology, rather than force our natural systems to become captive in artificial Mayas. I think that this is the most misunderstood and overlooked aspect of Spatial Computing/XR design.
Neurologically True Reality is much harder to conceptualize and design-but it is the design state where people can find a natural harmony with computing and the world at the same time. Perhaps we will split into the Naturals and the Artificials-but my hope has been, and remains, that Neurologically True Reality can be a bridge between the worlds we create and the world into which we are born.
R. Abovitz
February 10th, 2024
Earth & New Storyworlds
