2023, the Year Of Mixed Reality (But Only If Passthrough Technology Works)
As more brands and artists come up with augmented reality-based stunts from Louis Vuitton to McDonald’s, the arrival of the ‘AR-era’ is not even a question anymore. This is the mission Mark Zuckerberg believes in, we’re also waiting with bated breath for Apple’s announcement of its new AR glasses and the ecosystem it will inevitably bring. However, the arrival of this era has two crucial elements: passthrough and handtracking. In my previous post, I wrote about handtracking. Now let’s take a closer look at passthrough.
First, let’s start with the terminus technicus. AR (augmented reality), VR (virtual reality), MR (mixed reality), and XR (extended reality) are technologies that allow users to experience and interact with digital content in a way that blends or overlaps with the physical world. AR involves overlaying digital information in the real world, typically through a smartphone or tablet camera. VR is a fully immersive experience that replaces the real world with a simulated one. MR combines aspects of AR and VR to create a hybrid experience that blends the real and virtual worlds. XR encompasses all of these technologies, as well as any other immersive experiences that blur the line between the physical and digital worlds. To put it simple, I’ll just use XR from now on.
Passthrough in the context of XR refers to the capability of a device or system to allow digital data to be transmitted or passed through to another device for processing or output. It allows users to see the real world while wearing such next-gen glasses by using cameras built into the device to capture the real-world environment and then compositing it with the virtual environment being displayed within the device.
By allowing users to see their surroundings, passthrough can help to ground them in the physical world and reduce the sense of disorientation. It’s particularly useful in situations where users need to move around, like when using VR for exercise, physical therapy, or even shopping so that users can see obstacles in their environment.
Passthrough helps users orient themselves in the physical space and avoid accidents or injuries. But what’s more important (and indeed, quite cool) is that it can enhance the immersion of XR experiences. For example, a user wearing an XR headset with passthrough, could look around a virtual environment and see real-world objects and people in the periphery of their vision, which can add to the sense of presence in the virtual world.
Creating passthrough technology is a complex process that requires the use of advanced cameras, sensors, and algorithms to achieve a high-quality, low-latency video feed that accurately represents the real world. One of the main challenges is achieving a high-quality image of the external world. This needs specialized cameras and image processing algorithms to capture and transmit clear, detailed images.
The latency of the passthrough video feed can also be an issue. Real-time tracking of the user’s surroundings allows the passthrough feed to adjust in real-time as the user moves around. If the video feed is too slow or has high latency, it can cause a sense of disorientation or motion sickness for the user. Real-time tracking also poses extra challenges: it needs high-quality sensors and algorithms that can accurately track the user’s movements and adjust the passthrough feed accordingly.
How is the technology now?
The big race today focuses on image quality and better user experience. As the ways we use the internet and our devices are changing, our approach and needs will change accordingly. Soon we’ll initiate interactions in space and interact with spatial digital content. Static images should be easily transformed into virtual reality, creating detailed, realistic digital items. This is what many companies and startups are currently working on. Such solutions will soon be used on e-commerce platforms and across many industries.
Passthrough is a key important feature in product visualization. Seeing surroundings on the display will allow users to see how the actual product (technically, its “digital twin”) would look in the context of their actual surroundings.
For example, if a furniture retailer is using AR to allow customers to visualize a new couch in their living room, passthrough help showing customers what the furniture would look like in their actual living room, rather than in a completely virtual environment. Thereby they get a better sense of how the product will fit in their space and make more informed purchasing decisions.
As the market pushes for this technology in these devices, and there’s (obviously) no research backing up the use and the experience, in March 2023 we started a research partnership with the Budapest University of Technology and Economics (BME) about the future in spatial environments. Students will help research user experiences & customer behaviour in immersive spaces, along with the ergonomy of handtracking, using next-generation Oculus devices. The semester will also give the students, about 20 future product designers a unique possibility to experience and learn about web3, nextgen internet and the metaverse.
Will keep you posted about the results!
Take care,
Udeme
