The Future of the Spatial Web: A Look at the Vision Pro and Holographic Mixed Reality Powered by Render — (Behind-the-Network: Friday, June 9th, 2023)
Earlier this week Apple introduced the Vision Pro mixed reality (MR) headset at WWDC, one of the company’s most anticipated product releases since the launch of the iPhone. Presented as the introduction of a new era of ‘spatial computing’, the Vision Pro offers a combination of stunning graphics, audio, display, and, crucially, GPU hardware innovations that are converging to make immersive media a mass consumer reality for the first time after decades of research and development.
Behind the Vision Pro is a transformation to a new communications medium that has the potential to reshape the way we interconnect with each other, becoming the next computing platform after mobile. Powering this revolution is a transition from static 2D imaging to intelligent 3D immersive media that the Render Network has been developing decentralized GPU computing infrastructure to make available at scale.
This Behind the Network (BTN) will look at some of the underlying Render Network technologies that are important to make mixed reality available and widely distributed, ushering in the emerging era of spatial computing.
The Platform for Spatial Rendering
The Vision Pro introduces a new form of media that is much more focused on visual immersion than any other medium in the history of information technology. Lifelike mixed reality content not only requires displaying media at much higher resolutions than 2D screens, but it also demands a new level of hyperrealism — where virtual computer graphics and the physical world blend together naturally. As a result, high end media for the Vision Pro will require two axes of exponential increases in rendering and graphics demands — resolution and image fidelity — both of which do not scale linearly, creating new intensive computing pressures.
Looking at the specifications of the Vision Pro it is evident that the device is built around advanced imaging, focused primarily on visual immersion, including:
- Two ultra high pixel density Micro OLED displays that are over 4K in resolution per eye — with subsequent devices likely to increase resolution to over 8K per eye and beyond
- A high performance M2 chipset which released with Apple Silicon laptops, and that are now powerful enough to achieve parity with high performance desktop rendering from half a decade ago when the previous VR / MR development cycle took place
- An advanced 12-camera array with machine vision capability for motion tracking, controller free immersion and advanced depth sensing for mixed reality spatial media
- Advanced eye-tracking technology enabling “lean back” fully immersive, hands free ultra-wide extended field-of-view 3D experiences using passive eye movements
- Forward facing high resolution stereoscopic cameras with depth sensing, enabling high resolution 3D acquisition with depth data for next generation 3D creation workflows
From the Vision Pro hardware specifications, one can see the degree to which spatial computing relies much more heavily on image immersion than any other previous form of human-machine interactivity, providing tremendous needs for next generation graphics. As a simple example, the combination of eye tracking and 4K+ per-eye displays could lead to stereo rendering demands at over 8K — creating the latitude for images to track eye movements at 1:1 resolution without cropping / zooming that degrade image quality and break visual immersion.
As a platform designed to make the future of post mobile immersive computing accessible and widely distributed, the Render Network contains a number of unique technologies on top of peer-to-peer decentralized GPU compute that are instrumental for powering the new era of spatial media that the Vision Pro is unveiling.
Octane X Integration on Apple Silicon and macOS
The underlying built-in engine on the Render Network, Octane, is unique in its optimization for Apple Silicon chipsets like the M1 and M2, providing the efficient high end rendering for the Vision Pro — with full parity with desktop rendering on mobile devices.
Already artists using Octane X on Apple devices can take advantage of the Render Network by exporting scenes created in the Apple hardware ecosystem and accessing near unlimited decentralized GPU cloud rendering power on Render via Octane X. With a version of Octane X available for macOS and iPadOS, the rendering platform already bridges between high end desktop and mobile rendering, paving the way for the same architecture to support visionOS, the operating system for the Vision Pro. As a result, out of the gate the same artist and publishing workflow for Octane X will be supported by visionOS, enabling workflows to frictionlessly span all hardware platforms from Desktop to Mobile and Spatial — all augmented by the Render Network.
Alongside Apple’s other announcements at WWDC, including the newest line of Apple Silicon chips, which offer features like the remarkable 192GB of memory bandwidth of the M2 Ultra designed for AI computing and intensive graphics, the capabilities within the Octane X hardware ecosystem are rapidly growing, offering a new install base and hardware architecture for next generation 3D developers. Already M2 Apple Silicon chips can power multiple simultaneous 4K displays from mobile laptops, and this same processing power is now being leveraged for next generation immersive imaging in a mobile wearable form factor with high density ultra high resolution near field displays. With high memory bandwidth AI chips and increasingly powerful Apple Silicon graphics, high performance GPU rendering and next generation AI computing are converging to accelerate the rise of Spatial Reality production workflows on Render.
NeRFs: Using AI for Photorealistic Spatial Rendering
The Render Network has already been contributing to a graphical change that has the potential to power fully interactive spatial renders, a convergence of 3D rendering and AI technology: NeRFs. Showcased earlier this year in a February Behind the Network, Neural Radiance Fields or NeRFs, leverage Artificial Intelligence computing and 3D rendering to create real-time immersive 3D assets that can be viewed on mixed reality devices. For those that may not be aware of NeRFs, Render Network user Corridor Digital’s Wren produced an exceptional video on the subject using the network.
To summarize, NeRFs are an evolution of classic photo scanning techniques that 3D creators have been using for years to create photorealistic 3D models of real-life objects and environments. While photo scanning has allowed for creators to generate lifelike imagery, it has required quite a bit of finesse on the part of individual creators to make up for areas where photo scanning has struggled to capture real-life, such as accurately recreating highly reflective or transparent surfaces, let alone doing so in dynamic scenarios with moving cameras and multiple angles. NeRFs take this same photo scan approach, but AI is then utilized to accurately generate a full 3D recreation of all elements of a scanned environment. What this has shown is not only have NeRFs far more successfully been able to recreate those same trouble objects and reflections/light interactions, in creating a full 3D recreation, a NeRF allows for quick remixing, as all angles and digitally introduced objects are factored into environmental factors such as light and angle in the NeRF.
To process NeRFs however, requires a significant amount of pre-computing power, combining 3D rendering and artificial intelligence deep learning. With an increasing number of artists and engineers looking to develop immersive spatial reality experiences, scaling a compute layer for wide adoption of NeRFs is an important part of the Render Network’s technology designs.
Light Fields and Fully Immersive Imaging
While NeRFs offer a tool for creating lightweight immersive assets, the highest quality immersive experiences are precomputed light fields, tracing all permutations of the bouncing of light within a light field volume. Through precomputed light fields it is possible to create noise free lifelike holograms that fully bridge between the synthetic digital world and physical space at the highest levels of fidelity. To make light fields available on the Vision Pro for a wide range of artists, engineers, and developers, however, requires exponential increases in GPU cloud computing power, as each precomputed light-field requires 10–100x the rendering of 4k images.
The Render Network is designed for a future where Light Fields are widely available to consumers, and has already showcased the promise of this next generation imaging with demos of cutting edge Light Field display technology in collaboration with Light Field Lab.
NeRFs, Light Fields, Holograms and the Future of Entertainment
How do NeRFs, light fields and holograms play into the larger MR landscape, including the Apple Vision Pro? All are building blocks in the development of the next generation of the post mobile information landscape, where fully interactive and immersive media experiences replace 2D screen based communication. The Apple Vision Pro can be seen as an important tipping point in expanding immersive media in the public consciousness and encouraging mass adoption via a stunning mass market consumer device. The future that the Render Network is powering is one where mixed reality computing becomes ever more ubiquitous and accessible, creating a new platform for creativity, art and immersive applications, bringing high end 3D compute off screens and into the physical world via spatial computing.
This is where the Render Network and its emerging technology for NeRFs and light fields stand to form core pillars of the future MR creative landscape, with full field of vision, ultra realistic imagery processed and streamed through a decentralized computing platform. With the power of the Render Network, rendering new forms of 3D content for MR experiences are drastically more time and cost efficient, allowing for larger scale and higher complexity rendering jobs to scale to a wider group of artists, developers and engineers and usher in the age of spatial computing. In the (not-so-distant) future, wearable technology form factors will improve, and real time experiences projected in a 3D, on real space, will not only be possible, but interactive, using the same technology and principles that are being laid today by the Render Network.
The Next Generation of Holographic Virtual Assets
While the Vision Pro opens up a new era in computing and informatics, it also holds the potential to revolutionize the virtual asset landscape that has developed around blockchain non-fungible tokens. The creation of immersive 3D assets will add a new dimension of realism to digital ownership, enabling consumers to purchase virtual objects that can blend into the physical world seamlessly like a virtual sculpture. Through networked mixed reality, these objects can become interactive, enabling collectors and creators to share virtual objects in physical space using geolocation and advanced mapping technology. What this opens up is a whole new way of interacting with virtual assets that brings them from static images and videos into fully interactive objects that can be experienced socially in the real world.
Holograms or fully lifelike digital objects may become at the forefront of the evolution of the virtual non-fungible token (NFT) economy, as collectors and artists look to develop new ways to synthesize digital art with physical ownership. The built environment — from iconic landmarks to storefronts — can become platforms from which to overlay digital content and mixed reality experiences. Clothing, accessories, and physical objects can be sold with unique non fungible digital twins that come to life in Mixed Reality using fiducial tracing technology like QR codes. Composable layers on top of spatial reality assets enable new forms of creativity and interconnection that cross between social networking and IRL experiences.
With devices like the Vision Pro, not only is it now possible to own virtual assets using blockchain technology — adding an ownership layer to the Internet — but these assets will be ever more lifelike, networked, and composable, creating game-changing new layers of immersion and interconnectivity for digital art and experiences.
Intelligent Assets and the Future of the Spatial Web
While media created for the Vision Pro needs to “come to life” visually with immersive graphics technologies and advanced rendering, it also requires new levels of semantic intelligence to become fully interactive and composable in mixed reality. The Render Network’s ORBX data schema enables artists to create, share, tokenize and monetize virtual objects that have full portability into the spatial web with the highest levels of visual fidelity and digital provenance using the network’s Render Graph. The integration of ORBX within open standards like the ITMF (Immersive Technology Media Format) enables — for the first time — the distribution of holographic 3D scenes with full 3D scene data and digital rights metadata for each compositional element. Because scene graph data is hashed and tokenized on Render, it losslessly contains and distributes on-chain 3D data for applications like real time streaming, and complex multi-party royalty flows. This opens up new forms of commerce and experiences, where real time 3D assets can be consumed, monetized and remixed in the spatial web with full on-chain composability. Already many Render Network artists are thinking about how to translate their 3D scenes into spatial reality objects — it is an application that the Render Network makes possible.
As has been seen with the rise of digital art sold on-chain, some of these early virtual assets may become historically significant cultural objects, as some of the first on-chain primitives in the development of the next generation-defining computing platform, the spatial web.
Future of Immersive Location Based Entertainment
Concluding the Vision Pro release was a remarkable display of the future of entertainment, with 3D worlds enabling consumers to experience entertainment in a much more immersive format than ever before. Showcased by Disney CEO Bob Iger was a window into the future of Disney spaces — beginning with The Mandalorian — that are presented in immersive spatial reality. As demoed in 2016 during the Disney Accelerator, it is possible to create fully immersive custom holographic environments for iconic media properties, enabling media companies to create stunning experiences that replicate and go beyond the theatrical experience. Rather than watching a film or experience on a TV, entertainment brands may create full IMAX-like immersive experiences to be sold to audiences in mixed reality, similar to how games are streamed and sold online. The Vision Pro may create the first mass consumer adoption of immersive VR experiences through its stunning display and rendering technology — again opening up new channels for creators to produce and monetize ultra high resolution immersive experiences that are only possible through exponential increases in computing power.
Location based entertainment may further evolve to include immersive real-time 3D worlds that combine 3D storytelling with generative AI on the fly composition in mixed reality. Streaming experiences may become augmented by non-fungible tokens, where users get digital souvenirs for consuming an immersive experience, or even can create unique digital assets from within an immersive spatial world. Hands free AI-assisted prompt-based mixed reality authoring may make interactive on-chain digital authoring, collecting, and networking ever more frictionless and enjoyable bridging the worlds of art collecting, gaming, and social networking.
The Roddenberry Archive: a Window into Next Generation Media
Finally, the Render Network has powered a prototype that provides a window into the future of spatial reality media and entertainment. Throughout 2022 and into 2023 the Render Network has been instrumental in bringing next generation 1:1 life sized immersive Star Trek Enterprise and Enterprise Bridge experiences to the public using its computing and streaming ecosystem — showcasing in vivid detail the potential for fully immersive mixed reality experiences.
Both decentralized intensive GPU streaming and GPU cloud pre-computing are needed, along with native immersive file formats, to produce and distribute high end immersive experiences like the Roddenberry Archive. As the Vision Pro comes to consumers at increasing scale over the next decade, the same 1:1 experiences showcased in the Roddenberry Archive can be the template broader creative works and consumer experiences. They will also be augmented by ever more immersive viewing options like full light fields, which require orders of magnitude more computing power than 2D or 3D rendering.
Projects like the Roddenberry Archive are possible with the decentralized rendering and streaming capabilities of the Render Network, which can power stunning widely distributed holographic spatial computing experiences. It is an exciting future, one that promises to be augmented and accelerated by the Render Network’s decentralized GPU computing network, 3D marketplace, and application platform.
Join us in the Rendering Revolution at:
Website: https://render.x.io
Twitter: https://twitter.com/rendernetwork
Knowledge Base: https://know.rendernetwork.com/
Discord: https://discord.gg/rendernetwork
Render Network Foundation: https://renderfoundation.com/
