Learn all about AI in VR — Plus a VR Headset Comparison between the new Apple Vision Pro, Bigscreen Beyond and the Meta Quest 3!
Join the Mindplex Podcast team for a live-stream conversation with Ryan Sternlicht on Thursday, February 15th at 11 am PST/7 pm UTC.
Welcome back to the Mindplex Podcast and our new weekly live format!
Join us this Thursday, February 15th at 11am PST/7pm UTC with special guest VR expert Ryan Sternlicht. This episode will provide an insightful look into the world of AI in VR and then go on to compare the important features of the best headsets at the three different price points.
After a brief overview of AI in VR with Ryan, he and co-host Lisa Rein will report back from their recent Apple Vision Pro Demos. Lastly, Ryan will compare the Apple Vision Pro to the BigScreen Beyond and Meta Quest 3 headsets, to help you decide easily which of the three to purchase, if you are looking to pick out a new VR headset.
Desdemona Robot will also be on hand and we are ready to test out some new features of her dialog system that we are testing out. She welcomes your simple or complex questions about anything! Bring it on ^_^
Live Mindplex Podcast with Ryan Sternlicht
Thursday, February 15th at 11am PST / 7pm UTC
Episode 22: AI in VR
AI in VR
In our “AI in VR” discussion, we will give you a brief overview of some of the good and at least very decent AI implementations that currently exist within games and VR worlds. Then we’ll hypothesize a bit about one thing everyone is in agreement about: They could all be a lot better.
Embodied AI in VR Chat
One issue we will be discussing in the show is the role of “embodied AI” — which, in all fairness, we discuss in a lot of shows, because it’s such an important part of developing AGI that is empathetic towards humans that will seek to understand them, rather than just be annoyed by our presence.
Ryan Sternlicht will be going into a bit of detail regarding the embodied AI that has been or is currently in VR chat. Even the past implementations are useful to learn from and are useful to take into account when developing new implementations.
Note that cute and cuddly “Embodied AIs” tend to be more readily accepted and engaged with by users.
There have been a number of advancements connecting virtual reality (VR) and embodied AI. Popular VR YouTubers and V tubers, such as Filian’s VR streams and Neuro-Sama, who are well-known for their interactive AI, are created using facial tracking, VR motion capture, and embodied AI technology. These Vtubers have really helped bridge the uncanny valley and make humanoid avatars more widely accepted.
There are several AI avatars in VR chat, including: NotaAI, Joken, Celeste, and TIO. Other examples of VR in AI include mods for Skyrim VR and the game Event Zero. Which, although not in VR and over 7 years old, is still considered by many one of the best AI in games. Both Skyrim VR and Event Zero showcase pretty decent AI capabilities.
Overall, AI in VR chat presents exciting opportunities for enhanced immersion and natural interactions.
“Most AI personas in VR chat are set up in a similar way for interaction,” Ryan explains. “They use a speech to text model to convert user input into text, which is then input into an online AI model like GPT-3 or Watson to generate a response.
The response is then converted back into speech and sent back into VR chat. Some more complex models even allow for physical interaction, where users can ask the AI to pick up objects.”
AI in VR chat is valued for its ability to create immersive experiences and lifelike interactions, which traditional NPCs (Non-Player Characters) in video games often fall short of.
There is still one major obstacle making it difficult to reach the level of immersion required for a good VR experience: that most NPCs that you interact with are not very advanced, which quickly breaks the immersive effect. This problem is something that good AI integration will definitely be able to help solve. However, to date, only a few games in the past decade, namely Event 0 and Skyrim (with AI Mods), have even attempted true AI interaction.
But what kinds of “true” AI interaction are actually possible and how do you integrate them in a way that is seamless and blends naturally with the rest of the fabric of the game or experience? Let us know your thoughts and join in on the discussion.
Apple Vision Pro Features
Good VR is immersive, and Apple focuses on immersion with this device.
“Apple has truly pushed the boundaries of facial capture technology and visual fidelity in this groundbreaking new headset,” Ryan explains.
“It’s evident that they have invested significant resources into creating a superior user experience, and it will be fascinating to see how this technology evolves in the future.”
Avatar System
There is a sophisticated “Persona” avatar system with a 3D facial system that can be used for video calling and playing iPad and iPhone multiplayer games. If you’re saying something like “what a waste” to yourself right now, you’re not the only one. Apple made the very deliberate decision to make the headset available at this very expensive price — which Apple isn’t making much, if any, money on — so there would at least be a headset available to use for development.
Ryan noticed that there are still some minor “uncanny valley” issues, but Apple has basically found different ways of adjusting the software so the user’s disorientation is minimal.
Both the avatar system above and the facial tracking system noted below are using mechanisms similar to most v-tubing systems or facial motion capture systems and Ryan sees the potential for integration there.
Facial Tracking
Ryan had some fascinating insights about facial capture technology in the new Apple headset. The avatar system built into the headset is incredibly advanced, offering superior facial capture and movement tracking capabilities. It can accurately replicate facial movements, such as mouth, cheek, and eyebrow movements, in real-time, surpassing most other headsets on the market, and even many vtubing / mid-level motion capture systems. He would even compare the facial capture of the Apple headset to some of the motion capture devices used in Hollywood.
What’s particularly interesting is that this data can potentially be applied to other types of avatars, allowing users to have more customized experiences in virtual reality platforms like VR Chat. However, it’s crucial to address concerns about privacy and data security, as the accuracy of the facial tracking in this headset is so high that it could potentially be used for fraudulent activities. So, it could be great for when you are making your own digital twin, but “bad” if someone else were able to get a hold of that data and make a deep fake of you.
Realistic Passthrough Cameras
The Vision Pro’s passthrough cameras are very impressive, and Ryan noted that Apple has made significant progress in eliminating the visual distortions experienced in previous pass-through cameras. By aligning the cameras with the user’s eye locations and implementing advanced camera technology, interactions with the real world through the headset are nearly seamless and feel very natural.
“The pass-through experience with the Apple headset is far superior to any other device currently on the market. This level of visual accuracy and realism opens up exciting possibilities for immersive experiences and interactions with virtual objects in the real world.
The improved visuals and reduced distortions in the pass-through imagery contribute to a more comfortable and enjoyable user experience. In comparison to other headsets, where objects may appear warped or distorted, the Apple headset provides a true-to-life representation, allowing users to seamlessly interact with their surroundings.”
Hand Gestures Are Here
One aspect of the overall VR experience that is changing rapidly is the progression from hand controllers to just using natural hand gestures. Gestures are now a key aspect of VR interaction, allowing users to navigate and interact with virtual content with a wave of their hands.
While using gestures can be intuitive and efficient, there are some limitations, such as difficulty in typing and performing complex commands. For the most part though, the downward-facing cameras in this new headset enable users to easily control the device without having to hold their hands up for extended periods of time.
Only a few hand gestures are taught and utilized in the Apple Vision Pro demo, but only a few were needed to do a lot of things. Mainly, Lisa practiced focusing on an object and selecting it, zooming in and scrolling back and forth. She forgot about not having any hand controllers immediately, as opposed to past experiences when she kept reaching for them.
Eye Tracking
Eye tracking in the Apple Vision Pro allows the device to know where the user is looking on the screen, making interactions more seamless.
During the show, Ryan will discuss the potential for using external controllers with the Apple VR headset and the evolving landscape of VR gaming on Mac computers. Overall, Ryan is excited about the possibilities of VR interaction and the potential for further development in this space.
Ryan will also cover the release of the Oculus Quest 3, highlighting its improved resolution, tracking capabilities, and the ability to connect it to a PC. The Quest 3 is stillthe most consumer-friendly and affordable VR headset option currently available. There are rumors of a “Quest 3 Lite” in the works that might be even more consumer-friendly and even less expensive, but those are just rumors for now.
All and all the Apple Vision Pro’s quality is superior to all other headsets — and it will be really nice playing games and having experiences with it, once they are available to play on it.
Headset Roundup
Ryan will be giving a thorough headset roundup on the show, where he will be reviewing the three of the latest models to compare them in more detail.
The three VR headsets Ryan currently recommends are the Meta Quest 3, Bigscreen Beyond, and Apple Vision Pro. He specifically chose these three headsets to compare because he feels that they are the least compromised of any current headsets.
The Meta Quest 3 is still the best, if not only, the headset for $499 , and also has access to the massive Meta App and Game Library. The Quest 3 now also has decent and much-improved passthrough cameras for when moving around in your room. The Bigscreen Beyond headset is probably the best headset currently available — but the price will keep some people from using it. ($999).
There is also a SteamVR tracked headset as it is extremely small, light, is customized to your face, and has high-resolution OLED panels, all for a decent price if you are upgrading from a previous SteamVR headset, or a lower price compared to most other similar level SteamVR headsets.
The Apple Vision Pro is by far the best top-end headset in terms of quality. The level of polish to every aspect of this device is evident.
Right now an Apple Vision Pro will cost you $3,500-$5,000 — depending on where you get it. Unless you have specific reasons for wanting to purchase an Apple Vision Pro, buying one now would be basically buying an expensive toy. If you are a VR developer though, it’s probably a very worthy investment.
If you are looking for a headset for day-to-day use that has all the apps and games all ready to go, or if you are a developer, but it’s important that you launch your game on a platform with an existing user base for business reasons — you are still probably best with the Meta Quest 3.
So why exactly might you want to spend the additional $500 on the Bigscreen Beyond headset? That’s what we’ll be discussing on the show!
Episode 22’s Featured Guest: Ryan Sternlicht
About Ryan Sternlicht
Ryan Sternlicht is a San Francisco-raised educator, researcher, advisor, and maker, who has advised a number of startups in the fields of AR, VR, AI, neurotech, and 3D manufacturing.
As a maker, he has volunteered or advised several makerspaces and hackerspaces, as well as numerous technical non-profit organizations in California and upstate New York.
While in the Bay Area, he volunteers and teaches at Noisebridge hackerspace, and staffs a large number of film festivals and conventions, such as Aaron Swartz Day, Offkai Expo, Maker Faire, and Open Sauce. He also works at Alamo Drafthouse New Mission, and helps teach game development at CCSF.
About Mindplex
The Mindplex Podcast team invites you to join them as they contemplate and learn about this fascinating world. They look forward to meeting you so you can help each other understand the true nature of all the challenges involved as we work together to create a Benevolent Singularity.
The Mindplex Podcast and Mindplex Magazine — are both part of the Mindplex decentralized media platform spin-off powered by SingularityNet
We invite you to check out the Mindplex website, set up a profile, and perhaps even contribute to the magazine.
Follow us:
Visit Mindplex Magazine
Join our Telegram
Join our Mindplex Social
Join our Discord
Follow us on Twitter, LinkedIn
Like us on Facebook
