Varjo, with XR-3, has made the first fusion algorithm on top of lidar with video pass through.
Varjo’s fusion algorithm is combining volumetric capture (lidar) with video, with Nvidia’s deep learning optical flow algorithm, to capture the world in real time (90fps) allowing holoportation (like teleportation through someone else’s eyes) across the planet.
Story by Micah Blumberg (with help from Cecile Tamura), Silicon Valley Global News.
Varjo is combining lidar and video to do high resolution depth capture that goes beyond what can be done with lidar alone. This process allows really stable pixel perfect depth capture at 90 fps, for things like real time holoportation to take you to other places on the planet in VR or XR. When combined with 3D Semantic Segmentation the possibilities are mind-expanding.
Varjo uses the full field of view lidar on their new XR-3 headset (with the same fov for the lidar as the video) 108 degrees wide lidar feed of the world around you, it’s used as a ground truth with the VSD (video) cameras which are run through Nvidia’s deep neural network optical flow algorithm that create a depthmap for the video see through feed, for every single pixel captured by the VSD cameras.
Then there are three layers of filtering that bias the data based on the time of flight (lidar) data, the resulting depth image (seen on the right of the above image has incredible high resolution depth) with highly detailed, really stable, pixel perfect, depth contours in every single frame at 90 frames per second, it captures depth details like hair, this allows incredible beautiful depth gradients, and allows camera reprojections, 3D reconstructions, really high quality holoportation of the things in front of the headset, in real time, to other places on the planet.
See the full interview with Urho Kontorri, the co-founder and Chief Innovation Officer of Varjo, about the details of new XR-3 & VR-3 launching this month.
Silicon Valley Global News (with Cecilia Tamura and Micah Blumberg) had the chance to have a very long in depth interview with Urho Konttori, the co-founder and Chief Innovation Officer of Varjo, about the details of new XR-3 & VR-3 launching this month.
I’ve personally tried the Varjo headsets on 3 separate occasions before the Pandemic began. This is the highest quality possible virtual reality and or mixed reality headset that money can buy. I’ve also tried the Magic Leap and the Hololens 2 headset on multiple occasions but so far no Mixed Reality headset compares to the Varjo XR-2 or XR-3.
I say with confidence that the XR-3 really is the very best Mixed Reality headset that professionals can buy, but it’s also the VR headset of choice for big companies like Kia Motors in the age of the pandemic for remote design work with Autodesk software.
Varjo is also the latest VR XR headset maker to integrate Ultraleap’s hand-tracking technology (Ultraleap formerly called Leap Motion). A purchase includes licenses for the latest generation 5 algorithm (that was at the time of the interview still unavailable to the general public, but Varjo includes those Ultraleap licenses with purchase of the XR-3 or VR-3).
The Varjo XR-3 and VR-3 are each a major technological advance forward.
A critical new feature is that now you can have high resolution everywhere you look in the headset thanks to a higher resolution peripheral screen with foveated rendering technology.
Professionals can have the confidence now that what they are seeing in the color space is exactly the same across all Varjo headsets with Professional Grade Color Space (99% sRGB and 93% DCI-p3)
The Varjo screen is wider than any of the mainstream consumer VR headsets including the Valve Index and the Reverb G2 HP.
Notable improvements to the Varjo XR-3 & VR-3 include a 40% wider FOV than their previous generation for the sweet spot, and the center of focus was moved downward based on feedback from professional customers.
However as previously noted there is no longer a significant difference where you are looking depending on whether it’s the center display or the peripheral display because of the integration of foveated rendering.
Kia Motors
Kia motors, a Varjo customer, has changed their business around the Varjo headset, in part because of the pandemic, because of flight restrictions. So they meet once a week or more in the Varjo headset, to review Kia car designs in perfect clarity.
As a result of using the Varjo XR headsets Kia motor’s design flows are faster and their team coordination has improved.
Thanks for reading.
Cecile Tamura and I admin several great groups on Facebook that you can join or comment on this article in.
