Augmented Reality for Mobile VR Headsets
(Developer Post)
In a previous post we discussed the challenges of delivering Augmented Reality (AR) experiences for tourists in the field for our Yaturu tours. Our ideal implementation was to use ARcore within a google cardboard Virtual Reality (VR) headset. This solution would enable us to provide marker-less AR in the field delivered in a VR headset.
However, an off the shelf solution was not available so we had to make our own. We have just released this project to open source and you can find some of the more technical details here. In this article I’ll describe our journey of how we were able to solve this problem and some of the bumps we encountered along the way.
ARcore + Google Cardboard — Unity SDK
Obviously our first pass at solving the problem was to investigate the space and see what others have done.
We found that using ARcore in a purely VR environment and for inside out tracking it seemed to work fairly well.
However the tracking and placing of objects did not have good stability and seemed to drift.
We also stumbled upon a Google ARcore dev issue about this exact topic. So we set up a basic environment to do this basically integrating ARcore and Google Cardboard and reproduced the issues found in other previous attempts. The video below was probably our best shot with this approach, but you can still see the annoying loss of tracking as the camera is rotated.
We tried hard by trying to negate the drift and lack of stability by trying to tweak the rotations but ….
So we went back to first principles to try figure out what was going on. We asked ourselves how hard could it be toreproduce the hand held version of ARcore by just splitting the screen for each eye in a headset??
Google Cardboard and VR distortion
After doing some digging we came to the realization that the reason the tracking was not working in the headset was because of the distortion of the image caused by the VR goggles lens’. To understand this, you need to first understand the basics of VR and why large magnifying lenses are needed in the VR headset.
VR requires a large field of view, this gives the user the sense of immersion within the experiences. The human eye has an extremely large field of view (see image on the left). For mid-peripheral vision the field of view of our eyes is around 120 degrees. The Oculus Rift and HTC Vive approach this with fields of view of 110 degrees.
The lenses of the VR headset make this possible by highly magnifying the image close to your face and widening the FOV.
However this comes at a cost of distorting the image as it passes through the lens. The areas of the image further away from the center of the screen will appear highly distorted. In order to combat this VR systems use some cool optics calculations to distort the initial image so that when the images goes through the lens it will appear normal for the viewer. This distortion is known as barrel distortion.
In the above example the straight wall on the left appears curved (when not using VR glasses). When the image passes through the lenses to the viewer’s eyes, it will appear like a straight wall.
As ARcore was designed to be used on a handset alone, there is no concept of this distortion, so when applying Google Cardboard there is a conflict in the tracking and how virtual objects are projected onto the rendered image. Trying to fix this problem by constantly updating the two conflicting cameras, this causes the violent head banging in the GIF above.
Our Own Custom Google Cardboard
We then proceeded to remove the Google Cardboard SDK and make our own version. We ended up with the following pipeline:
Reality camera (phone video feed) → AR objects camera (Unity) → Barrel distortion → Display
The reality camera is set up to track AR feature points and the device pose using Unity’s AR Foundation abstraction layer. We chose AR Foundation because it represents a very good abstraction layer on top of ARCore and ARKit, and it is also an official package from Unity Technologies. At the time of writing, it’s still in “preview” stage but it has proven stable so far.
The video feed coming from the phone’s camera is rendered using a modified version of Unity’s ARBackgroundRenderer which comes with AR Foundation. At the time it had a couple of bugs when working with custom materials, so we had to “fork” it and fix them. (The bugs have since been fixed in AR Foundation.)
The background renderer simply writes whatever the phone’s camera sees onto a fullscreen quad. In order to make subsequent steps work, the shader’s “Z write” setting has to be turned off — this ensures that the video feed does not write to the Z buffer because we want all AR objects from the scene to be rendered on top of it.
After rendering a fullscreen quad with the video feed on it and getting the camera positioned correctly in the AR world, we proceed to render the AR objects from the scene. We’ve set up a separate camera for this stage, which is set to be the child of our reality camera. This way, it picks up positioning and orientation changes automatically. Since its render depth has been set to a greater value than that of the reality camera, the AR camera renders everything on top of the previous camera’s output.
In the next stage, we’re applying qian256’s barrel distortion shader on top of the current render, which at this point contains both the video camera image and the AR objects in the scene. We had to do some trial-and-error work on getting the FOV parameter exactly right for the devices we were using, otherwise there were disorientation and nausea problems when looking at this through a headset. Once they were configured correctly, we were all set.
In our solution we use simple fragment shader to implement the cardboard. We also had to find the perfect setting for our phones camera which as a horizontal FOV of around 60 degrees. (This means for each phone we might have to tweak this parameter a little)
With that being said, we should note that google cardboard’s solution for barrel distortion is more efficient and provides a better technical performance, for a great article on the topic see Brian Keher’s post for more info.
Our solution is suitable for our use cases but hopefully Google will release their own plug and play version which will work with ARCore. But for the meantime feel free to check out our first open source project.