Mixed Reality for Tourism
Challenges in delivering mixed-reality experiences to tourists
Delivering mixed reality experiences to tourists is challenging; whether it’s augmented reality, virtual reality or a mix of the two. Because there are a wide range of ever changing elements in the field, there is very little margin for error. At Yaturu our goal is deliver the best experience possible to our customers and accordingly, we are constantly working to push the boundaries of the available technical resources. The best example of our efforts in this endeavor are the experiences we craft in augmented reality (AR).
We believe that AR, is a perfect fit for tourism because it allows travelers to view location while simultaneously having additional digital layers seamlessly superimposed onto their line of sight. However, developing and delivering a compelling AR platform for tourists requires a solution which is:
- Immersive
- Reliable
- Able to accommodates a wide FOV (field of view)
- Able to transition to a virtual reality expereince
- Lightweight and durable
- Able to be used outdoors in sunlight
- Affordable
- Comfortable to use for a few minutes at a time
Passthrough AR
These features are not readily available with today’s AR options, so we’ve had to improvise. Our current solution is to use passthrough AR in a VR headset. The headsets we’ve chosen to use have a special cut-out window that allows the phone’s camera to see the world, thereby making it possible for the viewer to enjoy real-time vision, albeit through the camera lens. Then, we simply pass the camera feed through our custom-designed digital processor which divides the image into two (one for each eye) while magnifying the image in a way that offers a clear and singular vision through lenses of the googles.
This solution, was convenient. After all, tourists using our platform are already using smartphones to play the pure audio and video content which we provide. Now, AR experiences can be run through the same device.
As for the particular headset we chose to use: The Merge AR/VR headset was a perfect fit. Firstly, it protects our devices, preventing them from being scratched or from falling out onto a hard surface.
But, that’s not say it doesn’t pose risks of its own.
We found that when face-down and exposed to sunlight, the goggle’s lenses magnify incoming light-rays, which produces enough heat to irreversibly burn a phone’s screen. We solved this issue, however, with a pretty quick fix. We designed a simple, black foam cover that uses velcro to attach to the googles, completely shielding the lenses.
Also, when it’s not being used, the headset can be a bit uncomfortable to carry. For tourists, who often find themselves on longer tours or hikes, that can be a real hassle. We overcame this issue by developing a small velcro strap that attaches in seconds to either end of the headset, turning it into a small, light tote.
AR Tracking
So, we’ve come a long way. But, like everyone in the tech industry, we’ve still got a ways to go.
One particular issue we hit was the problem of AR tracking. Unlike Virtual Reality, which gives you full control over all elements of the picture, AR requires that you mix in digital elements into a real, natural image — one over which you have no control. That means, if the person is standing on a slope looking down, the digital image of, say Alexander the Great (whom we’ve actually made), must be constantly updated to match where the user is looking and located, so the character will stay put and not move with the motion of the user.
It’s a serious issue in the AR world, one which tech giants like Apple and Google have and continue to invested considerably in.
Originally, we utilized ‘markers’ to gauge the place of AR elements. This meant that each tourist would have to first look at a special image that was pre-programmed to spur a certain AR experience. The marker itself tells the program where the ground is, which anchors the digital elements and helps them match up to the tourists’ angle.
We matched the marker-method to our other requirements — namely, that for we needed the marker to be light, small, and simple enough for a tour guide to transport and setup on the fly. Mindful of these constraints, we designed a marker and special stand, which was strong enough to work even in windy conditions.
Markers, however, are difficult to use. The tourist must hobble over, goggles-on, toward the marker, and then bend down to stare at it. As you can in the GIF below, I often had to hold the tourists’ hands as they walked to and from the marker.
So, while it worked quite well, it had to be replaced by a more convenient and time-efficient solution. That would allow us to get the group ready for the experience much faster.
We initially started to experiment with marker-less AR tracking using certain features in the environment to serve markers, but which were more subtle and easily recognizable to the system. We experimented both with openCV and also Vuforia — using multiple photos of the background to trigger the AR experience and assist in its accurate placement. In the end, we got the Vuforia-method to work reasonably well at one location. However the technique had issues in reliability and also did not scale well, to cover lots of sites easily. Also on tour, changes happen in an instant, so this approach did not allow for too much flexibility.
We have recently built a new technique which will allow us to both get rid of the marker and provide a robust and reliable method to continue to give tourist the ability to enjoy AR on tour. Stay tuned for more about this later!
follow up article — published here