The new augmented reality (AR) navigation from Google Maps rolled out recently but it isn’t available to everyone.
For those who don’t know what AR navigation is, here is a link from Google IO 2017.
In a nutshell, it is directions overlaid on the camera view. Instead of having to look down at the phone, you can bring the phone up to see the world through it. Google will overlay the directions in your view with 3D icons, labels, and texts.
I saw some users online with access to it but didn’t find much information on how to get it. Posted a call on FB and LinkedIn in case my friends had it. I thought maybe they could invite me through the app. Reached out to a few friends who work at Google. They hadn’t heard anything about it either. It’s a big company!
My colleague found out that I needed to be a Google local guide level 5 and above to be randomly selected to receive the app.
A Google local guide? I asked.
If you are a user of Google maps and have rated a place because Google asked you to, then you are already a local guide.
To check, open Google Maps. Tap on the hamburger menu on the upper left corner of the search bar. This will slide out your details. Select “Your contributions”. You should see be able to check your level.
I was at level 3 which meant that I was not eligible to get the AR navigation. It is closely tied to my current work in AR, and an app from Google is a great reference for what’s coming and how far they have pushed the boundaries with this technology.
Google engineers often come up with fun and creative ways to solve common problems in the tech space, and I was curious about how the app would run on my phone. The battery drain, the heat generation, the design, responsiveness, the user experience, all of it.
In addition, within a few months, this app will probably be live. A lot of people use Maps. The way that Google designs its user experience has the potential to set the bar for this kind of geolocation-based apps. Albeit a lot will change. That’s the whole point of limited user tests. But to me, this is a tiny glimpse into the future. I wanted it.
I researched local guides and found out how to ladder up. Each rating is a point, each review is 10 points. If you write more than 200 words then you get another 10 bonus points, and if you post a picture, you get an extra 5 points which can all add up to 26 points per review.
I started writing reviews for every place I visited. I’m an avid photographer and had many pictures of places that I have visited in my album. Posted pictures for extra points. Each picture gets you 5 points.
By the end of March, I was at level 6, which is about 2000 points.
To get to level 10, I needed 100,000 points. The number of reviews to get there anytime soon just seemed impossible —visiting and writing 4000 reviews to be precise, and even if I did that didn’t guarantee an invite from Google to test this AR navigation app.
I didn’t give up, but the goal changed. At this point, the goal was not to get to level 10 anymore because that just seemed ridiculous but I didn’t give up because I found that I had started enjoying writing again. A few years back I used to write every Monday from 7–8 p.m and enjoyed it. I had lost that habit. Writing reviews gave me an opportunity to write every day, and I didn’t mind to just keep writing especially since Google kept reminding me that it is helping people. It didn’t matter if I was going to get to the next level or not.
To know more about local guides here’s a link:
One evening, while I was still at Level 6, I opened Google Maps to get directions and this popped up. I got the AR Navigation app!
Since it all started with writing reviews, here’s a review on how this app performs.
These are the areas that I took notes on.
- Night mode
- Battery Drain / Heat
- User Experience
It only works if you select you’re going to walk to your destination. The extra AR button that appears for walking navigation at the bottom doesn’t appear for driving directions.
It is the ability of the device to understand its position based on what it is looking at. Every morning, patterns of light on the ceiling, pictures on the wall, the straight lines where they need to be, and million other visual cues help us to know where we are. From our perspective, it is also clear where are in the room, and in what pose.
Our brains have localized our position and orientation in a fraction of a second. If anyone of these cues goes missing, we take a little more time. We look around to confirm, gather more information, connect more dots, validate and settle into a decision. We do something similar outdoors. A street sign or familiar buildings tell us where we are in the world and the direction we are facing.
Google Maps AR Navigation primarily does outdoor localization. The Indoor localization that you see in malls is mostly made possible by a company called IndoorAtlas, and we can talk about that another time.
Outdoors, Google Maps AR Navigation looks at the street signs, buildings, and other familiar features, and tries to search through its brain for a match. The brain is located on servers in the cloud on Google Cloud Platform. The memory of the brain (database of pictures) comes from two different sources -
- The Street View cars.
- Pictures that you and I upload to Google photos to store in the cloud.
Most of us have spotted plenty of Google cars driving by that are loaded up with cameras on the roof. Those cars take pictures of the surroundings that gets tagged at Google offices by computer vision and machine learning algorithms and gets uploaded to the servers for storage.
The second source of data is the millions of pictures that you and I upload on a daily basis. Searching for things online using Google generates a lot of business for Google (through ads). We also generate billions in revenues for Google by just uploading our pictures to Google drive. If you are a small business trying to use Google Maps service, you have faced the increasing cost and licensing fees last year. Google Maps is projected to be a 5 billion dollar business by 2020. This is why storing photos on Google is free if you have a Pixel phone or free up to 15GB for other phones.
The features that help an AR app to localize quickly are the ones that are distinct. Trees and bushes are too generic to be useful. We can’t look at only a tree to figure out our location, computers can’t either…not yet.
The UI says point your cameras at buildings. The UI also says its an Alpha version of the software. I pointed the camera at the buildings in front of me and moved it around from left to right in a rhythmic pattern like a fan at a rock concert waving candles at the stage. It didn’t work!
This could mean that there just aren’t enough pictures of this building on Google servers for the AR Navigation app to get a good read from this angle. The option was to go back to using the old maps or try again. I tried from a different angle to help it along.
This time it brought up 3d direction arrows with a little bubble marker that bounced around horizontally on the screen to point me to look in the direction that it wanted me to look at. This is what I was talking about earlier- the creative ways that Google solves some of the common UX problems. I thought this was a good way to orient the user towards 3D assets in AR that the user might otherwise miss. It's a common issue that all AR developers have to think about today.
This location is in the middle of tall buildings, which is not good for GPS. Tall buildings can obstruct GPS signal or worse bounce it so that the satellite appears to be at a different place from the phone’s point of view. In both cases, the results are inaccurate. Also, I was surrounded by trees which are hard to track because it's hard to extract feature points from trees. Given the challenges, I thought the app did pretty good with localization. But it doesn’t do so well ‘Tracking’ this particular surrounding.
I could be wrong but this seems to be using feature-based tracking. It displays dots while localizing. The dots look like feature points. Feature-based tracking tries to extract feature points from the point cloud data. These feature points are obtained using many different algorithms like edge detection, threshold detection, blob detection, etc. The system then tracks these feature points across a sequence of images to form an optical flow. The optical flow helps the system to track an object in the AR world.
The app was tracking fine until the ‘keep-your-phone-down’ message popped up. In an effort to save people from bumping into walls or walking onto oncoming traffic and getting hit by cars, Google decided to limit the time a user can spend holding the phone up in the AR mode. It dims the screen to signal timeout and then pops up a message asking you to put the phone down. Bad idea; as soon as I put the phone down it lost track. When I pulled it back up, it tried to relocalize all over again and failed.
As I said this was a challenging environment. Maybe this feature would work better in the middle of a city.
I took it out to North Hollywood by the Television Academy where there are plenty of bars, restaurants, businesses, and movie theaters. It relocalized a little quicker but the lag is enough that if you have to keep putting your phone down and relocalize every time, it makes me not want to reach out for this app. I have no doubt this will improve. This version is just Alpha which means it’s feature incomplete.
4. Night Mode
It was more of a hit and miss during the night. It is too much to expect from the app to extract feature points to track in the darkness so it wasn’t a surprise that it didn’t work that well. However, in well-lit areas, it worked just as well as it did during the day. The UI theme really pops against the dark background.
5. Battery Drain
It drained my battery at a rate of 1% per minute. On average, I’d lose about 5% every 5 minutes. I don’t think it means you can go for an hour and a half on a full charge. The phone does get pretty warm too.
6. User Experience
The Anchoring of the 3D direction arrows and the labels is impressive. Once the markers appeared in the world, I didn’t see any drift. You can walk up to them and around them. Also, the material design is easy on the eyes.
Even though the app is nice to look at and interact with, I felt that between the lag in localization and the notification to put the phone down along with Android dimming its screen due to inactivity, it was a hassle to use it. There were times when I’d show it to a friend but of course it wouldn’t work indoors so I’d try to point it at businesses and buildings outside the window. Most of the time that wouldn’t work. It did find a Starbucks across the street as we were standing in the lobby of another building. That way the app has plenty of surprises. You never know what it’ll pick up on.
Holding the phone up did tire my arms more than I’d have thought. That is something that every AR app will have to deal with. It was just not a very comfortable experience using my phone like that for an extended period of time. Also, since nobody reads their texts holding the phone up like that, some people tend to think that you’re recording them. They dart out of the way. Until all of us are walking around holding out phones at chest/face level it is going to draw some attention and can make people uncomfortable. I did tend to lose sight of my surrounding. My world was tunneled into what I could see through the phone.
Why is this app meaningful?
Right now it is a novelty. Fun to show it around. This is an alpha version. Things will improve by the time it is released. I can see how this app could be useful if the buildings around me start to get labeled in AR and the billboards start to get replaced with on-demand streaming ads and animated movie posters based on my interests. But practically, as a walking naigation aid, this app is probably not very useful. It doesn’t give me any new information that I wouldn’t get from the 2D map. We have the compass on the 2d map and voice guidance. Not sure if I’d go into AR and hold the phone up just to see the directions in 3D.
I’m not sure if Maps is the best use case of AR on the phone. I think this would make more sense on a smart glass. But, I’m thankful that I got a chance to try out an early release and more importantly it got me back in touch with writing.