Augmented Reality for the mobile web
A summary of the state of AR and an outline of a recent project I built.
I built a game for my kids during summer break; this is a summary of what I built and the tools I used.
First, Here’s a video recap I made of the game, including gameplay footage:
The state of AR:
Augmented Reality has been all over the news recently since Apple announced their new headset a few weeks ago.
Headsets
Apple’s vision is similar to what other companies like HTC, Oculus (Meta) etc have been working on for a decade or more.
Each of these requires a headset; thus far headset usage appears to drop off drastically after the novelty wanes.
Apple et al are making bold bets on what the future looks like, but nothing so far indicates that their predictions are correct; these companies are big enough that they can often create the future they want…it’ll be interesting to see where headsets are in a decade.
Mobile Devices
An area that seems more realistic is mobile devices. Niantic released Pokemon Go 7 years ago and it is a more probable version of what future AR will look like. 90% of the world has a mobile device, and these mobile experiences are available right now on most devices.
Designing AR experiences doesn’t require a headset or fancy equipment, and plenty of existing examples of great mobile AR exist today.
Mobile Web
Further, you don’t even need to publish an app in the app store; AR experiences are available in the browser and have been for a while.
I first started experimenting with Web AR 6+ years ago — back then there were few examples and the hardware wasn’t totally ready.
Today there are tons of tools and the available hardware already exists on many devices.
The Project:
Ideation
Niantic recently bought 8th Wall, combining their massive data set with 8th Wall’s technology. I wanted to experiment with their newer tools and was looking to build something that effectively showed the power and availability of AR on the web.
My kids were just starting summer break so a game that could be played outdoors made sense.
8th Wall uses Niantic’s database of public “wayspots”, the same ones used with Pokemon Go. This means wayspots on private land are not allowed. Any wayspots you publish need to meet certain criteria. However I discovered that private wayspots could be used in development mode.
Knowing this, I knew my game couldn’t be published but I also knew I could use private wayspots in dev mode; this played into my decision on what to actually build.
Setup
Niantic‘s tools are pretty straightforward. I was able to scan areas in my yard and upload these to my project that was registered on the 8th Wall platform.
Scanning produced a rough estimation of the locations:
It appears that 8th Wall uses a combination of gps location + 6dof + computer vision to identify the location ( I‘m just guessing, I don’t actually know how they do it. )
Once the locations were scanned and uploaded I created 3D models for ‘Prizes’ when the locations were found.
Code & UI
Niantic provides objects for their API to handle different events.
XR8.Threejs.xrScene() // object to handle the AR elements, they have a threejs version. Note that it does appear that 8th Wall favors A-frame over threejs et al.
//events
{event: 'reality.projectwayspotfound', process: wayspotFound},
{event: 'reality.projectwayspotlost', process: wayspotLost},Once I familiarized myself with their structure I was able to easily create the application in Javascript using threejs.
UI was created with SVG using Illustrator/Figma for basic layout and then polished in CSS.
Gameplay
My daughters played it, each played separately. My older daughter had seen me in the yard testing so she knew where to stand for some of the locations.
With zero idea of where to look I realized it was pretty challenging; 8th Wall understandably takes a bit to register a location. Using a default plane or image marker is much faster, so future experiences I build will probably rely on these approaches.
Summary
8th Wall is just one flavor of mobile web — the point here is that this type of experience is available now, for little cost, without restrictions of an app store or a headset because it’s all web-based.
As time goes on I hope to see (and create!) more of these kinds of experiences in the near future as we collectively decide how AR is implemented into our daily lives.
