Why Pokemon Go Will Make You Rethink the Internet of Things
Pokemon Go already looks like a pretty major technology shift. Commercially, it’s one of the most successful mobile games of all time, people are literally crashing into each other while playing, and the controversial headlines practically write themselves.
But the technology industry impact of Pokemon Go goes beyond just gameplay. At its core, Pokemon does a good job of integrating real-world paradigms like smartphone maps and cameras with virtual paradigms like animated cartoon characters — excuse me “fictional creatures” — and making it all work together in a brilliant way. The game’s goal: capture as many characters around your town/city/country as possible, do battle with others, earn points and status, and so forth. If technology is part of your livelihood or you have no hobbies, I recommend trying it.
Why? Because Pokemon Go showcases the parallel challenges for both augmented reality (AR) as well as the Internet of Things.
Augmented Reality 1.0
Pokemon Go utilizes AR and while today it is uncommon to see AR in the IoT, it will play an important role the future as the simplicity of just pointing your smartphone at a “thing” and viewing its “information shadow” overlaid on your screen has many potential applications for the IoT. You know, like this:
Information shadows include text and “virtual object” images that are superimposed over real-world images displayed on your smartphone screen, smart glass lenses, or other visually-based user interfaces. Here’s another example:
My experience with Pokemon Go suggests the game is a so-so example of augmented reality, not a great example.
Throwing a ball at a Pokemon juxtaposed on the actual sidewalk in front of you is neat and adds to the fun, but to me there is no inherent need for AR in this game. Yet this will be the first AR experience for many and Pokemon Go also makes extensive use of maps and “virtual objects” that are placed at fixed geographic locations on a map. So this is all very IoT-ey.
AR matters for the IoT in particular because it provides a powerful paradigm for interacting with endpoints. Browser-based IoT interfaces are normal today, but the user experience for users that are close to the actual IoT endpoints can’t compare with a the intimacy and interaction opportunities for pointing a smartphone at a thing and being able to interact with its information shadow. This matters more and more as users “in the field” depend on the IoT: field service techs, truck drivers, nurses, soldiers, firefighters, oil rig workers, and many more. Putting aside their access to reliable and low-latency cloud services, the ability to point a phone and begin interacting with an endpoint via AR creates big opportunities for developers.
One more point about the coming collision of AR and the IoT: AR is about placing virtual objects and information shadows on top of real world objects, e.g. Pokemon Go. The IoT is focus today on adding non-AR digital information (e.g. sensor logs) to real-world objects. In both cases, the technologies seek to enhance our interaction with the real/physical world. AR provides a new interaction paradigm that gels nicely with the needs of IoT users, albeit with local/on-premises users in mind. Similarly, IoT endpoints provide a new dimension of data that improves the performance of AR in many situations.
AR for IoT Needs Fixing
Obviously, AR is not right for all applications, but for many it will become an essential part of the IoT user experience. But before this happens for the IoT, the state of AR will have to overcome several hurdles:
1. Poor Location Precision
If your smartphone thinks you are standing somewhere you are not, your smartphone (using its onboard compass) is probably also going to think you are pointing your phone at something you are not. This is a problem for many AR implementations today — which rely on your phone’s compass and geolocation capabilities, including GPS — seeking to serve accurate AR information shadows.
Unfortunately, location precision in smartphones remains surprisingly weak. Plus or minus an average of 30 meters, due to a variety of factors. My own experience using the map feature in Yelp, for example, is that the location precision in my iPhone is usually “good enough”.
However for applications where hyper-accurate and precise location can be mission-critical — like mobile advertising or augmented reality IoT applications— even a few meters of variance can lead to errors and failure. With Pokemon Go, I found outdoor location precision to be erratic at times, but indoor location precision was always a disaster. GPS doesn’t work indoors, cellular trilateration is an unsatisfactory substitute, and WiFi location can actually result in even worse location precision. I fully expect the Bluetooth minions to be working to fix some of this, as everything looks like a nail when you think you have a hammer, but even this provides only a partial solution to both the location problem as well as broader AR bugs discussed below.
2. Reliance On The Cloud
If you point your phone at a thing and it only serves up an information shadow when a cloud server in, say, Ashland, Oregon is available, your AR experience will correlate with your ability to connect with that server. For me, this was the most frustrating aspect of playing Pokemon Go and it wasn’t limited to the AR features. Maybe the lag is just an overloaded data center phenomenon, but if the vision for AR is to rely on a cloud lookup every time we point our phones at something in order to view its “information shadow”, this AR business is going to take a while.
3. Poor Mobile Resolution
If you point your phone at something while you are moving at 20 mph in an attempt to view its AR information shadow, just know that Apple Maps will think you are somewhere different from where you were three seconds ago and your AR experience will likely vary. Same thing if you are standing still and you are pointing your phone at a moving object like a Ferrari or a dog running down the street.
The weak mobile capabilities of AR today is a subset of point #1 above and may be a less urgent bug/feature in Pokemon Go but nonetheless, it is of increasing importance for the IoT when required to connect under low power with a moving object (e.g. a car, bike, drone) or to connect while moving with a stationary object (e.g. highway infrastructure). Good augmented reality on smartphones — not necessarily the AR-lite on Pokemon Go — depends on precise, accurate location to determine where you are and in turn, at what are you pointing your phone. Today, this is hard with current smartphone and most IoT technologies.
The Ideal AR Experience
I have not seen criteria for an “ideal” AR user experience vis-a-vis the IoT so here goes:
- Pointing my phone at a thing and receiving a real-time information shadow about it, not a different thing located four feet away, while
- Querying or controlling the thing in real-time, and
- Without relying on a cloud connection, and with
- No regard to whether I am moving or standing still.
The AR in Pokemon Go does not meet the “ideal” criteria, but the game does provide a reference for future AR applications as well as a simulation of a wide area, open loop, multi-user AR IoT application that requires users to locate an object and interact with it, similar to the way public IoT networks will operate.
Simultaneously Improving AR and IoT Performance
At least part of the answer for better AR lies in work already underway with low power sensors and wireless communications.
- RTLS. Some IoT endpoints outfitted with real-time, P2P wireless connectivity offer a bonus feature called real-time location services, or RTLS. A group of things scattered throughout a building can communicate amongst themselves in real-time and, using the strength of their radio signals and basic geometry, can help determine the location of a smartphone or any mobile endpoint to within one meter. This is generally very useful given the state of location precision generally, but particularly useful indoors, where location remains largely unresolved despite the best efforts of the WiFi community and others. Specifically, consumer apps like mobile advertising and industrial apps like in-person measurements of environmental sensors on pieces of industrial equipment will benefit tremendously from this improvement.
- P2P. Again, some types of low power IoT endpoints offer real-time, peer-to-peer networking. The ability to query a thing in real-time, without the need for a cloud lookup, brings obvious decreases in latency, but it also creates opportunities to directly query or control the thing being viewed. You can easily imagine AR-enabled IoT apps that are interactive and do more than just passively view endpoint data via a smartphone and provide entire user experiences, not dissimilar from the virtual experience of Pokemon Go, without the need for a cloud connection.
- Instant-on Connections. A satisfactory AR experience enabled by IoT endpoints should not require lengthy handshaking or setup times every time a user comes into contact with an endpoint. Connections should be more or less instant. This is similarly true for connecting with moving objects or connecting with stationary objects while you are moving. Some low power IoT technologies offer such an instant-on feature.
The Near-Term Steps to Better AR
Why all wireless IoT protocols weren’t designed with RTLS, P2P, or instant-on is the subject of an earlier post. Unfortunately, most low power wireless IoT implementations are not real-time and two-way in any meaningful sense: some continually broadcast a one-way unique ID (Bluetooth) that requires a cloud lookup, and other two-way technologies like WiFi can take minutes just to get a two-way conversation going, forget about something as real-time and interactive as AR.
Getting to a real-time, P2P AR will require silicon support on the smartphone to support it. Today, getting new silicon onto a smartphone is like getting an expedition of 50 cats safely to the summit of Everest, therefore I see two paths to AR 2.0 while relying only existing silicon on a smartphone:
- LTE. The cellular industry is doing its utmost to convince us all that they will finally deliver a wireless IoT standard that allows endpoints to communicate over several miles while preserving battery life of 3–5 years. And all with a chip that costs one dollar. Alas, these chips are not available today — yes I am shocked, shocked — and it could be years before they are, but when the time comes for NB-IoT (the most interesting of the various LTE-based IoT standards now in the works), it is possible — with help from a real-time networking stack, which NB-IoT lacks — to perform direct, P2P AR using your phone. LTE is everywhere on smartphones so this approach makes a lot of sense on the phone, but the idea of cellular carriers charging subscriptions for thousands of endpoints in an industrial plant, for example, is more fantasy than probability.
- NFC. I’ve written about this before, but the NFC chip in most smartphones now can be re-purposed to support other frequencies and protocols including sub-1GHz frequencies where LPWAN technologies like LoRa or SigFox deploy. Retrofitting NFC to support one or more of these would be an IoT game changer and would cost next to nothing.
Some Final Thoughts on IoT Application Opportunities with Next-gen AR
- I am convinced, based really on my own experience with “AR-lite” in Pokemon Go, that AR is coming in a big way for the IoT, technology limitations notwithstanding. The cloud will probably serve as AR’s IoT training wheels at least initially, but real-time P2P connections are where the most exciting applications will be built.
- Pokemon Go deals primarily with “virtual objects”, while the IoT deals with physical objects or things. Overlaying a virtual object on a physical IoT endpoint is something for developers to contemplate, as well as overlaying a physical IoT endpoint on a virtual object. Adding, for example, a low power wireless endpoint at the same location where a Pokestop is located presents interesting opportunities for game developers and IoT developers alike.
- “Gamification” of the IoT is also intriguing from the perspective of mobile endpoints. In the Pokemon paradigm, there is no reason your dog could not be a mobile Pokestop. Or your car. Or your drone. Or you.
- Final-final thought: for the first time, I am considering the possibility that the entertainment industry might have a role to play in the IoT. By driving innovations in AR and through the sheer force of millions of AR users, the gaming industry has enough heft to demand real-world connectivity to their AR games. Nintendo, for example, is about to release a Pokemon Go Plus wearable device to help you play the game without using your phone. Extending this concept beyond the wearable will spawn a new category of hybrid AR/IoT gaming.
Enjoy this article? Feel free to recommend this post by hitting the little green heart below!
