Augmented Reality’s ‘Killer App’: Transforming How We Relate to Physical Places

How AR Can Transform Where We Go, Why We Go There, and How We Make Meaning of Space

Note: I recently came across a thoughtful twitter thread by analyst Doug Thompson where he makes a convincing argument that how we relate to “place” is, in a way, AR’s “killer app.” Changing our relationship to the world is a far more important concept than the more obvious idea that AR simply floods our experience with visual or audio information.

I’d like to use a tiny cafe in Amsterdam and a security guard in South Africa to build on Doug’s point.

Fundamentally I believe that one of the most powerful things about AR will be the idea that it can modify our behaviors in the real world and influence human activity (economically, socially, and professionally) without changing anything in our physical world.

Several years ago I was excited to be invited on a work trip to Johannesburg South Africa, and when I shared the news with my then-girlfriend, that work was bringing me further away from my house than I’d ever traveled before, the conversation immediately turned to Pokémon Go. It meant I’d be able to find us both the regional “Tropius” — an exclusive Pokémon that appears only on the African continent. My work trip wasn’t just providing an adventure; I’d be bringing a homework assignment as well.

Catch my girlfriend one of these, or don’t come home.

That search for a Tropius, through the streets of Johannesburg, would deliver me into a head on collision with security personnel, in a secure area, in a foreign country I knew little about.

Pokémon Go, developed by Niantic Labs, had been a meaningful part of our relationship for many years. For a while, date night often involved riding the bus, which move through the streets of San Francisco slowly enough to spin PokeStops and catch Pokémon. We weren’t the only ones. Was it possible that an augmented reality game could so thoroughly change some citizens’ relationship with (and how they use) their city’s public transportation infrastructure?

We once met a stranger in the Marina district of San Francisco who displayed, from a distance, the hallmarks of a Pokemon Go player; short shuffling steps, looking down at the phone, one finger spinning in circles before swiping up (clearly throwing PokeBalls). We learned, after striking up a conversation that he had travelled all the way from Arizona almost exclusively to catch water-type Pokémon which he couldn’t easily find in his own city. (He had recently made a successful exit from a company and could afford the lavish travel expense to pursue his hobby).

This anecdote stuck in my mind. How could something as trivial as an augmented reality game result in the second order economic affect of purchased plane tickets, hotels, and everything else that had fueled this man’s journey through the world delivering him to my neighborhood in San Francisco.

Pokémon Go, though an unusual outlier in its outsized early success within the AR landscape, showcased just how significant location-based augmented reality could be in transforming our relationship to ‘places’. AR can change where we go, why we go there, and what we do when we get there; without changing a single thing about the physical world.

This mob was caused by the rare public sighting of a wild Vaporeon and nothing about the ‘physical’ world.

In 2015, during a trip to Amsterdam where my girlfriend was living at the time, she took me to the Blue Teahouse in Vondelpark, an outdoor cafe in the center of the Netherlands’ largest city park. It wasn’t crowded and we had no trouble finding places to sit. One year later, it was nearly impossible to find space. The reason was, of course, the release of Pokémon Go.

Before and after shots: Pokémon Go released (also nice weather).

To this day, and after almost four years of playing, I have never seen anything like the collision of digital and physical infrastructure transforming human activity in a shared public setting than what I saw at the Blue Teahouse that summer. It was due to an unforeseen oddity in the deployment of PokeStop locations in the game’s map.

In the physical world, the Blue Teahouse is already a social space designed for people to sit, drink coffee, and enjoy being outdoors with friends. In the digital world the restaurant space, for whatever reason, sat well within proximity of three overlapping PokeStops. For those who don’t play the game, it’s worth noting how unusual that is. The purpose of Pokémon Go is to encourage players to move physically through the world accumulating items and finding Pokémon. Yet here was a spot, where players could circumvent the need to move through the world in a physical location already designed for social activity. It was a killer combination.

The Blue Teahouse as it appears in the game. Players can spin a PokeStop every 5 minutes to get Pokeballs. The pink confetti shows that all three PokeStops have lure modules applied by someone nearby. Lure modules last thirty minutes and cause Pokémon to spawn. Everyone now has what they need to sit, enjoy, and play at the cafe.

For that entire summer, The Blue Teahouse became an oasis for Pokémon Go. Almost overnight the cafe’s culture transformed, and the value that guests assigned this physical space now derived just as much from activities in the digital world as much as any purpose it served in the physical one.

In a discussion with a manager at the restaurant, I later learned that the cafe increased their revenues significantly. So many people were there playing, visitors were forced to crowd the front lawn sitting picnic style on the ground; most of them playing too. Teenagers were showing up to the lawn well before it opened and (since Dutch kids have no bedtime) were staying long after it closed. On cloudy and rainy days, usually slow for the business, tables were just as crowded as days with sunny skies. For this tiny cafe in Amsterdam, the game was an unexpected stroke of good fortune and the business impact was enormous.

Sorry we’re at capacity. Please enjoy this lawn.

The broader impacts of human activity being changed by the game weren’t always so pleasant. These anecdotes were well covered in the media, but one interesting example occurred in Milwaukee Wisconsin where players left trash and disrupted the ecology of a public park. The city tried to require companies like Niantic to request and be granted a permit to operate their games on public land; an attempt which failed.

All of these developments suggest that we’ll need entirely new policy frameworks for dealing with a world ‘painted with data’ (as Charlie Fink describes this coming world filled with AR content). What types of issues may arise?:

Should a company be able to sue someone who leaves digital graffiti on their building?

You can ban a person, but what about their hologram? Who regulates the free movement of AR technology?

What are the limitations to where AR game artifacts can be placed?

Should companies be able to freely place AR content wherever they please?

Is digital “property damage” a thing? It’s the wild west in AR and this is cold-blooded from Burger King.

Example: In response to a Snapchat marketing concept that included a location-based art exhibit which placed AR sculptures by Jeff Koons in several locations around the world, artist Sebastien Errazuriz thoughtfully used a vandalized Koons sculpture to challenge the notion that corporations be allowed to freely place AR content without paying anything back for the value they extract by using public land.

Errazuriz asks:

“Should corporations be allowed to place what ever content they choose over our digital public space? Central Park belongs to the city of NY. Why should corporations get to geo-tag its GPS coordinates for free? We know they will make money renting GPS spots to brands and bombard us with advertisement. They should pay rent, we should choose to approve what can be geo-tagged to our digital public and private space.”

All of this suggests that AR is about more than just feeding information into our eyeballs. It can change the world we live in by reshaping how we engage with it. AR can determine where we go and why we go there but also influence the way we construct meaning about the significance of a place.

The world map that matters: Regional exclusives in Pokémon Go.

Before our meetings in Johannesburg, my coworkers and I budgeted one extra day so that we could acclimate to the time zone. Not me though! That first day on the ground was for finding a Tropius. Fueled by a primal need to prove myself as a hunter, provider, and partner; finding us this Pokémon became critical. Some might argue against the ethics of me logging in to her account, a form of business trip spoofing, rendering me little more than fleshy telepresence for her character to pop into South Africa to catch it. I, however, would argue that it is not cheating. It’s love.

I sat at our hotel in the lobby doing work while my phone was open to Pokémon Go next to me, just in case a Tropius showed up on the “nearby” list. The game had recently made changes to the feature which prioritized showing where Pokémon not yet in your collection were located. This was big for me, and within the first hour of turning on the app, a Tropius showed up on the map. Time to go.

Google says that is an 8 minute walk.

I packed up my stuff as quickly as possible, looked up the location on Google maps, and headed into the world. I had no idea how long the spawn had been up or how long it would last so I needed to hurry. In Pokémon minutes, eight is a long ways away. I felt a surge of anxiety as I started to walk, quickly, and at some point made a wrong turn, stopped for directions, recalibrated, before finally I arrived at the place on the map.

Glory to me, the app showed the Tropius was still there! But there was a problem.

Uh oh.

It was in a parking lot. Blocked by a security guard. This can’t be happening. Normally, I’d like nothing to do with any form of law enforcement, especially in a country I’ve never been in my life. But the urge to bring joy to my girlfriend overpowered and I nervously stepped forward to try and talk my way through.

First, I had to explain to the guard (his name was Frans) what Pokemon Go was. Knowing the urgency I kept the description as brief as possible. Next, I had to convince him why, given what little he now knew about the game, he had to let me into this secure area to capture my Tropius. I’m not proud of this, but I ended up giving him the last US currency still in my wallet, a five dollar bill to let me into the area. (*Editor’s note: This bribe is still to this day, the only money I have ever spent playing Pokemon Go*)

worth it. those heart eyes scroll on for a while.

There are, of course, certain places in the world that Niantic should not be encouraging their users to go and catch Pokemon; memorial sites, hospitals, and certainly secure areas. But on this day, that’s exactly what happened.

It turned out that this particular parking lot was usually available to the public, but there had been a festival taking place over the weekend. The vendors and event producers were busy tearing down their tents, and the security guard was working the last shift before the space opened back up.

Here was a lesson in the way AR content technology makes sense of real places. The world is a messy, chaotic, and ever changing landscape of activity. This ‘place’ may have been one part parking lot and another part private event. To me, it was simply “where the Tropius was.” Could Niantic’s technology have understood that today was a bad day for Pokémon in this particular area? Today the answer is no.

But soon that could change. But that will require the AR Cloud.

At its core, the AR Cloud may be the most important technology development no one is paying attention to. The term, first coined by Ori Inbar, refers to the notion that we’ll someday have a digital copy of everything in our world. To oversimplify, the AR Cloud is a living, breathing, digital map (or collection of maps) of everything happening in the world. It will become the canvas that allows AR creators to layer the world with information.

Inbar writes that:

“the AR Cloud, will be the single most important software infrastructure in computing, far more valuable than Facebook’s Social graph or Google’s page rank index.”

Edward Miller, the CEO and founder of Scape Technologies, an AR Cloud startup recently acquired by Facebook, describes the technology as consisting of three things.

First this technology will need to know where you (and your devices) are. Second it will need to know what is around you. And third it will understand what is changing. From these three things your devices can one day begin to gain a semantic understanding of the world.

It’s not enough for Pokémon Go to know “this is a parking lot” or “here is a security guard.” It must be able to infer meaning to understand that this has become a secure area and today is not a good day for Pokémon (especially relationship-saving rare ones) to spawn.

This, for example, is close to the work that people like Matt Miesnieks are doing and thinking about.

Before Covid19: monthly community days in Pokémon Go still drew big crowds: Oxford UK (top left), Amsterdam (top right), and in San Francisco Fort Mason usually turned into a hotbed of activity.

Even before true AR (in the device-wearing sense) spills into the world, we’re already part of a society whose understanding of “place” is shaped by layers of information attached to physical spaces. For example, Oxford Internet Institute’s Mark Graham, Professor of Internet Geography, points out that users searching for ‘restaurants’ in Tel Aviv, are shown entirely different landscapes of the city depending on whether they search in English, Hebrew, or Arabic.

Graham writes:

“Not only is there a difference in the amount of material available — Hebrew and English both have considerably more hits than Arabic — but the restaurants that are selected and the order in which they are presented is considerably different for each language.”

Divergent realities are already constructed for users by Google depending on culture. I have once seen Harry Potter Wizards Unite players, another Niantic title, descend on the same landmark that Pokemon Go players were using for a raid battle. Not unlike the way religious stories can fragment people’s meaning of ‘place’, here were two vastly different mythologies making use of the same literal building.

Content creators and technology companies who then design our AR layer of data will have a massive role in shaping how we make sense of and move through our world. There are huge ethical and sociocultural aspects to these developments.

The biggest impact of AR, then, will not be that it gives you superhuman vision (though it may also do that) but rather that it can meaningfully change how we relate to physical places.

For a long time, I routinely walked one block out of my way to workout at my gym so that I could pass the Starbucks where there were usually good Pokémon spawns. In that sense, an AR game changed how I moved through my neighborhood. And one day Pokémon won’t be our primary example. (*Waze GPS navigation — which is also AR — already tries to get me to add extra stops to a paid advertisers’ location during a drive somewhere, but I have never once done so. Too inconvenient).

For now we can only speculate how new layers of data on top of our neighborhoods could change our behaviors. We’re in the earliest days of AR Cloud technology development and Pokémon Go is still a massive but early anecdote which can show us what’s coming.

It’s a commercial for Facebook. But also a good vision of the way map data could transform our world.

Big investments at companies like Facebook, Google, Apple, Niantic, and others are coming. Soon AR could play a bigger role in our lives and not only can it transform human activity, but it can also reshape how we come to understand and relate to our world.

The physical places we live in will be increasingly transformed by the augmented realities and layers of data that exist as a very real — and ever expanding — part of those spaces.