Augmented Reality will not be graphics on a screen
Hot take: We’re looking at Augment Reality backwards.
What the future is not
The promise of augmented reality is to embed the world around us with the amazing capabilities of the digital world. Traditional thoughts of what Augmented Reality is viewing the world through a viewport. That viewport is either a phone like in Pokemon Go or some other screen like Google’s Glass or any Heads-Up Display.
I’m proposing that Augmented Reality will also have an alternative future. Augmented Reality won’t be humans seeing reality over laid with digital graphics and data, it will be the computer seeing the physical world embedded with digital qualities. Put differently, we will not be the viewers of Augmented Reality, but the computer will be.
Our life will not, or at least should not, progress to the point where everything we own is a surface of a screen. As someone who works in the tech industry, I welcome the future which tech will build, but I am not excited about a life of glass. I believe in textures, in the look of things well worn, of objects knowing you without being Smart. This is not nostalgia of how things once were, but I say this as a human with 5 senses.
Even TVs are becoming less obviously electronic devices. The big massive black boxes of the past are beginning to look like picture frames as they fade into objects of our home rather than Grand Purveyors of Bits and Pixels. Technology will fade into the background as the functionality it provides will come to the forefront. That functionality will be to serve a purpose and only when asked to.
But this argument goes beyond preferences of textures or tactile experiences. The user experience of a life lived through screens is terribly limiting. Instead of embedding smarts into that cup of coffee next to you, imagine if a external sensor knew what was in the cup, how long the cup has been there and even whose cup it is.
What the future of AR is not
Samsung recently launched a new refrigerator that tells you what is inside. It’s an interesting idea, but I think fundamentally having a screen show you whats behind a door is not as compelling as just knowing what’s on the other side. You don’t need a screen on the door to do that. The value is not in being able to see it, but being able to know the information.
The promise of Augmented Reality is having more information in context. You don’t need a screen to do that, but you do need sensors.
Even Pokemon Go, the darling of AR, was compelling not because you could see pokemon through your screen, but because the pokemon were aware of where you are. It’s emphasis on the Go, as in getting out of the house and experiencing the world, is telling. The bodega on the corner of your street was embedded with the values of an Evee. That’s what made the game sensational. Don’t get caught up in the interface when the value is a layer deeper. It is simplistic, but contextually aware; that’s the sort of thing that makes Augmented Reality compelling.
There will uses like games or snapchat filters where changing a photo will be entertaining. Admittedly, Apple’s ARKit and Google’s ARCore is counter movements to this argument. And maybe changing the physical world through a digital screen is a necessary first step. However, I believe AR will be everywhere and it will follow you and you won’t have to wear it or see it through a screen.
VR will be perfect for going to the movies or playing video games, but Augmented Reality will be more ubiquitous. It connects you closer to the physical world where as VR removes you from your physical world. Both will have their place, but my vision for AR will not conflict with VR.
What will it be then?
Think of the first iterations of the Fitbit. They were a couple sensors that connect to your phone via bluetooth. They measured a few things and then sent it off to some far away servers. To get information about what a Fitbit was collecting, you had to look at the information on your phone. Your phone didn’t store the data, the cloud did. The cloud took the data, combined it with past data and then gives you actionable information about your activity.
The version of Augmented Reality that I see will be similar, but more advanced. It will be a series of sensors ranging from simple cameras to more advanced sensors. They will be sending data into advanced data science systems like convolutional and recurrent neural nets that turns the data into information. Tracking things and information through time will be important, and that information will be associated to actual objects.
Benedict Evans has some great aphorisms regarding how the camera is the most important sensor.
Using these simple sensors, we can see how our smart refrigerator mentioned before will benefit from this inverse AR. If there are appropriate sensors and smart enough neural nets, then theoretically our refrigerator could always be stocked without effort from the user. It’s not all that outrageous of a thought.
Here is a simple illustration:
- Build a neural network trained on understanding brands and items of food and drinks that live in your refrigerator. Use sensors, like cameras, to understand what is in your refrigerator and then find data based on the specific items of food from an existing database.
- Have another sensor, like a scale inside the refrigerator, to understand the weight differences between when you took your OJ out and put it back in to calculate how much was consumed.
- Use more data science to plot usage over time of day/day of week/week of month/time of year and then anticipate how much will be used to match supply with demand.
- Then purchase accordingly for a shipment of groceries to arrive just-in-time at your home.
Why is this valuable?
In the previous example, it seems silly to have all this technology just to answer the question of what you need from the store. But it’s not there answer the question, it’s eliminating the need for the question.
The recent announcement of the TrueDepth camera on the new iPhone X does something similar. Think about how much effort went into the sensors on this device.
Think about how all that technology is built just to unlock the phone without using your finger to unlock it. It’s not that using a finger was particularly laborious, but what FaceID does is eliminate the need to take any action. Your phone is unlocked by the time you look at it.
It’s all about eliminating the need to ask questions.
You can easily extrapolate our orange juice scenario to illustrate questions that would be eliminated:
- Your refrigerator know the caloric measurement in OJ that was consumed based on the weight and nutritional information of the OJ. Answering the question, what’s in the food?
- Your kitchen sensor will understand what percentage of the food you ate vs your family vs how much you threw away. Answering the question, what did you eat and how much of it did you consume?
- Your personal activity sensors, like fitbit or phone, will then record how many calories you burned throughout the day to automatically tell you if you’ve gained or lost weight today. Answering the question, how does the food you eat and your activity affect your health?
- After breakfast, you realize your spouse left the OJ out. Your kitchen sensor saw when it was left out and knows how long it can be left out before it’s spoiled warning you before you take a potentially sickening sip. Answering the question, can I drink this?
It’s the computer that has embedding value into the products inside your refrigerator. It’s the computer seeing those values. And it’s the computer stringing that information together to give you value. You are just consuming the value it’s produced.
The neural nets could figure out our recent buying history, matched with a recent eating history, matched with what we actually have in our refrigerator, it could automatically understand what needs to be ordered and order it.
Taking it one more step further
But is what I’ve described augmenting reality or is it just showing you what reality is? I think there is a lot of opportunity of explicit augmentation without screens.
As a designer, I think of all the ways you can tie physical objects to certain values that creates a amazing new world of physical-digital interactions. Simple things, like lights switches, could be a thing of the past. If you have a sensors that can see anywhere in your room, you can associate tapping a specific part of the wall could turn off a light.
Or if you always have a small item, like a table coaster, on your coffee table you could rotate that coaster and it changes the volume of your television or speakers.
Imagine a scenario where you can throw a bill from the mail onto your table. As you open it, the office sensors scan the contents and saves it. Then you tap the bill to pay it. You can then immediately throw it away, knowing it’s been paid and digitized for future reference.
You leave a notepad on the kitchen table for your spouse when they arrive home. You can embed a message onto it by tapping it and then talking. Then when your spouse finds the notepad, they tap it and it plays back the message.
All of this is understood by passive, always-on sensors and embedding digital values to “dumb” physical objects.
This is the vision of the future that I want to see. It’s one of texture, of the physical world, of meaning, and of technology thats fades away more and more as it becomes more powerful.
The whole physical world becomes something that it already is: a potential interface for you to interact with and no screens necessary. But this time, it’s digitized, has super powers, and hopefully with the personality of 18th century french butlers.
