Augmented Reality: The Inexorable March
With the advances in technology over the last ten years, it is amazing to see that the retail experience has remained largely unchanged. Although the internet has stolen the echo of a million footsteps, those that still brave supermarkets and clothes shops are mirroring a pilgrimage which hasn’t changed since the dawn of time. They might be self-serving at the checkout but, in essence, this is a domain untouched by technological advancement.
As someone who feels a moment of dread, a deer in the shop assistant’s headlights, when they lock eyes and utter those fateful words: “can I help you with anything?”, I am relieved that we might be on the cusp of a new generation of shopping experience. With the advance of augmented reality (AR) on both iOS and Android we are at a point where AR has the ability to extensively change not just the products we buy, but the essence of shopping itself.
In this article, I am looking at the impact AR could have on the retail space, ways in which it might affect the consumer, how it works and where we can apply it. I will also be analysing common misconceptions about what AR is and exploring ways of maximising its value. Getting a consumer to take out their phone and use an app is an uphill battle — meaning we, as developers, need to be very discerning in how we approach a solution.
What is AR?
AR is a technology which enables us to impose a computer-generated image onto a user’s view of the world. Using the real world as a background, objects can be inserted and interacted with. AR can detect horizontal planes, the floor or surfaces, adapting the experience based on where the camera looks, how the user moves, and their physical location in 3D space.
If AR is completely new to you, check out this great video which goes over the basic concepts in more detail.
How does it work?
To fully realise the potential of AR we must recognise both its strengths and weaknesses. For this, we need a rudimentary knowledge on what is going on behind the scenes.
Everyone has used the maps functionality on their phone. The application uses your location, which way you are facing, and your elevation to get you to your destination. This works excellently on a large scale, utilising data from multiple GPS satellites, where a ten metre discrepancy only puts you on the other side of the road. For building navigation, a higher level of control is required to know exactly what the user is doing. This is made possible by the recent availability of both hardware and software which combine to provide the significantly higher degrees of freedom and accuracy required by AR.
Device rotation allows us to sense the pitch, roll and yaw of the device. In layman’s terms this is the rotation of the device in x, y and z planes of movement.
Movement in 3D space is detected using the sway, heave, and surge of the device. These let us know when the device actually moves in those three planes of motion.
Image analysis is the final piece of the puzzle. With the capacity to identify planes and specific shapes, QR codes for example, it gains more information on what it is looking at.
AR is the combination of all the above — device tracking, scene analysis and 3D scene rendering. The device knows exactly where you are, which you’re facing and precisely when and how you move. Using this information, it can insert and manipulate computer-generated images to make them appear as though they are fixed in space. The cherry on top is being able to partially understand what it sees — allowing it to add information based on what it is looking at.
Armed with this knowledge we are now ready to look at the two biggest innovations that I can see coming to the retail space:
Path-finding: Helping users navigate a store to find specific products
Enhanced experience: Analysing information the user receives when viewing products
Path-finding
Path-finding overlays information and route details onto the user’s screen. Using knowledge of the user’s position, objective and obstacles, an optimal route can be shown on their screen.
Most of us know the frustration of not knowing where a specific product is located. This might occur when visiting somewhere new or after a shop has rearranged. Similarly, in the shops we don’t visit regularly, we waste time looking for someone to ask for directions — before we even start navigating there.
This is where path-finding comes in.
The most basic use is being able to guide a user to the product they are after. Having virtual arrows guide them around the store directly to the tinned sardines allows shoppers to quickly navigate to their desired product without needing to call for assistance.
Improving efficiency
The obvious next step might appear to be utilising this further. To link with a user’s online basket, plotting their most efficient route. At this point we must step carefully with AR, looking at how it can be used to best effect instead of attempting to throw the kitchen sink at it. We get more value by focusing on what the user can’t currently do, using AR to improve this experience, rather than offering too much by trying to fix everything with AR.
For this reason, I feel that the technology is best suited to the following simple functionalities:
Item location
Way-finding truly shines in larger stores or ones not visited often, with consumers wasting significant time finding their bearings and knowing which direction to go. It couples well when searching for a single specialised item, when a direct route in and out of the store is the best outcome for the shopper.
Calling for help
Many stores save money by hiring less staff, but they reap what they sow when customers need to ask a question and can’t find anyone to help. Path finding enables them to signal for help, see the assistant’s location and see them making their way to you. This frees you up to continue shopping while waiting for the assistant to arrive and alleviates the frustration this scenario usually induces.
Setting your point of reference in AR allows you to know your position relative to items or shop assistants
Enhanced customer experience:
So far, we’ve looked at how we can make the user’s shopping experience better by improving their efficiency finding a specific product. Now we’ll take a look at the possibilities once they find it. With hundreds of products available, consumers have their own unique brand and cost preference. AR allows all of this information to be taken into account.
Before we get our teeth stuck in, it is worth understanding a little more about image recognition. This touches on a misnomer around AR. AR uses image recognition to analyse what the camera sees and insert images into this frame. People often assume all image recognition is AR, this isn’t the case. Machine learning and its common use case of image classification can be combined with AR but is considered a completely unique field in its own right.
There are two main ways we can utilise image recognition to identify objects:
Specific image identification: The technology behind AR can identify certain images. These might be added manually to the project, with information assigned to it, or be a more general class of image, for example a QR code. Whereas QR codes have a certain amount of information baked into their design, manually images can retrieve information from a database once they have been identified.
Image recognition: Companies are doing a huge amount of work on machine learning at the moment. Instead of recognising a specific image, added to an application, it analyses the camera input and identifies what is actually there. It looks at the colours and shapes, using deep machine learning, to identify what it is looking at.
It is worth remembering once again our value adding mission statement. What do consumers want that will enhance their shopping experience?
Search and filter
We have all, at one point in our lives, wished life had a search functionality.
CTL+F — “Missing brown sock”An ability that only mums seem to possess when looking for the ketchup in the refrigerator. In our high speed, global environment this is even more apparent. With multiple brands all sitting on the same shelf it is easy to become paralysed by choice.
This is where image recognition comes in. This technology can simplify what the user sees, either searching for something particular or identifying additional items that might be of interest. In doing this we remove the superfluous information from the real world, allowing the user to focus on what they really want.
Are you a vegetarian? Want to buy more Australian products? Trying to reduce your sugar? These filters can be applied by simply pointing the phone camera at the display shelves, automatically identifying the best products for you.
This simple example highlights the strength of this technology. Our brains can only process a finite amount of information at once, while our phone processor can analyse, filter, and display a huge amount of information very quickly.
Information enhancement
With this window to the augmented world we can provide access to not just more information but targeted information. A tablet or mobile phone allows easy representation of data and comparison of products, something not easy in the real world. This is where image analysis comes in. The ability to identify brands and understand what is being looked at allows us to make predictions on what the user wants to know. For example, looking at a shelf of house plants we could see which require a similar amount of attention and could therefore grow together in the same window box.
We have looked at the strengths of AR. Before we finish we should look at its limitations, too. AR is not a magic bullet. The technology is improving in leaps and bounds, but as developers we still need to take its weaknesses into account when designing a new experience. In our above example a key problem is applying our solution to the real world. Not all stores are the same size or shape, and stock might change location regularly — how will this affect our way-finding? We need our experience to be easily translated to different environments instead of requiring significant work for every new location.
The answer depends on the model we base our application on. In this case we would split it into two distinct parts. The first would contain the immutable information. The shape and size of the store, the shelving units and other objects customers can’t walk through and won’t change position regularly. This would be coupled with a more transitive layer where we would place our stock, advertising boards and other changing objects.
This allows us to split up the problems we are looking to solve. We group specific objects into categories and then place these on our immutable map. This allows us to create routes from the user to a specific area. We can then use image recognition to identify the objects they are after and provide detailed information about this object.
In this way we can manage AR’s limitations by carefully planning the model used before starting development. AR must be used like any other tool — at the right time, in the right way, and with a strong framework of programming supporting it.
The enigma behind AR seems to be that, for once, technology is not trailing implementation. We have all the tools but are still trying to understand what it is that consumers want, what they will use, and how the technology will add value to businesses. With developers all over the world learning the language and frameworks behind this technology, we are now waiting for this innovation to change the world and the landscape in which we live.
Augmented reality is a door to another world of possibility. The technology is well and truly here and, for the last few years, companies have been hesitating to adopt it. We see flashy videos showing its possibilities but don’t see it in our everyday life. All it will take is the first killer experience to be developed and the floodgates will truly be opened. I believe that this breakthrough will have an elegance and simplicity which will leave the world wondering how we didn’t see the wood through the trees.
Have a question about the technology talked about in this blog? Don’t hesitate to get in touch with us at Outware, part of Arq Group, to have a chat about the research we are doing or how it can be leveraged for your business.
Simon Smiley-Andrews is an iOS Developer at Outware.
Originally published at www.arq.group.
