How Amazon created the magical Amazon Go buying experience.
“Any sufficiently advanced technology is indistinguishable from magic” — Authur C. Clarke
This story first appeard on Quora (http://qr.ae/THTPMh) and Forbes (http://www.forbes.com/forbes/welcome/?toURL=http://www.forbes.com/sites/quora/2016/12/06/amazon-go-ends-checkout-lines-and-shoplifting-begins-a-new-era-in-retail/&refURL=&referrer=)
Just 1-click And The “Self Driving” Store
It was a cold day at Amazon headquarters on September 12th, 1997. Jeff Bezos was nervous as he submitted a new and unique patent to the USTPO called “Method and system for placing a purchase order via a communications network “ . This came to be known as the Amazon 1-click patent and a few years later it was licensed by Apple for their websites and later iTunes. It defined the web experience since.
“Shop And Just Walk Out” — Amazon, 2016
Amazon just invented the one click buying experience for retail.
Specimen: Front elevation of a typical Amazon Go retail store.
Today, December 5th, 2016 Amazon announced the Amazon Go Store a new shopping experience that fundamentally changes the very fabric of shopping and payments at the same time. Amazon has combined decades of research with artificial intelligence and machine learning along with image recognition. They have combined all of the knowledge that they’ve gained from the one click buying experience and brought this to the retail store.
In 1861 John Wanamaker invented the way we currently shop . Today Amazon just reinvented the entire retail and payments experience. It is a store with no credit card machines or cash registers, just AI. Amazon is using the same technology that can be found in the life-saving aspects of self-driving cars, in the Amazon Go Store. It is very powerful technology.
Specimen: Amazon Go retail store video demonstration.
Amazon Go Ends Check Out Lines And Cash Registers
It’s called “just walk out” technology and when you walk out, your purchase is complete with a receipt in your app, charged to your Amazon account. This is achieved by an entryway that is similar to the subway turnstiles that you see in major cities. Yes this sounds like magic, retail magic.
Specimen: Of the subway turnstile entrance area for the Amazon Go retail store.
Specimen: Of the QR code used to enter the Amazon Go retail store.
Specimen: Of a cell phone scan to enter the Amazon Go retail store.
A smartphone with the app installed is required to enter the store via presenting a barcode to a sensor. This barcode scan (along with other sensor technology eg: GPS, etc) tracks that you have entered the store, identifies you’re moving through the store and then identifies the product you pick up. To complete your shopping experience just walk out the door. Yes, that’s it. The image recognition combined with a sensor fusion of technologies has already confirmed your order and totalled it up. All billed to your future Amazon Bank Card, but currently to the payment card you have on file with them, the nearing 1 billion payment cards they have on file.
Specimen: Of the automatic shopping cart in the Amazon app for the Amazon Go retail store.
Specimen: Of a customer leaving the Amazon Go retail store.
Specimen: Of a customer verifying a receipt for the Amazon Go retail store.
The End Of Shoplifting?
This Amazon magic is all achieved through a number of very advanced technologies. The most fundamental is the use of hundreds of image sensing cameras. Using advanced artificial intelligence and machine learning, Amazon can detect a product that is not only missing from the shelves, Being held in your hands, put into a bag, or even under a shirt. Yes, Amazon has solved shoplifting almost entirely.
Specimen: Some of the sensor technology at the Amazon Go retail store.
Members Only, No Cash Or Credit Cards Accepted
Amazon is fundamentally changing everything we thought we knew about retail sales. They have merged AI and ML data science that they just about invented for online sales and meshed it for a unique retail buying experience. The ideas here are familiar and can be found at play at Apple stores when using the Apple store App and self check out. Amazon took it to the highest degree, there is no self check out. Like a buying club or Costco store, you have to be an Amazon Prime member to complete a transaction. In fact they don’t accept cash or payment cards in the store, there are no credit card machines and there are no cash registers or POS systems. This is the beginning of the end of the POS system.
Custom Real-Time Coupons And Discounts Based On Prior And Current Behavior
Just like when you visit a website and you are logged in, the Amazon Go shopping experience is tracking all of your shopping behaviors. Over time this will inform Amazon on the exact placement of products and how consumers may interact with them. Machine Learning about the amount of time spent in the store and the transverse path you make through the store will assist Amazon in creating customized, on demand discounts related to your current or prior buying behavior. Some may find this a bit troubling to be watched to closely while shopping in a retail store. However live video cameras and real-time AI systems have been used for years. The Las Vegas Casinos have bene expert at trailblazing this technology to spot banned gamblers and/or cheats.
The Tale Of Two Patents
Those of you have been following me on Quora know that I have been talking about Amazon entering into not only retail selling but online and retail payments. I wrote that at some point in time Amazon would become the AWS for payments both retail and online just like they became the AWS for web services. There were several things that allowed me to understand just how this would happen. In two patents filed on June 26th, 2013 Amazon cited “Detecting item interaction and movement”  and on September 24, 2014 cited “Transitioning items from the materials handling facility”  . Both of these patents telegraphed the future that has now arrived in Amazon Go.
Initially these Amazon patents seem to be related only to a use case inside of warehouses or fulfillment centers. However the insights I had in 2013 suggested Amazon was seeking to patent a product that duplicates the one click buying experience that they have mastered on the Internet now applied for a retail store.
The “Detecting item interaction and movement” patent is about as close as you get to what Amazon Go is doing today. Here are some of the patent’s embodiments:
If the user enters or passes through a transition area, such as a rebin station, another inventory location, pack station, shipping, exit from the materials handling facility, etc., the items identified on the item identifier list may be transitioned. Because the items have already been identified, the transition may be done automatically without any affirmative input from the user or delay to the user. For example, if the user, such as a shipping agent, is exiting the materials handling facility with the items, the user will pass through a transition area (such as a loading door, a door exiting the materials handling facility, etc.) as they exit the materials handling facility. When the user enters or passes through the transition area, the items identified on the item identifier list are automatically transitioned from the materials handling facility to the user. Continuing with the above example, the items may be automatically transitioned from the materials handling facility to the shipping agent in response to a determination that the shipping agent has passed through the transition area. As another example, if the transition area is a rebin station within the materials handling facility, the items may be transitioned from the original storage location to the rebin station. As still another example, if the materials handling facility is a retail location and the customer is purchasing items, when the customer passes through the exit (transition area) of the retail location, the items picked by the user may be automatically transitioned from the materials handling facility to the user and the user may be charged a fee for the items. In a similar manner, if the materials handling facility is a rental location, a library, etc., the transition area may be near the exit of the facility. When the user leaves with items, the items are transitioned from the materials handling facility to the user that is taking the items. In such a case, a rental or borrow time identifying when the items are to be returned may be associated with the items and the user. The transition of the items may be done automatically and without any affirmative input or delay to the user. For example, if the user is purchasing items from a retail location, rather than the user having to stop and “check out” with a cashier, teller or automated check station, because the picked items are already known and identified on an item identifier list associated with the user, the user may simply exit the retail location with the items. The exit of the user will be detected and, as the user passes through the exit (transition area), the user, without having to stop or otherwise be delayed, will automatically be charged a fee for the items (the items are transitioned to the user).
Here is the patent’s abstract:
This disclosure describes a system for tracking removal or placement of items at inventory locations with a materials handling facility. In some instances, a user may remove an item from an inventory location and the inventory management system may detect that removal and update a user item list associated with the user to include an item identifier representative of the removed item. Likewise, if the user places an item at an inventory location, the inventory management system may detect that placement and update the user item list to remove an item identifier representative of the placed item.
Specimen: Amazon patent.
The “Transitioning items from the materials handling facility” patent also shows
This disclosure describes a system for automatically transitioning items from a materials handling facility without delaying a user as they exit the materials handling facility. For example, while a user is located in a materials handling facility, the user may pick one or more items. The items are identified and automatically associated with the user at or near the time of the item pick. When the users enters and/or passes through a transition area, the picked items are automatically transitioned to the user without affirmative input from or delay to the user.
Here are some of the patent’s embodiments:
By detecting the removal and/or placement of items by users in the materials handling facility, the movement of items within and exiting the materials handling facility may be accomplished without requiring the user to undergo additional transition of the items. For example, if the materials handling facility is a retail store, users may pick items from within the facility, place the items in a cart, bag, pocket, or otherwise carry the items and the items are automatically identified and associated with the user. When the user exits the facility the items may be transitioned from the facility to the user and the user charged for the items. This may be done without the user having to undergo the additional step of checking out with a cashier, the cashier identifying and scanning each item and the user paying the cashier for the item.
Specimen: Amazon patent.
It is clear Amazon thought about this for over 4 years and perfected the use case inside of their own warehouses. Quite unknown and unseen by many is how Amazon cross confirmed the ML and AI based image recognition they pioneered.
In 2009 The Image Recognition Technology Seed Was Planted
This goes way back to 2009 and the acquisition of SnapTell  by Amazon. They developed image recognition technology that could identify a huge number of popular products just by their images. This technology was absorbed by Amazon’s very private “Bell Labs of this century” labs. By 2014 Amazon integrated this technology in its app for what has become known as “showrooming” . This allows consumers to visit a local store, take a picture of a product and instantly get a price comparison. It has since been de-emphasized.
Specimen: Amazon image recognition system used in the Amazon app.
The End Goal Is To Be AWS For Retailers And Payments
Amazon Go will obviously disrupt retail. However I do not think Amazon will go too far with retail stores. Of course there will be many of them. However I think that Amazon wants to be to be the AWS (Amazon Web Services)  for retail stores. Amazon already convinced online retailers to trust them with AWS for their web hosting and today a vast majority of commerce sites use this service. Additionally >65% of sales at Amazon are via small merchants where Amazon is simply fulfilling and billing for the transaction.
Amazon’s long term goal is to be the technology supplier for all online and retail commerce. It seem clear to me that Amazon will license this technology to small and large retailers just as they have licensed AWS technology. Payment companies, both young and legacy, along with merchants both online and retail will need to vastly adjust their business models. If not, they will be obsoleted. Thus far not a single young or legacy payment company is positioned to take advantage of the shifts that are taking place, and it will be a hard road ahead. This will begin and end with payment companies. Astute founders and VCs that work with me, knew these details over a decade ago. Gordy Howe, the legendary hockey player, said “The secret to success in hockey is not being where he puck is, it is being where the puck is going to be”. Some will be left standing flat footed and astonished.
Cashiers In The Future, There Will Be Less But Not For A Long Time
The Bureau of Labor Statistics says that cashiers are the second-largest occupation, with 3.5 million employed in the U.S.
Amazon is pursuing the grocery and convenience store business, which represents about 17 percent of total U.S. retail, or nearly $800 billion.
I presented the idea that Amazon is very likely going to license the technology around Amazon Go to other retailers (with some limitations) just as they created the AWS service for websites for just about every major retailer. AWS is the largest service of this kind and the most profitable part of the business.
On the arc of 10 years one can assume that directly Amazon may impact at minimum 10% of the ~3.5 million cashiers. However it is clear other companies will cobble together perhaps a less advanced version from Amazon Go and likely also impact the number of cashiers employed.
In the worse case one can conclude that on the 10 year arc up to 50% of the jobs could be impacted. On the other hand there will be an increase in workers to maintain these systems, albeit not nearly as many jobs will be created. Additionally, there may be a bit of a rebellion on a local scale that may crave a human and person touch.
Thus on the 10 year arc there will be jobs lost to the cashier worker category. However It took three decades for ATM Machines to even slightly impact the number of Bank Tellers.
A Hybrid Of Everything Amazon Learned Online
Amazon Go is the hybrid of many technologies and ideas that started with the very first website Amazon created. When Jeff Bezos invented and patented the 1-click shopping experience, that Apple licensed later, few could imagine it could come to retail stores. It is the paramount amalgamation of:
- 1-click-like web shopping in retail
- Powerful App using location based services
- QR Code IDs
- Integrated Payment
- Image Recognition
- Multiple Sensor Technology
- Artificial Intelligence
- Machine Learning
These ideas are not entirely new. IBM spoke to this in the mid 2000s centered around RFID technology. Although Amazon is not using RFID to any material level at Amazon Go stores, it is a foretelling of the future of retail from IBM:
Specimen: IBM RFID video.
Amazon Go Is Part Of The Voice First Revolution
Amazon is also pioneering in Voice First systems like Alexa. Voice First systems will become the best selling items sold during the holiday shopping season of 2016 at Amazon. There is no doubt Amazon will integrate Alexa into the shopping experience at Amazon Go stores . I have written in detail about this new modality I call Voice Commerce and Voice Payments . This is all an expansion of what I call the Voice First revolution.
Specimen: Amazon Go architectural side levations showing solar cell covered parking structures for order pickup.
An Amazon Voice Commerce transaction will begin simply by asking an Amazon Echo device to order the items for local pickup. This will include typical items found in the store including prepared food items for lunch and dinner. Amazon Go stores will have 8–10 parking spaces for the customer to park . The Amazon App will automatically detect the customer has arrived at the store and a clerk will either deliver to the driver side or load the items into the trunk.
Specimen: John Wanamaker’s location circa 1921.
Shopping Returns To Where It Began
When John Wanamaker invented the retail buying experience in 1861 at Oak Hall, at Sixth and Market Streets in Philadelphia, Pennsylvania he could not have imagined the arc it would take. History will record today Jeff Bezos, our century’s John Wanamaker, just about invented web commerce and now is reinventing retail commerce.
We have traveled so very far to come back to where it all started.
Specimen: Jeff Bezos displaying happiness.
 Brian Roemmele’s answer to Where does Brian Roemmele see Amazon moving next in payments (after building a ‘Square killer’)? and Brian Roemmele’s answer to Why does Amazon’s Alexa team need 1000 people?