Analytics at Amazon Speed: The New Normal

The world changed in February 2005 when Amazon Prime brought flat-rate, unlimited, two-day shipping into a world in which people expected to pay extra to receive packages in 4–6 business days. Since its launch, Amazon Prime has completely transformed the retail market, making low cost, predictable shipping an integral part of consumer expectations. This business model, which some have called the “on-demand economy,” is popping up in many industries and markets across the globe.

For example, some may remember places called video stores where people would rent movies for later viewing. Today, 65% of global respondents to a recent Nielsen survey watch video on demand (VOD), many of them daily. With VOD, a person’s desire to watch a movie is fulfilled within seconds. While not the market leader, Amazon participates in the VOD market with their Amazon Video service. Instant fulfillment of customer orders seems to be part of their business model. They have even brought that capability to information technology (IT).

About ten years ago, Amazon Web Services (AWS) began offering computing, storage and other IT infrastructure on an on-demand basis. No more waiting weeks or months for the IT department to order and install computing resources. Whether your need is for one server or thousands, whether you need them for hours, days or months, you only pay for what you use and they are available in just a few minutes.

Companies that successfully compete in the on-demand economy, in which we now all live, are going to need to learn how to deliver their products and services just like Amazon has done. In other words, at Amazon speed. What might be surprising to many is how the expectations of instant fulfillment are crossing over into other domains like data analytics. Like everything else in the on-demand economy, analytics is now expected to happen at Amazon speed and with Amazon predictability.

A typical example: the VP of Sales enters the office of the Chief Data Officer (CDO). She’d like to cross-reference the customer database with some third-party consumer data. The CDO asks for time to study the problem. Days later, he has planned the project. Resources will be allocated and configured, schemas will be updated, reports will be elegantly designed and the delivery pipeline thoroughly tested. The changes will take several weeks. “Not acceptable,” the VP of Sales fires back. The new analytics are needed for a meeting with the board later in the week. “The competition is ahead of us, we can’t wait weeks.” This scenario is being replayed in one form or another in corporations across the globe.

When did the requirement of fulfilling data analytics at Amazon speed cross over into corporate boardrooms? Consumer services like Amazon Prime may have had something to do with it, but even these are based upon radical evolution in the software industry. We’ll discuss the origin of these trends in our next blog.

Want to learn more? This blog was turned into a Webinar.

This blog series explores how data analytics teams can cope with delivering analytics at Amazon speed. We proceed by describing how we can apply best practices from lean manufacturing and software engineering to data analytics. We call this new approach DataOps. The series will finish with a simple plan for implementing DataOps in your enterprise. This blog series is based upon an article originally published in the Business Intelligence Journal.