Releasing New Analytics Every Second

Analytics at Amazon Speed: Part II

In this blog series, we explore data analytics in the on-demand economy. Companies like Amazon and Google have turned instant fulfillment into competitive advantages and are being rewarded in the marketplace. As consumers adapt to this “new normal,” the expectation of instant delivery is crossing into other domains. For example, data analytics users can’t or won’t wait weeks or months for new analytics. In this installment, we discuss the origin of this new mindset. This blog is part 2 of a 4-part series.

Over the past decades, the average frequency of software releases by development organizations has dropped from years to months to weeks to seconds. This incredible improvement stems from evolutionary changes in software development tools and methodologies. In the early days of computing, most large development organizations used the waterfall model of project planning, adapted from the manufacturing and construction industries, which required detailed planning and highly structured project management. Projects progressed through the phases of conception, initiation, analysis, design, construction, testing, production/implementation and maintenance sequentially with little ability to change course midway. Using the waterfall model, the average frequency of software releases in the 1980’s was about 12 months. The waterfall model is best suited to projects where the requirements are well understood in advance. The model is less effective, however, in fast-changing industries where requirements rapidly evolve.

In the late 1990’s, software development organizations began to abandon the waterfall method in favor of a new methodology now called Agile Development. Agile throws away the old method of project planning with its detailed, up-front project plans and bureaucratic, sequential phases. Companies began to organize code development in short iterations, producing valuable incremental changes in short periods of time. Customers or users provided immediate feedback on each release, which then was used to influence future priorities. Agile has revolutionized software development, increasing the speed and productivity of programmers significantly. Using the Agile methodology and principles, the average release frequency dropped to about 3 weeks in the 2000’s.

Average software development release frequency over time. Source: DataKitchen, 2017

The Leap to Continuous Delivery

More recently, development took another giant leap forward. Agile development evolved into DevOps, which includes additional process and tools that have improved the average delivery frequency of software projects from weeks to days and even to seconds.

Before on-demand cloud services were available, each of the different groups in a software development organization had to request that IT purchase and configure servers, networks, storage, software and any other components needed to run an application. In those days, groups such as software development, testing, and quality assurance (QA) could each be running different environments. Customers could run in yet another environment. If a test group discovered a problem, it might not be replicable in the development group due to differences in the environments run by the two groups. This often led to a lack of trust and communication between different members of the organization. It also fostered misunderstandings and delays.

With on-demand cloud services and the inexpensive provisioning of compute resources on-demand (infrastructure as code), organizations have been able to write code that defines the run-time environment. No need for an IT person to spend time setting up a physical server. In a matter of minutes, a script can run that provisions a cloud server resource and a software environment all tailored for a particular application. Development, test, QA and others can now easily create the same environment. This breaks down barriers between the groups, leading to much better trust and collaboration. With IT operations functions now handled by code, operations functions are merged with software development. This combination of development and operations is how the term DevOps originated. The unification of these functions on a common platform (the cloud) has enabled companies to implement continuous delivery. They deploy new releases to customers many times a day; sometimes even faster than once per second (and getting faster).

Amazon Embraces Continuous Delivery

Amazon moved to web services and then opened up those services to be shared between development teams. This allowed them to move quickly to continuous delivery for new features and capacity. When this article was first written, several months ago, Amazon Web Services (AWS) was releasing improvements to its platform every 11 seconds. That has improved to several times per second. By the time you read this, it could be even faster. AWS is Amazon’s most profitable division and has become a foundation upon which the services and subscription-based economy is being built.

AWS developers perform thousands of software changes per day. The software is tested internally and then deployed to customers on a rapid and continuous basis. As you read this, thousands of servers are being updated to deliver new functionalities to be deployed across the range of Amazon services. This efficiency, adaptability and customer focus allowed AWS to consistently reduce prices, diminishing margins for competitors, and increasing AWS market share. According to Gartner, AWS is 10x bigger than its next 14 cloud competitors combined. More importantly, by providing compute resources that can be quickly provisioned to match demand, AWS has enabled numerous organizations to become much more nimble.

Companies like Amazon have become extremely large, yet remain nimble by adhering to Agile methods, DevOps and continuous delivery of software. They have shown that the way to grow in the on-demand economy is to apply a philosophy of continuous improvement to products and services. Companies that can follow suit will thrive in the new economy.

This blog series explores how data analytics teams can cope with delivering analytics at Amazon speed. We call this new approach DataOps. In our next installments, we describe DataOps and provide a simple plan for implementing DataOps in your enterprise.