The Future of Cloud is Clear, and its not all AWS
A few years ago, conventional wisdom seemed to be that, eventually, all workloads would migrate to the public cloud; more specifically that they would all move to the AWS public cloud.
In this blog, we predict a much different future. But first, let us review some cloud history.
Phase 1 — The Cautious Phase
The first phase of cloud started in 2006 with the launch of AWS. I call this first phase the cautious phase. Enterprises were slow to adopt the public cloud initially. Cloud capabilities were limited, cost was high for steady-state workloads, and there were many legitimate worries about lack of security and loss of control in the cloud.
As time went on, AWS was successful in signing up more and more customers that liked the idea of someone else taking care of their infrastructure needs. More providers began to offer public cloud services — Microsoft Azure launched in February 2010, IBM Public cloud in April 2011 and Google Cloud Platform in July 2012. Capabilities, cost and security all improved steadily during this first phase. Many organizations liked the idea of “pay only for what you use” and of operational expense versus capital expense. In 2009, following a data center failure that shut their service down, Netflix began a much-publicized effort to move all its apps to the public cloud. They re-architected and re-wrote all of their apps to run on AWS, a monumental migration effort that took 7 years, completing in 2016. At more or less the same time, in December 2010, the CIO of the United States issued a “cloud first” mandate to all the Federal agencies. This set off a public cloud mania and pushed the industry towards its second phase.
Phase 2 — The Irrational Exuberance Phase
As the public cloud improved and more and companies adopted a cloud first mandate, we entered the second phase of cloud computing which I call the irrational exuberance phase. Driven by the success of the early adopters, customers went from being cautious about using the public cloud, to being overly aggressive and indiscriminate about using the public cloud for all their applications. In this phase, those who did not adopt the public cloud were viewed as laggards who lacked in innovation. There was intense pressure on IT organizations to move their apps to the public cloud and shut down on-premises data centers. GE, for example, is reported to have shut down 30 data centers around the world as it moved to AWS.
Some vendors attempted to counter this trend by delivering a private cloud — a cloud that you ran inside your own data centers, and provided a cloud-like experience. The idea was to retain the security and control advantages of being in your data center, while delivering cloud-like simplicity.
These first attempts at private clouds consisted of a cloud software stack that included software-defined compute, software-defined storage, software-defined networking, infrastructure management, and cloud management with intelligent operations and automation. For example, the VMware cloud software stack consists of vSphere for software-defined compute, vSAN for storage, NSX for networking, SDDC manager for infrastructure management, and vRealize Suite for cloud operations and automation. Similar cloud software was available from OpenStack, an open-source software platform. A customer would install this software on hardware that they purchased separately.
These initial forays into private clouds were unable to match the simplicity and ease of use of the public clouds. Furthermore, VMware private clouds were expensive, and OpenStack private clouds found to need a large number of administrators to deploy and run. As a result, they did not slow down the rapid adoption of the public cloud by many customers.
Phase 3 — The Mature Phase
In this third and current phase of cloud development, a second generation of private clouds began to be available (e.g. Microsoft Azure Stack, Lenovo ThinkAgile CP powered by Cloudistics). Every public cloud vendor now has their own private cloud offering — even Amazon recently announced AWS Outposts, though it will not be really available till late in 2019.
Several of these are easy to deploy appliances that can be upgraded and managed remotely by generalists without any training. They support cloud-native applications, modern constructs like containers and serverless, and federated orchestration between the private and the public cloud. In some cases, they even allow for “pay by use” rather than pay upfront. In other words, they really can provide you a true public cloud like experience, but behind your own firewall. Many studies have conclusively shown that for steady-state workloads whose resource needs are predictable, a private cloud is cheaper than a public cloud.
At the same time as the rise of private cloud 2.0, the industry also began to learn the value of being more discriminating about which apps should be moved to the public cloud and which apps should remain in their own data centers on these modern private clouds. They learnt that there are both technical considerations (such as workload elasticity, performance, security and volume of data) and business considerations (such as capex vs. opex preference, cloud expertise in the company, compliance/data policies and so on) that come into play in making this choice of where to run an application.
As a result of these developments and the maturation of the industry, we are seeing customers make much more informed decisions about when to use the public cloud and when not to. We are also seeing customers that had previously moved many applications to the public cloud during the irrational exuberance phase now start to move some of them back to private clouds in their own data centers. 451 predicts that by 2020, 38% of applications will run on private clouds while 30% of them will run on public clouds. IDC similarly predicts that almost 1/2 the apps that were moved to the public cloud during the irrational exuberance phase will be repatriated back to the private cloud in the period from 2018–2020.
The Future — Fewer Clouds, More cross-cloud services
While the 1st phase was cautious about what goes to the public cloud and the 2nd phase was overly aggressive in this regard, we believe the future will be balanced.
We are entering a phase where customers understand that they will need private clouds (possibly more than one) and multiple public clouds. When IBM announced their acquisition of Red Hat a few weeks ago, their executives talked about how some of their customers had as many as 16 different clouds, and how use of at least 5 different clouds by their customers was not uncommon. Similarly, IDC reports that only 24% of customers have a single cloud, the remaining have multiple clouds they are dealing with.
Here are our key predictions for 2019 and beyond, in the evolving cloud journey
1. Customers will use multiple clouds for the foreseeable future, both private and public.
2. Fewer than 50% of all applications in the world will ever run on public clouds. The rest will run on private clouds and edge clouds.
3. Only about 5 viable general-purpose public cloud vendors will survive. This is because successful public cloud vendors will need to spend a lot of money, and few can afford to spend as much as the Top 2 — AWS and Microsoft Azure.
4. There will also not be very many viable private cloud options. Building all the software for a full private cloud stack (such as VMware) is non-trivial, not many will do it successfully.
5. There will be big opportunities in cross-cloud software and services — services that work seamlessly across multiple private and public clouds. Examples include monitoring services (e.g. New Relic, Prometheus) backup and DR services (e.g. Veeam), container management and orchestration services (e.g. Red Hat OpenShift, Pivotal PKS), and the like.
6. Public and private clouds will be much more accommodating of each other. More and more of the services provided by a public cloud vendor, such as their AI services, will become accessible to apps running elsewhere, including on private clouds. At the same time, there will be more and more examples of private cloud capabilities extended to the public cloud — such as VMware Cloud on AWS. Finally, federated orchestration and management of workloads across private and public clouds, all from a single, easy to use, portal will be commonplace.
7. The two most successful programming paradigms of the future will be microservices (which structures an application as a collection of loosely-coupled services) and serverless (where applications are structured as a collection of event-driven actions).
