How Amazon Came to Control Much of the Internet
Did we learn nothing from Die Hard 4?
A new breed of mysterious concrete building is mushrooming throughout the globe. In Manassas, Virginia sits a vast, windowless structure which looks like a modernist warehouse and operates under the name “Vandelay Industries”. On the other side of the Atlantic, a similarly cavernous facility is under 24-hour surveillance by the security company G4S in an industrial suburb of Dublin.
So what are these strange buildings? A Wikileaks data dump published in October 2018 revealed that they have something important in common — they are both giant data centres secretly operated by Amazon Web Services (AWS).
Most people probably think of Amazon as an online retailer, the maker of the Kindle e-book, or perhaps a video streaming service. In fact, however, Amazon’s most profitable division receives the least publicity — AWS now represents about half of the tech giant’s operating income.
Like many of the greatest tech products, the development of AWS owed as much to entrepreneurial serendipity as it did to careful corporate strategy. At the turn of the millenium Amazon was a very different business to the one which exists today: an e-commerce company struggling to cope with the pace of its own growth.
Initially, a team at the firm simply wanted to create a development platform to enable third-party, bricks-and-mortar retailers to create online shops using the same back-end technology that powers Amazon. However, this proved more difficult than expected because the underlying codebase had not been written with third-party access in mind. In addition, managers noted with frustration that many development projects were taking much longer than expected due to duplication of effort — different programmers in various divisions of the company writing similar code to do things like build a database or manage storage.
This led Amazon to change their procedures in order to focus on establishing a set of common infrastructure services to help their staff work more efficiently. Without realising it, they had created the seed that would develop into Amazon’s most profitable division.
AWS officially launched in 2002. In its first incarnation, it merely provided a simple way for developers to access the Amazon product catalog in order to place affiliate links on their websites and earn referral revenue. However, over time, the full potential of the platform became clearer. In 2006, AWS relaunched, offering not just file storage, but also cloud computing services. What had started out as the solution to an internal software development problem grew to become a quasi operating system for the internet. By 2018, AWS alone was a business worth $25.7 billion a year in revenue. In fact, this single division generated more revenue than chip-maker Qualcomm or fast-food giant McDonald’s in the last fiscal year.
All eggs in one basket?
But all this success has a downside. The internet ecosystem is increasingly reliant on the cloud and recent figures show that 41.5% of cloud computing is controlled by Amazon alone, whose share of cloud application workload is more that Microsoft Azure, Google and IBM combined. Without AWS, many web services that we take for granted would cease to function. Household names like Netflix, Uber, Expedia, Yelp, and Pinterest all run on AWS. Even tech giants like Apple heavily rely on AWS as the underlying infrastructure of its iCloud system. It is also currently the frontrunner in a two-horse race with Microsoft to win a $10 billion contract to supply the Pentagon with a cloud computing network. The influence of the platform has become so enormous that it is becoming difficult to see where the internet stops and AWS begins.
If you take a look at the map of infrastructure on its website, it is obvious how it works. Although (unlike Wikileaks) it is vague on the exact location of assets, the map shows a huge interconnected web of data centers spanning every major population center in the developed world. This provides the economies of scale to drive costs down while providing enough redundancy to make the system reliable — if you are in the UK trying to watch Netflix in Leeds and the local datacenter has a problem, the system will seamlessly switch over to the datacenter in London.
The huge scale of the investment required is why few companies outside of the tech titans can afford to run their business purely on in-house servers. It would seem that increasing centralisation of both data and computational power (and the formation of businesses with close to monopoly power) is the natural direction of travel of the current system.
John McClane can’t actually stop anything
This may explain the need for all the secrecy and security. Just as AWS’s success was to some extent unforeseen, its meteoric rise poses structural risks that could not have been imagined when the project started out. An increasing consensus has formed among policymakers and defense specialists that the internet now forms a part of our critical infrastructure which is as important as bridges, roads or power stations. The increasing centralization of server capacity and computing power provides a central target for bad actors such as cyber criminals, foreign intelligence agencies, and terrorists.
But could there be a way to ensure the same ease of use and reliability, without needing to centralise everything in one proprietary cloud? At Catalyst, this is exactly the network we are building — the mission that drives us every day. This involves going back to first principles and reimagining what decentralisation the internet can be. If we move away from a rigid client-server model towards a system with hundreds of thousands of nodes which collectively form a distributed network, we can create viable decentralisation in the internet. But we don’t just want to create an interesting sandbox or an intriguing proof of concept — we are steadfastly focussed on building a system that is fit for deployment in the real world. Visit our page catalystnet.org to learn more.
