Amazon Web Services — infra at scale
This article from late 2014 tries to estimate the overall scale of AWS infrastructure (upfront disclaimer: I really really really love AWS). Quite impressive numbers, especially the following estimate in the article:
So, the answer is that AWS probably has somewhere between 2.8 million and 5.6 million servers across its infrastructure.
Does it sound plausible at all? Let’s play with the numbers a bit more — starting with a peek at Amazon 2015 Annual Report.
- “… 33 Availability Zones across 12 geographic regions worldwide …“
- “… now is used by more than a million customers …”
- 2015 net AWS sales of $7.8B
- 2015 AWS operating expenses of $6.0B
- $4.6B spent on AWS “property and equipment” in 2015 (1)
- Reported AWS “assets” at the end of 2015–$8.3B (1)
It’s interesting that during 2015 AWS invested equivalent of more than half (55%) of total reported AWS assets at the end of the year. That seems to be reasonably in line with 70-80%+ YoY growth and increasing capacity and corresponding infrastructure accordingly. Note that Amazon defines useful life of a server for amortisation purposes as 3 years (5 years for network equipment).
I’m making an assumption that majority of data center costs have already been incurred beforehand and therefore reported capex spend is mostly on equipment — both replacing out of date equipment and adding to the overall capacity. Buildings are relatively cheap and are amortised over a longer period of time (40 years in Amazon case), servers and networking/storage gear are dense, expensive and expire or become uneconomical to run required workloads quickly.
Let’s round the total number of servers at the end of 2015 to 3 million (erring on the conservative side, see note 3), make a guess that 50% of servers got commissioned (new) or replaced during the year (erring on the ambitious side, see note 4) and see how it all comes together:
- Per server sales: $2'600 per year ($7.8B / 3M) — or $7800 over its expected lifetime
- Average per server cost: $3'066 ($4.6B / 1.5M)
- 4'109 new servers per day, every day
Of course the numbers above are a pure guesswork just to look at the scale, not arrive at the actual actual costs. I’m looking at the overall budget, interpreting the reported numbers with zero accounting experience, mixing in a few numbers completely out of the thin air, ignoring the capex portion of commissioning new datacenters that might not yet be online (not to mention the investment in hundreds of miles of fibre, power lines and stations, cooling, roads, etc) and simplifying it to an arbitrary server which might range from a ZX80 clone to 512 port custom asic network gear in practice.
Still, cost of circa $3'000 on average per custom built server at a guaranteed order of 4 thousand units per day sound quite realistic. Even more so when starting to think about split into different CPU, network and storage instances and their relative costs and deployment ratios.
I just wonder where they build them. Logistics chain to support their commissioning must be quite a feat too (5).
- Annual Report page 70. Given the accounting notes voodoo, the numbers might warrant a long and close look with the help of previous annual reports, good scotch and an accountant.
- As above, caveat emptor.
- Amazon engineers have claimed that each availability zone consists of at least one data center and contains on average circa 50'000 servers. Most online estimates claim 2 data centers per AZ on average.
- A 3 year amortisation cycle implies that circa 33% of existing servers will have to be replaced each year just to keep the number of servers constant. Given the advances in processing power, density and storage, I am assuming that majority of increased capacity comes from replacing existing servers with more performant hardware.
- In the calculations I’m assuming that costs of shipping, installation and the likes are part of opex — therefore not looked at for the purpose of this article.