Deciphering AWS Reserved Instance Billing

Would you like a side of Lambdas with that? Photo Credit: Olivia Dawd

When you’re at fast growing startup it is easy to forget about your COGS (cost of goods sold, i.e. all those EC2 instances you provisioned last week) because you are laser focus on building and shipping product. Your sales team is bringing in new customers, your customer success team is up-selling like crazy because of all the cool new features you built, and you just raised a boatload of money from the VC shop down the block. So what if you’re spending an extra few thousand dollars on AWS each month?

This past January I returned from a combo winter-break + honeymoon super refreshed and ready to tackle whatever came next. Over the years we were always a “scale at all costs” organization but were at the point where we need to scale and care about costs. Fast forward another couple months and COVID forced our hand and accelerated the urgency of this work even more so.

I approached what I started calling the “costs project” with excitement. I broke the project down into 5 steps:

1. Understand where every $ goes
2. Figure out why the $ goes there
3. Decide the value that $ brings
4. Evaluate technical and/or architectural change to reduce the $
5. Make changes

There is a ton of interesting work that happened through this project (cue future blog posts) but today I want to focus purely on one particular AWS feature and all the hiccups it caused along the way: Reserved Instances.

What is a Reserved Instance?

A Reserved Instance (RI) is a type of AWS instance that provide two main benefits to the user:

1. Reserve capacity of a particular type of instance. As “infinitely scalable” as AWS is, they sometimes run out of capacity of various instance types. By buying an RI you get guaranteed availability for what you bought.
2. Saving $$. RIs can give you a discount up to 72% off the standard on-demand pricing of an instance depending on the length of the contract (1 or 3 year), the type of the contract (standard or convertible), and when you pay (all upfront, partial upfront, or no upfront).

Buying an RI is like committing to a certain number of EC2 Instances per hour for a given time period. Under the hood you can think of it as a $/hr commitment, particularly if you have a convertible instance as you can move that $/hr from one instance type to another.

The biggest discount is a 3 year standard all upfront RI. In our case we had bought a ton of 1 year convertible no upfront RIs. To break that down:

• We were in contract to pay the total dollar amount for an entire year (1 year)
• If we wanted to change the instance type (ex: from a r5a.4xlarge to c5.2xlarge) we could do that (convertible)
• We paid in monthly installments (no upfront)

For the most part we were happy with this decision, so happy that year after year we renewed the RI contracts and added more and more as we scaled. We had a fairly predictable set of EC2 instances we provisioned and enjoyed the discount and comfort of knowing we had the specific machines available for us when we needed them.

AWS Billing Console

As we started up the costs project one of the first new pieces of software that entered my peripheral was the AWS Billing Console. If I wanted to understand where every $ goes, this was the tool that would help me figure it out. The billing console is incredibly powerful and allows you to slice and dice your billing data in endless ways. Luckily, we had previously spent time tagging our resources with useful tags to help drill down our spending as we saw fit (cost center, customer, role, etc). Having the ability to break down an AWS resource (such as an S3 Bucket or an EC2 Instance) by one of these tags, or multiple at once, was instrumental in the transition from understanding where every $ goes to understanding why it went there. Suddenly I could allocate a $5/month charge to a particular cost center and for a particular part of our product.

S3 was surprisingly simple to explore and understand. Look at every bucket, make sure we knew why we had each bucket, make sure we had a good expiration policy on each bucket (hint: we did not) and fix up anything that didn’t look right.

EC2 on the other hand was a bit more confusing, in particular the “EC2-Instances” service (there are a few EC2 related services in the billing console including EC2-ELB and EC2-Other).

The first question I asked was “How much do we spend on development?” As I dug in I was surprised that there were instance types we used some days but not every day. That turquoise and red on the graph above, why didn’t we use those instances every day? I was less concerned with the ups and downs of our bills as I could hand wave those away when thinking about our development practices (weekends were always lower, more instances were up and running in the middle of the week).

Let’s filter down to the instance type represented by the red:

Spending for a particular instance over a 30 day period

I had enough architectural knowledge of our platform to know that this was an instance type almost every development cluster provisioned; it was a standard instance type. I found it weird that one week had incredibly high usage while other weeks had close to no usage, some with actually no usage.

Costs ($) vs Usage (Hrs)

Those familiar with the AWS Billing Console may notice that the two graphs above are Costs ($) graphs. Cost graphs are what you’ll see every time you use the billing console, no matter what. It is the big, generally colorful, main graphic you always see on the screen.

AWS doesn’t only tell you how much you spent, it will also provide you with how many units you used, but only if you know how to ask for it. When you are viewing a cost graph, like the one above, you may inadvertently be merging together a bunch of different usage types which are charged by different units. When that is the case AWS does not provide you the units (the usage). For example, S3 Storage charges by ByteHours but S3 Requests are charged by Requests. If you start by filtering on the S3 Service you’ll never get the usage graph because you have multiple usage units mixed together. You need to pick a subset of S3 usage types, such as TimedStorage-ByteHrs and TimedStorage-GlacierByteHrs which share the same usage unit, GB-Month, in order to get a usage graph.

Looking back the EC2 example above, if we modify the filters to a specific set of Usage Types (in this case BoxUsage) which all have a shared unit of measurement we’ll see a bit more information:

What…….why is that so different? The bottom graph, Usage (Hrs), looks exactly as I would have expected our development usage for this specific instance type to be; fairly consistent with some ups and downs. Why weren’t we getting charged to use these boxes for many of the days?

Reserved Instances ❤️ AWS Billing

When you buy a Reserved Instance you aren’t buying a specific instance (e.g. instance id: 190hs01r23). You are instead buying “some” instance of that type. Let’s imagine you get a single r5a.4xlarge RI. Monday morning you wake up, spin up a new fancy r5a.4xlarge instance. Tuesday you wake up and terminate Monday’s instance and spin up a new r5a.4xlarge. Each of these days that specific r5a.4xlarge you had will be attributed to the single Reserved Instance you bought as they both match the specific instance type even though they are different actual instances. This abstraction applies across any tags you use to manage your billing. Cost center? RIs don’t care. Customer? RIs also don’t care. RIs are even shared across accounts within your organization. It turns out that the “shared discount” the RI comes with has a mind of its own and get applied to what they want.

So how do Reserved Instances affect your billing? I’m going to focus on no upfront convertible instances since that is what we have. You should be able to apply similar logic to the other types as well.

1. The RI monthly cost is applied on the 1st of the month and amortized through the rest of the month, 1 hour at a time. You can see the initial charge by looking for HeavyUsage usage in the billing console. The usage is attributed to the specific instance type you bought, just in case you want to filter on instance type. These items will not get any of your custom tags (cost center, customer, etc) because they are not allocated to a specific AWS EC2 instance.
2. If you convert any RIs in the middle of the month you’ll see a new HeavyUsage charge on that day representing the remaining spend for the month. The charge from (1) will be retroactively updated and decreased by the amount of this new charge.
3. When looking at BoxUsage (like the above graphs) you’ll see actual usage hours (the bottom graph) with the cost (top graph) being somewhere between $0 and the on demand $ for those boxes depending on when AWS decided to allocate the RI spend towards that instance type. Note: shared tenancy RIs are shared across an entire instance family. One 24xlarge r5a machine may get applied to 6 4xlarge r5a machines within the billing console. Dedicated tenancy machines must match the size perfectly.
4. Finally, on your bill at the end of the month you’ll see “abc.2xlarge reserved instance applied, abc.4xlarge instance used” to make it clear which RIs you paid for and which boxes they got applied to.

Can you guess why the graph above looks so funky now? We had RIs for that instance family which we bought for our predictable static set of production resources. AWS decided to apply those RIs to our dev cost center instead. When I initially looked at our production costs I was surprised to see daily spend even though we had an RI! Mentally I was allocating the RI to production but AWS wasn’t.

If your organization uses Reserved Instances, knowing these facts will be vital to diving in and understanding where money is being spent on AWS. In addition, as you scale and try to decide if you need Reserved Instances, it is very important to understand how they will get applied to your resources and what flexibility you have. For me, when I look at EC2 usage, I focus on hours and apply the on demand and/or reserved instance price based on where I expect the RIs to be applied in order to get a better idea of our spend. It is also important to compare Reserved Instances with Savings Plan and where applicable, Spot Instances, to better manage your costs.

Please look out for future posts on some of the work we did to reduce our AWS costs and feel free to shoot me an email if you have any Reserved Instance billing questions.