7 Weird Tips To Reduce Your AWS Bill

8 min readApr 15, 2016

I recently attended a very insightful AWS Meet Up where Paul Wakeford (a Systems Architect at Australian media company Fairfax Media) shared some great tips to identify costs and reduce your AWS bill.

With Paul’s permission I’m sharing, in his own words, his views on a subject very close to our heart here at GorillaStack: how to go about making your AWS spend more efficient.

Context

Fairfax has around 30 AWS accounts, 800 to 1100 instances, and a growing seven digit $ annual AWS bill.

The graph above shows Fairfax’s daily EC2 instance hours. From May 2013 to December 2015 the growth is about 7x.

And here’s the bill growth:

They realised the extent of their billing issues in early 2015 and started to take steps to limit increases then. You can see they peaked in September last year and have been dropping back since then, while still seeing growth in usage. This is what we mean by “efficient” — AWS consumption increases while cost decreases.

Why AWS billing matters

As techies you may be thinking ‘why bother? Isn’t billing a bit boring? And why should I care?`

The straight-forward answer to those questions is that someone has to pay. You may be a private user or a small business — the money is coming right out of your pocket — money that can be reinvested in the company for a bigger office, employ someone to do the the bad jobs, or just to buy cool toys.

If you work for Global MegaCorp then you could be a hero — having the reputation as someone who brings in projects under budget is not a bad thing. Or maybe you can buy cool toys / tools with the saved money — maybe better monitoring or alerting tools, New Relic etc.

And because we all want to play with cool things, AWS is a cool thing. Plus ‘management’ has been told many times that “cloud is cheaper”. Do you really want to go back to on-premise or VMWare? And even worse… risk lose all your cool AWS toys?

Part of architecting solutions is being cost effective — if you do any of the advanced AWS certs — Pro SA or DevOps Pro you will find there are plenty of “choose the most cost effective option” questions.

So Paul described his top tips to reduce your AWS bill. I prefer to call them ‘best practice’. Here they are:

1. Tag Everything

You need to be able to slice and dice your bill so you can identify high cost areas. Tagging is vital for this.

Tags are labels — a key / value pair you can apply to most AWS resources — EC2 instances, EBS volumes, S3 buckets etc. Use tags everywhere. Within the AWS console you can have up to ten tags — and don’t forget that Tags are case sensitive.

Fairfax uses Project to identify resources for cross charging. You may want to tag front end web servers, app servers, Varnish servers, mobile API servers, etc.

Come up with your own tags based on your use cases — the important thing is to have a standard and stick to it.

Tag support in the AWS console is much improved — you can edit tags, find resources that are not tagged correctly, and create groups of tagged resources. And of course the CLI fully supports tagging.

There are third party tools to manage tagging too — Paul has contributed to Graffiti Monkey, a tag inheritance tool for EBS volumes and snapshots, which is on Github.

He also noted that GorillaStack has open sourced tools that auto-tag AWS instances at the point of creation (hat tip).

Tags are visible in the billing console and also carried through to your detailed billing file.

You should also enable all Billing Reports in your billing console, and choose which tags to show in the reports. You won’t want them all. Your billing file is then built up during the month (not quite real-time, there is a delay) as a CSV file in your nominated bucket.

2. Measure The Baseline, Monitor, Alert On Exceptions

Baseline metrics — as a business would typically do for performance metrics, do for costs. You need to know what is normal. Then measure exceptions and rates of change.

This example is for a Fairfax CMS project using Cost Explorer in the AWS billing console. You can see the tags on right.

In this example you can see the visible effect of purchasing reserved instances in September onwards, then some expiring in Feb 2016. Also note that ELB costs are now assigned due to better tagging, and a S3 reduction due to better asset management.

You should also set up billing alerts, regardless of what size organization you are. You don’t want to be another story on reddit where some API keys were lost and used to mine bitcoins.

These billing alerts are for the whole account only, however you can also set budgets with alerts and they can be set per tag.

All GorillaStack plans offers a series of options to set customizable periodic and threshold based AWS Alerts that push messages through to Slack, Hipchat, email and SMS.

3. Establish A Responsibility Model

Ensure costs are known at every level, from beginning to end, from project owner to engineer.

Ensure project owners / budget holders know what their costs are up front and ongoing.

Ideally have a pair of people resposible for AWS cost management.

When Fairfax does a design they include a bill of materials which the project owner has to sign off to say they accept the costs. The project is then assigned a code which is visible in the AWS CSV reports. That allows them to cross charge the project and compare it to projections.

You should also make cost control a KPI for team leaders and engineers. How you do it is up to you but having something in performance reviews, bonus reviews or salary reviews helps keep costs in focus.

You can also give business owners access to AWS Billing Reports or to third party tools or an internal AWS cost dashboard.

4. Reduce Usage

There are bunch of tools you can use to make your usage more efficient, including:

Third party tools such as GorillaStack.
AWS tools — Autoscale scheduling. Read more here about why GorillaStack’s Scheduled Auto Scaling aka Power Scale is so powerful when compared to out of the box auto scaling groups.
Open source tools from Fairfax — such as CloudCycler / FlyWheel.
Cost section of Trusted Advisor (which requires a support plan).

Go through your account and billing files carefully — find unused ELBs, unattached EIPs, old DynamoDB tables (maybe from EMR use) and so on.

5. Design For Cost

Consider using cheaper alternatives such as containers to drive up utilization and for on socket provisioning.

Here (above) is a Spot / EC2 example. You can register two AS groups behind an ELB. In this case Fairfax used t2.small instances for base load and use the CPU Credit Cloudwatch metric as the basis for adding capacity to the Spot instance group.

Another question to ask is ‘do you even need to run instances?’.

Fairfax had a statistical tracking solution: when a user clicked on a link, that click was sent to a HA nginx setup and then stored in a MongoDB cluster. This could all be replaced with a serverless option at a cost one fifth the size of the EC2 solution.

Another example: a developer asked for an EC2 instance to host a solution in production. So that would mean 2 instances. Dig further — publishing microservice — use Docker? Dig further — NodeJS app — use Lambda?

6. Use Accounting Tricks

Use consolidated billing with multiple accounts — one bill, and volume usage discounts, e.g. S3. RIs are active across consolidated accounts.

For assets that don’t need low latency consider other (cheaper) regions (e.g. Cloudfront from US only for zip files).

RIs for base load. Your DBR can be analysed in Redshift and Tableau to produce visualizations like this (above). If you have enterprise support your TAM should be doing this for you, or you can use Trusted Advisor or another third party tool.

7. Review And Restart

Unlike physical servers it is very easy for AWS resources to be accidentally hidden at any time. This is not a one-off task — it’s an ongoing process and commitment, and one where you definitely get out what you put in.

Get everyone involved.

Credits:

Paul Wakeford, for his permission in sharing the above. Thanks Paul — keep up the good work and evangelism.

Gary Capps and Darrell King, from PolarSeven, for their continued organization and leadership for the Sydney AWS Meetup.