Take Control of your Hosting Costs
“Heads up,” Rick, one of our system administrators said to me, “our AWS bill has grown, and we are trying to identify why.” In fact, I had already started an investigation myself. Both Rick and I checked the forecasted budget numbers for Amazon Web Services (AWS) on a daily basis to make sure we caught issues early, and that was about to pay off. I had noticed that February’s bill was significantly higher but contributed it to extra resources we spun up for a project. The resources were now shut off, but our bill continued to climb… It was time to take immediate action.
Although the incident was costly, I was happy with the outcome. We caught the trend early, took quick action to fix the issues, and we identified some configurations that ended up saving us a lot of money in the long run. We are at record lows for hosting costs and saving more than 40% compared to the same costs a few years ago.
There are a lot of great articles on how to cut costs and AWS themselves have a lot of great resources that help you identify ways to save. Even our representatives at AWS actively participated in helping us reduce the amount of spend. There are many articles that talk about rightsizing, using reserved instances or spot instances, the use of autoscaling, and many more ways of saving while in AWS. I would love to share some of the specifics on how we’ve saved, and will later in this article. Before I do that, I want to share some true measures we’ve taken in controlling and driving down costs that can be applied to any environment, and I haven’t found an article that talks about these steps yet.
Great minds think alike!
Or do they? There are a lot of great minds at Learning.com, but we don’t think alike. That’s one of the things that makes us so great. When we tackle a problem, we come together, brainstorm, troubleshoot, and work as a team to tackle the issue at hand. We each come with a different perspective. We don’t always agree, but working as a team, we think of a lot of great solutions and implement the best ones. Why not extend the teamwork philosophy into driving down costs? Bottom line: equip your employees to make good financial decisions and help develop fiscally responsible solutions.
Although there are many ways we do this here, one step shines above the rest. Every month we hold a meeting to talk about our spend and ways we might be able to save. We gather system administrators, development managers, architects, and database administrators all in a room and look at our bill. Throughout the meeting we sort our expenses by many different aspects to include AWS services, Learning.com applications, monthly trends and daily expenses to get a better perspective on the data. At any point, meeting attendees can ask to drill down further into costs to see what resources we are spending our money on. As a result, we’ve identified many ways we could cut costs. Solutions included rewriting code, dropping AWS services, and making architectural changes to our databases. Many times there are many costs associated with making the changes, but the month to month saving benefits have been amazing.
Think Consistency
Anyone can make a one time change that can save a quick buck. The question is, how do you continuously drive costs down?… Through good habits and routines. Having a monthly meeting helped reduce the cost of our AWS bill. We also are consistently auditing our environment to ensure we don’t have orphaned resources that are no longer needed. We do daily checks on our projected monthly hosting spend, monthly trending reports, and yearly licensing audits. Creating good habits ensures there is not ongoing waste, helps identify runaway costs before they get too large, and creates a mindfulness around cost cutting.
Keep a pulse on changes
Similar to consistently looking for ways to cut costs within your environment, It’s also important to know about changes by your hosting provider. AWS is an ever-changing marketplace and in continual development. I found that even 3rd party consultants sometimes weren’t up to speed on the latest and greatest cost cutting tips. Attend conferences, watch webinars, read blog posts, consult with experts, join user groups, and talk with your hosting representative. In other words, always be learning and growing.
Always think Return on Investment (ROI)
Our VP of Technology quite often will ask, does this “move the needle”. At times, I find myself focusing on things that really won’t make a big impact. At one time, there was talk about cleaning up an S3 resource to cut on storage costs. The process of identifying and archiving the old data in the bucket would have taken up time and generated transfer costs that would have exceeded how much we would have saved. I had to set aside my perfectionist tendencies, delay the cleanup task, and focus on something that would “move the needle”.
Ultimately, this comes down to prioritization. Get all your cost saving tactics on your project board and organize your to do list based upon ROI. Consider the time and resource cost that will go into each endeavor and know what result you expect in the end. Continually ask yourself, will this “move the needle”.
The Specifics:
So now you know the tactics of what has made us successful in cutting costs. Here are some specific steps we’ve taken.
Hosting in AWS: The move to AWS was by far the best move in cutting costs and improving our service response times than any other move that we have made. I am not saying AWS is the best move for everyone, but after assessing our options, it was clear: AWS was the right spot for us. Our experience with AWS has shown that we are getting better services for less than the hosting companies we’ve used in the past.
Rightsizing: Cluster sizes, instances sizes, allocated IOPS, and the amount of resources we consume is consistently being monitored, and we are making adjustments as necessary.
Reserved Instances (RI): There are huge savings by using RIs. We’ve standardized on several instance types to make purchasing of RIs easier. It’s not as critical with Linux machines to pinpoint an instance size because a larger RI can accommodate running of multiple smaller instances.
S3 Endpoints: Many of our production EC2 instances were pulling resources from S3 buckets. That data being pulled was going through our NAT gateway, which in turn incurred transfer costs. We created an S3 endpoint so that machines would bypass the NAT gateway and traffic is sent directly to the S3 buckets. Result=huge savings!
Bring your own licensing (BYOL): SQL Server licensing is expensive no matter how you pay for it, but why pay more? In the long run, we pay less by purchasing our SQL Server licensing rather than using the AWS EC2 instances with SQL Server licensing.
Auto off script: We have a lambda function that turns off any EC2 instances that have the tag ‘Auto Off = True’. At nights, weekends, and holidays, machines that aren’t needed are shut off.
Combining resources: Sometimes, we put multiple services on the same resource to cut costs. One method we use is by running instances with Docker and running our services in containers.
Lifecycle rules: Get that old data out of there!!! We switched on S3 bucket versioning so we could enable cross-region replication. Our backup files in the S3 bucket were still being overwritten, but now it was keeping all prior versions of backups. Our S3 bucket grew more than 100 Terabytes in a very short amount of time. This became costly very quickly. We’ve enabled lifecycle rules on our buckets, and now things are much more reasonable.
Running your own instance vs paying for a service: There are times when it pays to spin up an EC2 instance and install a service. Other times, it’s just best to outsource. We used to run our own ELK Stack. Now we are running an instance of AWS Elasticsearch Service. The change helped us to cut costs, improve stability, and improved manageability.
AWS’s Trusted Advisor: This tool works great to find orphaned or underutilized resources in AWS. We checked it daily and when old resources were found, a work ticket was created to clean up the resource.
Tagging Resources: Although I don’t consider tagging our AWS resources alone as being a money saving step, it has helped us to analyze our expenses better. Depending on how these are used, you can easily identify resources, how they are being used, and ultimately, determine if the resource is still needed or not.
Cloud Management Software: We use CloudHealth, a software that has helped us identify what our biggest expenses are. It is the primary tool we use in our monthly meeting to analyze our AWS bill.
Conclusion
Ultimately, each company’s needs are different and requirements are constantly changing. It’s not enough to implement a change and then say you’re running at peak efficiency. Rightsizing, Reserved Instances, and setting s3 endpoints are just a few of the money saving steps we’ve used to cut costs. However, I attribute a lot of our success to the routines we’ve implemented. Audits, routine checks, monthly meetings, and on-going tasks ensure there isn’t runaway costs and that we continue to drive our expenses down. Our success was recently affirmed in a recent meeting with two AWS representatives. We talked with them about how to cut costs, and within the conversation, they stated we were one of the best shops they’ve seen when it came to reducing costs in AWS.
