The Hidden Cost of AWS ELBs

Collegedunia
Sep 7, 2020 · 4 min read

By Rohit Gohri

Or, how we saved $1200 every month with Let’s Encrypt

This past month we had a really high AWS bill, and it was supposed to be high but not THIS high. The culprit — “Data Transfer Out”. I’m sure I’m not the only one who has had to deal with this.

Some Context

We had a NodeJS service running on EC2 in an Auto Scaling Group (ASG) behind an Application Load Balancer (ALB) that was supposed to handle a huge volume of requests. We were expecting a consistent increase in traffic this month, so we had made estimates on how it would affect our costs. We had expected some high data transfer out of our Application Load Balancer (ALB) but this was a consistent 4–5x of our estimates. Time to investigate.

Analyzing the Data

After filtering Cost Explorer data with tags, it was clear that the ALB was the source of these costs. I immediately went to CloudWatch and checked the “Processed Bytes” metric of the ALB.

Cloud Watch

Processed Bytes is supposed to be “the total number of bytes processed by the load balancer over IPv4 and IPv6 … includes traffic to and from clients”, so ideally Processed Bytes should be greater than the “Data Transfer Out”. But it was not..?

Our stats showed Processed Bytes to be around what our estimates had predicted for this amount of traffic. But Cost Explorer was showing a number that was 4–5 times that. Time to contact support.

AWS Support

We raised a support request to AWS, but we didn’t have high hopes because we were not on the premium support plan. But it was worth a shot.

ELB Access Logs

Next up, we enabled ALB Access Logs to an S3 bucket and let it generate some logs for some time. Upon checking the logs, we were certain that this wasn’t a problem at the application level. The access logs showed response size consistent with what we had estimated, no inflation due to extra headers or something like that.

VPC Flow Logs

We wondered if there was some other service running that might be communicating over an Elastic IP or public IP instead of a private IP. To check that, we enabled VPC Flow logs to another S3 bucket and let it generate some logs for some time.

We didn’t find any services that might be communicating over Elastic IP but we did find some services that could be moved to the same Availability Zone (AZ) and reduced some Regional Data Transfer costs.

Frantic Googling

After eliminating all possible sources of Data Transfer we still had no clue what was the source of the Cost other than that it was from the ALB. This led to frantic googling and sending/explaining the problem statement to various people.

Then over the weekend, trying to think of keywords that would not lead me down to more documentation, I finally found this StackOverflow answer:

Q. AWS ELB: How can I see how many bytes of outbound network traffic an SSL-enabled ELB generates

A. The handshake of a TLS connection is pretty much constant and do not depend on your application. So your function is really : number of connections * ( size of http responses + constant)

And it finally clicked, the missing piece of the puzzle. Since HTTPS was handled directly by the ALB, it wouldn’t show in the Access Logs as the instance didn’t have to deal with it.

Here is an article that estimates the payload to average ~6.5k (depending on your certificate size). Our response sizes were really small, and this could be a major part of the response size if the user was visiting the site for the first time. Since a lot of the traffic was new, it would be a significant chunk.

That led to even more Googling on how HTTPS might affect AWS ELB costs and we came upon this very good article that also suggested increasing the default timeout for idle connections to 10 minutes for the ALB along with changing the certificate.

Testing Different HTTPS Certs

We were currently using the default certificate generated by Amazon Certificate Manager (ACM) with the Load Balancer. We tested the size of the TLS handshake with Wormly Test SSL Tool and then replaced it with one generated with Let’s Encrypt. To be thorough we also tested TLS handshake sizes of different websites with different Certificate Authorities.

Image for post
Image for post

DigiCert was the best with respect to size, but Let’s Encrypt was close enough (& free!). ECC certificates are not compatible with some older browsers, so we generated both the smaller ECC and RSA certificates through acme.sh and uploaded them through AWS CLI. The ALB will automatically select the best one depending on the client (read more here).

After adding them to the ALB listener and letting Cost Explorer catch up for a day, we saw a significant decrease in our Data Transfer Out Costs for the same number of requests.

It was still not equal to the estimates we had, but it will never be because we never considered the TLS handshake size in our estimates. AWS Costs really are a mystery until you actually get the bill.

Image for post
Image for post
source: nclouds blog

Collegedunia Engineering

A know-it-all of Engineering stream

Collegedunia

Written by

Collegedunia is an extensive search engine providing students, parents, and other players in the education industry with the most comprehensive and informative

Collegedunia Engineering

Get to know all there is to learn about Engineering as a stream

Collegedunia

Written by

Collegedunia is an extensive search engine providing students, parents, and other players in the education industry with the most comprehensive and informative

Collegedunia Engineering

Get to know all there is to learn about Engineering as a stream

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store