How we cut data costs by more than 60% by leveraging a multi-cloud strategy
At Dcode we manage the infrastructure of a diverse range of
customers, both on the 3 major clouds and on-premise. One of these companies specializes in ad server technologies.
Cloud services represented a major part of their operational costs, so controlling their spending on AWS was a high priority.
Architectural Overview
Our client’s architecture handled around 10 TB of outbound traffic a day, and was split into 2 parts:
- Application architecture consisting of EKS clusters with autoscaling deployments running the ad-server software.
- Data Lake consisting of AWS Kinesis FireHose processing metrics generated by the app, batching them to an S3 bucket, with AWS Glue manipulating the data and querying it through AWS Athena, and a couple more data processing services running on top of EKS.
Ad servers typically process tens of thousands of ad requests per second, with each response sent within a few milliseconds. Being able to respond to so many requests so quickly means that availability, scalability, and low latency are all key requirements of an ad-serving solution.
Data Transfer costs explode
As we scaled to include more and more ad servers, our AWS bills started to skyrocket. We saw this by looking in AWS Cost explorer and grouping by Usage Types. The EC2-User DataTransfer-Out-Bytes usage type started from around 7000$ per month and nearly tripled itself only 4 months later. We predicted this would continue to grow as the ad servers scale, so we needed to quickly find a solution. We looked for opportunities to cut spending, starting with the biggest factor, data costs.
We started by doing the following three things:
- Moving the ad servers to a public subnet. When running a service on a private subnet, and querying a remote server outside of AWS, the request will hit the subnet’s NAT Gateway and the costs are for both the data transfer out and data transfer in of the request. So the first thing we did was move all of those services to a public subnet, connected to Internet Gateway, so we can pay on data transfer out only.
- Creating a VPC Endpoint for the ad server’s connection to Kinesis Data Lake for quicker and private connections.
- Moving the EKS nodes to a single Availability Zone. We wanted to remove the cost of data between AZs with the con of not having multi-zone high availability.
We found that doing all these improved performance, but barely made a dent in the data transfer bills, and realized that the main chunk of the bill was the data egress from the ad servers, which would only continue to grow. At this point, we looked to other cloud providers for different network pricing options.
After seeing multiple comparisons between AWS, Azure, and GCP (Like this one for example), we saw that our options were quite limited and we would probably get billed similarly on the other major cloud providers.
For 10TB of data egress on the 3 major cloud providers we would pay:
AWS: ~900$
GCP: ~1K$
Azure: ~800$
DigitalOcean to the rescue!
While digging deeper into our investigations, we stumbled on an article about DigitalOcean, a cloud provider with which we had some experience, and were fond of for their great user-friendly interface and cheap pricing.
When we looked through their docs, we were amazed to see that their Virtual Machine offering (a service they call Droplets) came with 1000GB a month of free outbound data transfer for their smallest plan each. This was pooled at the account level, meaning the Droplets could share the full account allowance between them.
We realized that by moving all our ad servers to DigitalOcean Droplets, we would not need any excess data transfer and we could theoretically slash our outbound data prices to nearly ZERO. This was a massive game-changer for our operational costs, and we decided to move all EKS Clusters from AWS to DigitalOcean.
We preferred to continue using the AWS data services as their prices were low and we were satisfied with them. Since AWS doesn’t charge for inbound traffic and our DigitalOcean traffic quota was more than we’re using — splitting the architecture to both clouds didn’t come with additional costs.
We slowly started to forward a small percentage of clients to DigitalOcean, and are continuing to move more and more of our traffic.
After moving our servers to DigitalOcean, we didn’t hit the network outbound data limit, so we weren’t paying anything for the data there, only for the Droplets themselves.
As the following screenshot shows, we started migrating our servers during February and the traffic costs were cut by March, while we still kept our Data Lake & data-related services in AWS.
We still have some network bills in AWS, but we are at a much larger scale than before and are paying about the same network bills that we started with. We expect the bills to drop more when we migrate more of our ad servers.
Final Thoughts
At Dcode we commit to finding the solutions and technologies to fit the customer’s exact needs, and breaking the barriers between clouds using a multi-cloud approach is a great way to enable us to do so.
We suggest that when facing complex cloud architectures, engineers and architects should try looking into smaller cloud vendors, such as DigitalOcean because they might have interesting offers and solutions that can fit your needs.
