AWS — Keep an eye on cost

In cloud world, things have become easier as we need to spend less time on setting up infrastructure, doing operational work and we can focus on building up the core business logic.

Whenever we are building AWS architecture or using any service, it’s important to carefully consider all the cost aspects and missing this may give us billing shock. There are hundreds or may be thousands of articles on the internet to avoid these cost surprises and i am adding one more in the list.

The main part of building any solution is the core logic and how/what services we will be using and we should make a habit to look into the pricing documents of those services. Summarizing here some of things which are helpful in cost optimization.

Minimum Billing constraints: There are minimum charges for AWS services and we should be knowing them before implementing any service.

Some of the examples are -

S3 Standard-IA is designed for larger objects and has a minimum object storage charge of 128KB and minimum 30-day storage charge.

Glue ETL Jobs — $0.44 per DPU-Hour, billed per second, with a 1-minute minimum (Glue version 2.0) or 10-minute minimum (Glue version 0.9/1.0) for each ETL job of type Apache Spark

AWS Lambda used to charge for a minimum 100ms duration. Now in recent announcement billing granularity changed from 100ms to 1ms.

For small file problem, we can use EMR S3DistCp to combine the files to bigger ones with one-time spend. Better during ingestion phase itself we can use micro batching services like Kinesis Firehose with appropriate Buffer size and buffer interval.

Apart from Spark job, Glue also provides option for python shell job where a job can be run on a minimum .0625 DPU. This is suitable for processing small amount of data. For Ex — scheduling a stored procedure on a Redshift cluster as all the processing happens inside Redshift cluster so we just need a client machine to run the procedure. BTW Redshift also announced scheduler for SQL queries. There is a nice python library called AWSWrangler which can be used in a Glue shell job for easy integration with AWS services like S3/Athena/Redshift.

Snapshot Charges: Snapshots are expensive and We need to carefully consider the backups retention period and also delete manual snapshots which are not needed. This is how RDS snapshot costing works:

There is no additional charge for backup storage up to 100% of your total database storage for a region. Additional backup storage is $0.095 per GiB-month.

Cases where business needs snapshots preservation for longer duration due to compliance reason, consider logical backups and keep those backups on S3 (Standard/IA/Glacier). RDS introduced Snapshot Export feature, using this snapshots can be exported to efficient Parquet format on S3 and queried using Athena.Important thing to consider here is that restoring logical backup to database will take more time compared to snapshot backup restore.

Unexpected Bills: While S3 and Glacier storage costs are well known but sometime we miss the life cycle transition cost from S3 to Glacier. When we are expecting to lower the cost in transitioning huge amount of data, but this transition activity could spike the bills. Why?

• AWS charges .05$ per 1000 put requests to Glacier and hence a lot of small files transition will increase the cost.
• Glacier adds up 32Kb metadata to each file. Again if we are having millions of small files that 32Kb overhead will cost more than expected.

Update: AWS recently annouced the price reduction for data transition from S3 to Glacier.

Integration/Usage with other services: When using Athena/Redshift Spectrum, we know the pricing is 5$ per TB data scanned. Here we need to consider other related cost components as well.

• S3 cost due to data storage.
• Athena/Spectrum doing get requests on S3 objects.
• KMS charges, if encryption enabled.
• Athena result set charges on S3.

CPU bound licensing cost: When using EC2 machine for external software where those software charges based on number of CPUs, we can think of using z1d Instances.

Others: Cost Optimization with recent features:

• Redshift Data Api — With Redshift data api, we can asynchronously run queries and we don’t need to wait for acknowledgment.
• Aurora Fast clone — In case we need to test some feature we can create Aurora fast clone. This saves storage cost as storage is shared initially (until we are making block changes).
• AWS Instance Scheduler — This is nice utility to automate RDS/EC2 start/stop at you defined timings.
• GP3 Volumes —With recently announced GP3 volumes, IOPS and throughput can scale without need to provision additional block storage capacity.

Data Transfer Cost: Data transfer cost can be reduced with endpoint services so that data is not traversed through internet and that is more secure as well.

One more recent experience i had with CloudWatch logging for Postgres Aurora cluster where CW logging was costing us around 5k$/month. Upon further checking i noticed that db parameter log_statement was set to “all” while we wanted to audit only “ddl” statements. I changed the db parameter to reduce logging and also set retention on CW log group to 1 month.

Closing Notes:
There is rapid development happening in all AWS services and AWS continuously launches new features. We must need to sync up with new changes coming, to build and improve our architecture.

There are native AWS provided tools to suggest cost optimization opportunities like Trusted Advisor, Compute Optimizer. And then there are many external players as well who charges some bucks in return of your cost savings. Recently i came to know about granulate. Will explore sometime, Interesting to see how without any code changes they are bringing so much values.