The Hitchhiker’s Guide to Costs and Usages in AWS

Published in

Nordcloud Engineering

10 min readJun 14, 2022

Introduction

There are a lot of challenges when it comes to cost optimization in the cloud, and it’s more of a journey than a destination. Cost optimization is a journey, not a destination. To understand cost optimization from a holistic perspective, we like to think of it as a lifecycle.

Here are four general phases of the lifecycle: analysis, cost optimization, cloud governance, and usage and cost monitoring.

AWS Cost Explorer helps us to explore and analyze spending across our AWS accounts, offering a wide range of filtering options to help us dig down into the details. With AWS Budgets we can set up the reports and alarms we need to keep a close eye on our spending. Lastly, the Cost and Usage reports give us a more granularity spend on the AWS services and resources for each service used.

Tools

We will explore some of the AWS tools which can be used to understand the cost and also create some notifications-based alerts to tame the usage of the AWS services.

Cost Anomaly Detection
AWS Cost Explorer
AWS Budget
AWS Cost and Usage Dashboards

Cost Anomaly Detection

AWS Cost Anomaly Detection is a monitoring feature that utilizes advanced machine learning techniques that identify anomalous and suspicious spend behaviors as early as possible so we can avoid costly surprises. Based on the selected spend segments, Cost Anomaly Detection automatically determines patterns each day by adjusting for organic growth and seasonal trends. It triggers an alert when spend seems abnormal.

AWS Cost Explorer

The Cost Explorer provides us with granular insight into the AWS service usage and costs accrued. It gives us a detailed breakdown of all services in the table view & visually. We can get AWS costs and usage over a daily or monthly granularity. It gives us the capability to create reports.

With granular insights, we can identify services and usage types that are costing more. It also helps us visualize cost trends over time. We can look for any sudden spike in usage.

AWS Budget

Sitting alongside each other within the AWS Cost Management group, AWS Budgets and AWS Cost Explorer are complementary services from Amazon Web Services (AWS). Using them together, we can analyze our cost and usage patterns and use that analysis to implement effective governance controls and cost optimizations. AWS Budgets lets us track service usage, utilization, and coverage for Reserved Instances and Savings Plans.

Creating budgets

There is an AWS setup wizard that starts when we create a budget through the console. Within the wizard, there are “Info” links at the top of each page, which lead to more detailed instructions and AWS documentation.

Budget alerts

Alerts are attached to a budget and can be created when creating or editing that budget. They consist of a threshold and a notification. The threshold contains a trigger, which determines whether the alert fires when today’s actual usage or the forecast usage for the budget period crosses the threshold.

We can also configure alerts to trigger automated actions when they fire. An action can be one of:

Attach an IAM policy to a user, group, or role.
Attach a Service Control Policy (SCP) to an Organizational Unit or the organization’s root.
Stop specific EC2 or RDS instances.

Forecasting in AWS Budgets

Forecasting gives us an additional option for how we set up our alerts. Using actual spending, we might set up an alert to notify us at 80% of our monthly budget, say. Using forecasting, we might want an alert based on projected overrun, say 105%. We can use both types of alerts with the same budget — a budget will support up to 5 alerts.

Limitations :

Forecast alerts won’t fire at all unless we already have enough (roughly 5 weeks) usage data.
As with any forecasting, although there will be some intelligence in the underlying algorithm, it’s only based on ourprevious usage patterns and may be inaccurate.

Budget Reports

Budget reports are sent by email on a regular cadence.

AWS keeps these reports simple (the only complexity is in the budgets themselves) — for a single report we can customize:

Which budgets are included (we can include more than one)?
How often the report should be sent — daily, weekly, or monthly.
Who should receive the report — a list of email addresses.

We will look into both services in more detail and see how they differ and how they can be used together.

AWS Cost and Usage Dashboards

The AWS Cost and usage report contain the most encyclopedic set of cost and usage data available. We can use cost and usage reports to get our AWS billing data in the form of reports. We can receive reports that break down our cost by the hour or month, by a product or a product resource, or by the tax that we define ourselves. There are three basic benefits of setting up cost and usage reports as listed below:

We can deliver report files to our Amazon S3 bucket.
We can also update the report up to three times a day.
Creating, retrieving, and deleting our reports can be done using the AWS cost and usage reports API reference.

Note: The following setup can be either done by a standalone account or consolidated reports for multiple accounts by enabling the consolidated billing feature in AWS organizations.

We will set up the following solution pattern using multiple AWS services which will not only enable us to check the cost and usage reports but also to fix dashboards in AWS Quicksight to monitor any anomaly in the usage of resources inside AWS services.

Here are the steps to enable the report generation:

Enabling the reports from the Billing section in AWS Console and creating the configurations for report generation.

Adding configuration includes creating the bucket and defining the granularity of the data ingestions into the report. Also, we need to update the AWS Services like Athena, Quicksight will also be using these reports.

Once the above configurations are set up, We will see the reports getting generated and stored in the S3 bucket. Also apart from the data, we will see some other files are also present.

Next, We need to download the crawler-cfn.yml file and use it to create a Cloudformation Stack. This Stack creates the AWS Glue Crawler and also the lambda’s which are responsible for pulling the data in Athena.

AWS Athena

Inside the AWS Athena Service, We can find the AWS Glue crawler has already created the database and tables schema based on the reports from the S3 bucket.

Once the stack is ready, we can check the CUR status by going to our database and running the following query:

SELECT status FROM cost_and_usage_data_status;

The above query will return either READY or UPDATING , the latter indicating that Athena may return incomplete results.

If the status is READY , we can verify it by counting the total number of items in our Athena database so far.

SELECT    COUNT(identity_line_item_id) AS Count
    
FROM my_cur_db.my_cur_tableWHERE
    MONTH = CAST(MONTH(CURRENT_DATE) AS varchar(4))
    AND YEAR = CAST(YEAR(CURRENT_DATE) AS varchar(4));

The MONTH(CURRENT_DATE) statement will fetch the current month as an integer, and CAST() will convert it to a string.

The above query will return the number of rows currently in the billing table.

Data Wrangling and Customization

Moving on to explore the datasets and columns of the reports, AWS provides us with extensive documentation for all the definitions of the columns for the report. Some of the important columns are :

Billing columns

line_item_blended_cost : The blended cost of this line item.
line_item_line_item_type : The type of charge covered by this line item. Possible values are Credit, DiscountedUsage, Fee, Refund, RIFee, Tax, and Usage.
pricing_public_on_demand_cost : The cost for the line item is based on public on-demand instance rates.

Resource columns

line_item_resource_id : The resource ID of this line item, if enabled in CUR. For example, an Amazon S3 storage bucket, an Amazon EC2 compute instance, or an Amazon RDS database can each have a resource ID.
line_item_line_item_description : The description of the line item type.
resource_tags_user_name : Contains the value of the Name resource tag.

Configuration columns

line_item_availability_zone : The Availability Zone that hosts this line item, such as us-east-1a or us-east-1b .
product_instance_type : Describes the instance type, size, and family, which define the CPU, networking, and storage capacity of our instance.
product_instance_family : Describe our AWS EC2 instances family.

Usage columns

line_item_usage_account_id : The account ID that used this line item. For organizations, this can be either the master account or a member account.
line_item_operation : The specific AWS operation covered by this line item.
line_item_usage_type : The usage details of this line item.
line_item_usage_start_date , line_item_usage_end_date : The start and end dates for the corresponding line item in UTC. The format is YYYY-MM-DDTHH:mm:ssZ. The start date is inclusive, and the end date is exclusive.
month , year : The month and year of this line item.
line_item_product_code : The product code of this line item. For example, AmazonEC2 is the product code for Amazon Elastic Compute Cloud.

Some of the relevant queries to find out the customized tailor-made queries are:

Identifying the most expensive services used

SELECT

    line_item_resource_id AS Resource,
    SUM(line_item_blended_cost + discount_total_discount) AS Cost
    
FROM aritrausagereport

WHERE
    MONTH = CAST(MONTH(CURRENT_DATE) AS varchar(4))
    AND YEAR = CAST(YEAR(CURRENT_DATE) AS varchar(4))
    AND line_item_product_code = 'AWSLambda'
    AND line_item_resource_id != ''
    
GROUP BY line_item_resource_id

ORDER BY Cost DESC

LIMIT 25;

Most expensive AWS EC2 resource

SELECT

    line_item_resource_id AS Resource,
    SUM(line_item_blended_cost + discount_total_discount) AS Cost
    
FROM aritrausagereport

WHERE
    MONTH = CAST(MONTH(CURRENT_DATE) AS varchar(4))
    AND YEAR = CAST(YEAR(CURRENT_DATE) AS varchar(4))
    AND line_item_product_code = 'AmazonEC2'
    AND line_item_resource_id != ''
    
GROUP BY line_item_resource_id

ORDER BY Cost DESC

LIMIT 25;

Most expensive S3 buckets:

SELECT    line_item_resource_id AS Resource,
    SUM(line_item_blended_cost + discount_total_discount) AS Cost
    
FROM aritrausagereportWHERE
    MONTH = CAST(MONTH(CURRENT_DATE) AS varchar(4))
    AND YEAR = CAST(YEAR(CURRENT_DATE)  AS varchar(4))
    AND line_item_product_code = 'AmazonS3'
    AND line_item_resource_id != ''
    
GROUP BY line_item_resource_idORDER BY Cost DESCLIMIT 25;

Lastly, another important use case would be to examine the cost of resources by their names.

Let’s say I have several resources with the name of my project, cdks3reactspa . These could be cdk-backendcontainer-jave , or aws-dynamodb-serverless etc. We can use the resource tags column to group costs based on names as well.

SELECT    resource_tags_user_name AS ResourceName,
    SUM(line_item_blended_cost + discount_total_discount) AS Cost
    
FROM aritrausagereportWHERE
    MONTH = CAST(MONTH(CURRENT_DATE) AS varchar(4))
    AND YEAR = CAST(YEAR(CURRENT_DATE)  AS varchar(4))
    AND lower(resource_tags_user_name) LIKE '%cdks3reactspa%'
    
GROUP BY resource_tags_user_nameORDER BY Cost DESCLIMIT 25;

Quicksight Dashboard Integration

Note: Setting up the Quicksight roles/permissions is out of scope for this blog

This AWS service helps us to create custom dashboards based on the datasets which are getting generated from the above queries. Inside the service, We need to perform the following steps to create an automated refresh of the datasets.

Inside the portal, We need to create a new dataset and mention the source as AWS Athena.

2. Based on the roles setup, it will automatically find the workgroup of the cost and usage report

3. Once we have added the connection, All the valid tables and schemas will be shown inside the AWS Athena catalog.

4. There is an offering from AWS Quicksight called SPICE which can generate an analytics dashboard quicker. Else, We can also our own SQL queries to create the dashboards.

5. Finally, We set up the refresh schedules of the datasets inside the AWS quicksight console and also enable sharing options for different people to visualize the dashboard.

Once, all the above setup is done. We move on the create the dashboard based on the datasets.

Here is the sample dashboard, Where we can see all the costs that happened for AWS services in the last 3 months. For e.g, the Month of May has seen a spike in usage of AWS ELB as well as AWS GuardDuty.

All the data will be incrementally changed based on the refresh rates as well as the setting of Cost usage reports in the billing console.

Conclusion

The Athena and CUR combination can help alleviate a lot of my-cloud-bill-is-a-huge-black-box problems. The Cost Explorer and Budget Reports are fine, but there are some problems only a heinous quadruple-join can solve.

CUR is also available for Redshift. If Athena’s concurrency limits are causing issues or if we need a full-blown RDBMS for cost analysis, then Redshift is the way to do it. For visualizations, QuickSight can use the columns directly but not the query results, so something like Redash or Tableau might be better for more complex dashboards.

The Hitchhiker’s Guide to Costs and Usages in AWS

Introduction

Tools

Conclusion

References

Written by Aritra Nag