Setting up Pod Level Cost Allocation for AWS EKS

As cloud adoption surges, effective cost management is paramount for Application teams. Many modern products and applications are developed on the cloud agnostic Kubernetes infrastructure. While AWS Elastic Kubernetes Service (EKS) offers powerful orchestration capabilities, dissecting costs at a granular pod level can be very challenging. Without added tools, it was/is nearly impossible to get it done.

With some recent releases, it has become more manageable to get detailed data about the consumption vs costs of your kubernetes environment. Here’s a breakdown of how to leverage Split Cost Allocation Data (SCAD) or Kubecost for precise cost management in AWS EKS.

Enabling & using SCAD in AWS

Split Cost Allocation Data (SCAD) in AWS provides an intricate view of costs attributed to Kubernetes pods. For EKS there are two data types available you can inject in your CUR:

• Resource Requests: If you choose this option, the data is allocated by Kubernetes pod CPU and memory resources only. This way you visualize what part of the consumed EC2 resources are allocated to the pod based on what the pod need. Note that this is not the same as showing what the pod has consumed. In a sense, you just shift the view one layer lower instead of providing an intelligent view. It is the most easy option out there however.

Note: If a cpu/memory definition is not given to a pod, there will be no split cost allocation data available for a pod because there’s nothing to go on for AWS.

• Amazon Managed Service for Prometheus: this would set-up amazon managed services for Prometheus for the cluster, so the exact metrics consumed are monitored and used to do cost allocation. By using this option the application team will be paying what they actually consumed out of the provisioned instance. The difference with resource requests is that you are now using consumption metrics instead of provisioned metrics.

Note: This comes with the additional cost of Amazon Managed Service for Prometheus. Additionally, to do this, you need to ‘enable all features in your organization’, which is done under AWS Organizations -> Settings > Organization Details -> Feature Set.

Enabling SCAD is fairly straightforward. You can follow the steps written out in the detailed set-up guide for SCAD and Cost Intelligence Dashboard.

SCAD integrates seamlessly with AWS Billing tools, providing detailed insights into cost consumption on a pod level. When enabled, SCAD adds the following new columns to the AWS Cost and Usage Report (CUR).
split_line_item_actual_usage

• split_line_item_net_split_cost
• split_line_item_net_unused_cost
• split_line_item_parent_resource_id
• split_line_item_public_on_demand_split_cost
• split_line_item_public_on_demand_unused_cost
• split_line_item_reserved_usage
• spit_line_item_split_cost
• split_line_item_split_usage
• split_line_item_split_usage_ratio
• split_line_item_unused_cost

These columns allow for granular tracking and correlation of costs, facilitating deeper insights and better financial management. In addition to that, a few cost allocation tags are also created. Don’t forget to activate these so you can use them to structure your reporting.

• resource_tags_aws_eks_cluster_name
• resource_tags_aws_eks_deployment
• resource_tags_aws_eks_namespace
• resource_tags_aws_eks_node
• resource_tags_aws_eks_workload_name
• resource_tags_aws_eks_type

Enabling & using Kubecost

Kubecost is an alternative tool providing pod-level cost data. It deploys Prometheus resources and offers a decentralized approach to cost tracking, suitable for smaller environments or specific application teams. Kubecost started in 2019 as an open-source tool and has a history of being the go-to tool on the market to help you slice and dice your kubernetes costs. The recent collaboration with AWS is a great way to start using it in your environment.

While it requires manual data integration into centralized billing, Kubecost is easy to deploy and manage. With the deployment of Kubecost come similar Prometheus resources as with SCAD (it needs a db component after all). It then uses the data captured to create cost views in a separate console. You can use PromQL to get the data out of the prometheus database into your external billing tool, but you would have to do that for every individual deployment across your environment.

Additionally, Kubecost’s separate console must integrate billing data to account for credits, EDPs, and other rate adaptations already in place. Although the integration between AWS EKS and Kubecost represents progress, this requirement can complicate its use in complex FinOps environments.

Detailed information can be found in the Amazon EKS Integration guide on Kubecost.

Conclusion

For both product teams and FinOps teams, leveraging SCAD and Kubecost can significantly enhance cost transparency and optimization in AWS EKS environments. SCAD’s seamless integration with AWS billing makes it ideal for large-scale deployments, while Kubecost offers flexibility and independent control for smaller teams.

In my opinion, deploying SCAD is a must for larger enterprise organisations. At the very least ECS and EKS Resource Request data should be pumped into the CUR. Your current of the market tooling might not support the ingestion, but they should. And if you are doing FinOps at scale you should consider using custom built and managed tooling anyway.

Kubecost is lightweight and nimble and is great for the Spotify-model organisation where teams have a large level of autonomy. It’s also good as a tool to deploy at customers if you are an AWS reseller or consultancy partner.

If you plan on using AWS Cost Intelligence Dashboards to report on the costs, here’s a useful comparison:

References

• Cost Monitoring for EKS (Overview): https://docs.aws.amazon.com/eks/latest/userguide/cost-monitoring.html
• Understanding Split Cost Allocation Data: https://docs.aws.amazon.com/cur/latest/userguide/split-cost-allocation-data.html
• Split Line Item Details: https://docs.aws.amazon.com/cur/latest/userguide/split-line-item-columns.html
• Detailed deployment guide for SCAD: https://catalog.workshops.aws/awscid/en-US/dashboards/additional/scad-containers-cost-allocation
• Example CUR structure for SCAD:https://docs.aws.amazon.com/cur/latest/userguide/example-split-cost-allocation-data.html
• Detailed deployment guide for Kubecost: https://catalog.workshops.aws/awscid/en-US/dashboards/additional/kubecost-containers-cost-allocation
• CUDOS Demo Environment for EKS based on SCAD: https://cid.workshops.aws.dev/demo/?dashboard=containers-cost-allocation
• CUDOS Demo Environment for EKS based on Kubecost https://cid.workshops.aws.dev/demo/?dashboard=containers-cost-allocation
• Kubecost detailed deployment guide: https://docs.kubecost.com/install-and-configure/install/provider-installations/aws-eks-cost-monitoring