Tint: Multi-Cloud Cost Visualization for Terraform
Introduction
Terraform Cloud recently added a new feature called Cost Estimation, which allows you to integrate cloud infrastructure spend estimates into your provisioning workflow and Sentinel policies. Cost Estimation builds on top of the pricing tables from AWS, Azure, and GCP and provides a standard interface across all three clouds in both Terraform Cloud (TFC)and Terraform Enterprise (TFE). The latter is the self-hosted version of Terraform Cloud.
This standard interface provides a valuable way to analyze, report on, and visualize cloud spend across cloud providers. In this post, we’ll dig into this interface deeper, talk about the data that is collected, and then walk through Tint, a project that I created which aggregates Terraform’s cloud estimation data, dumps it to a CSV format, and provides a simple React-based visualization tool.
Terraform Cloud Foundations
Before we dive into Cost Estimation, let’s take one step back and level set on the foundational concepts of Terraform Cloud.
Organizations
Organizations represent the topmost object or container within Terraform Cloud. Generally, you can think of these as business units or organizations like Engineering, Sales, Marketing, etc.
Workspaces
Workspaces are the next step down within this hierarchy, allowing you to represent various projects and their multiple environments in a more granular way with Terraform configurations. Typically, you’ll see dev, staging, and production environments separated into different workspaces as they each need their persistent state file that differs from one another.
Run
The final piece of this hierarchy is a Run, which consists of four stages. A run can be triggered either manually or via a change in an underlying VCS repository.
Stage 1: Plan — This step runs theterraform plancommand against a Terraform configuration associated with the workspace. Effectively, this is responsible for going out to the various Terraform Providers used by the Terraform code and performing a diff against what currently exists and what needs to be provisioned.
Stage 2: Cost Estimation — As mentioned above, Terraform Cloud adds Cost Estimation functionality, and this happens to be the next step. This step will calculate an estimated cost for the supported resources based on price tables and some models behind the scene.
Stage 3: Policy Check — The next step for TFC or TFE is to check any Sentinel policies that are attached to the run’s workspace. This allows you to automate infrastructure policy enforcement based on cost, security, best practices, and other factors. Check out this Introduction to Sentinel and a more advanced use case around combining Sentinel with ServiceNow.
Stage 4: Apply — Finally, if all the other steps succeeded, Terraform Cloud or Terraform Enterprise will apply the generated plan with the terraform applycommand and begin provisioning resources.
Cost Estimation API
The Terraform Cloud Cost Estimation API brings a lot of fascinating and granular data to the table. In the screenshot above, you can see the data that is provided to end-users within the TFC UI. However, if we dig into the Cost Estimation API a bit more, there is additional data supplied in an easy-to-consume JSON format.
In the screenshot below, I’ve captured the relevant JSON objects that represent the resources from GCP, AWS, and Azure. Namely, we get information about which provider is used, the monthly costs for each resource, and even breakdowns of the actual VM instance, including storage and compute resources.
Tint
The primary goal of this project was to iterate and aggregate data from all of the organizations, workspaces, and corresponding runs that have been applied within a Terraform Enterprise deployment. Once this data is aggregated, Tint calculates the length of time those resources have been running and the total cost for each provider and then dumps this data to disk in a CSV format.
In addition, Tint includes a simple React-based visualization tool that displays this data in a more aesthetically pleasing way.
Components
Backend — Tint relies on Go to serve a simple REST API and aggregate this data on startup from a Terraform Enterprise deployment. In general, Tint uses the Terraform Go Client library to fetch data; However, this library doesn’t expose all of the Cost Estimation data. In this specific scenario, I opted to make an HTTP request to TFE and parsed the JSON manually.
Frontend — The Tint visualization is built on top of React, Typescript, Redux and, Chart JS. While this was a overkill for the size of the project, it opens Tint up to be expanded in the future.
Conclusion
The TFC Cost Estimation API is an exciting addition and, I am excited to see how this functionality continues to grow over time. As people and organizations continue to expand into the public cloud and take advantage of best-of-breed services, having centralized cost reporting and policy enforcement makes this easier to manage.
