Making The Invisible Visible: Costs
ServiceRocket (SR) Engineers are unable to make optimal decisions because they do not consider the true cost of services being used.
It is easy to see a price tag, but it is difficult to take into account all of the factors that make up a product’s true cost.
How can hidden costs be made visible? With a bit of effort, true monetary costs of many services used primarily on Amazon Web Services (AWS) can be calculated and made visible.
Cost Allocation Tags
AWS by default allows a tag to be assigned to an AWS resource, i.e EC2, RDS, etc. Tags can be used to organize resources, and cost allocation tags to track AWS costs on a detailed level.
AWS has two types of cost allocation tags, an AWS generated tag and user-defined tags.
AWS defines, creates, and applies AWS generated tag, and user-defined tags can be defined, created and applied by the user. Both types of tags must be activated before they can appear on a cost allocation report or to be used for budgets in the Cost Allocation Tags page under Billing Management Console.
SR Engineering has two platforms, Learndot and Apps which is comprised of many teams that develop different products utilizing different environments. The image below illustrates the user-defined tags created and applied on AWS resources created by Engineering.
All the teams in Learndot and Apps either use Terraform or Cloudformation with a Jenkins pipeline to provision their infrastructure and deploy their applications, thus implementing the user-defined tags above was fairly straightforward.
Budget on AWS
After having the teams implement cost allocation tags as one of the Ops Missions, Yuen-Chi Lian created some budgets with guesstimated limits for services such as EC2, RDS, CloudWatch, and CDN based on cost usage by Engineering (quarterly) as a whole and by Learndot and Apps (monthly).
These budgets were implemented with notifications to the Engineering Operations (Ops) mailing group. Ops receives emails in regards to a service for a particular budget in AWS surpassing the limit set.
However, this did not provide the ability to visually understand if and when the budget needs to be realigned or if a service was costing way more than it was intended and requires attention. If needed to be done, it would require tremendous manual effort.
Dashboard on Datadog
Hence, the Datadog dashboard above was setup to display the budgets from AWS which provides better visibility on a monthly basis and over a longer period of time for each service and platform which otherwise could not have been done via the emails Ops received.
Additional continuous efforts include:
- Ops to analyze AWS dashboards on a quarterly basis and set close to realistic budget limits on all the AWS services and account.
- Team to setup similar dashboard for products under Learndot (e.g Moo, Hamburger, and etc) and Apps (e.g. SFJC, SafetyOfficer, and etc) thus taking on cost ownership.
