Don’t have time or want to just get to coding? We understand.Source code: https://github.com/indigo-ag/serverless-spending-sanity
Indigo is a DevOps engineer’s dream. A rapidly growing company, using cutting edge technology, trying to change the way things work throughout 12 different services that are customer-facing.
But as the company grows, so do the number of resources and resources mean money.
Problem: How Do We Get Granularity with managing costs without having to update all the Infrastructure?
Sure you could go and individually tag each resource, but as I usually say: manage it once, manage it forever. Automate it, watch it forever.
We run all of our architecture on CloudFormation, but asking architects to write in tags, is the same as asking developers to write tests: sounds good originally, but unless strictly enforced, can fall by the wayside. We ran into this challenge at indigo and after some collaboration, we decided on the best answer to this business-critical question.
The answer: Serverless.
Serverless gives us the ability to solve the problem with three major benefits:
- It does not intrude on the deploy process but completes a job that is a critical metric to the product.
- We can improve it going forward as AWS continues to make improvements to cost explorer.
- We can enable it and disable it without breaking anything.
The lambda will be triggered every time a CloudFormation stack updates. The lambda will then go and look up all the resources that are within the stack and tag them with a specific set of tags from a JSON document.
Question: Why not just tag the stack?
Answer: If we want more granularity, we have to tag each resource. Within CloudFormation, there are stack specific tags that will fallto the resources.
Setting up Serverless:
We will download the serverless NPM package and install it globally then write the initialization:
npm install -g serverless
serverless create — template aws-python3
This writes the serverless.yml file which is a perfect boilerplate for deploying our functions.
Next, we use a package.json file that will have all of our deploy scripts and be able to hold the serverless packages that give serverless its superior powers.
Npm init -y
yarn add serverless serverless-aws-alias serverless-prune-plugin
We are now ready to start building some lambdas.
Serverless init gave us more to the boilerplate, including a basic handler page for our python function. This is an okay idea, but there is also the ability to make things look a tad cleaner by putting our functions in a folder.
Now all of our function handlers will just have to start with src/name_of_file.handler_function
Feel free to check out the source code for the full breakdown of the code we will be using, and for some helper goodies.
There are some key concepts that our cft_tagging.py function does.
- Look for the stack
- Look for a block in the JSON document
- If it isn’t there, write It then tag each resource depending on their type.
- Go through each ARN within the stack to be able to tag the resource with the block specified.
Serverless will handle the deploy of our lambda function and all of the logging that goes with it. Running the following will deploy the lambda and resources to the current profile set in your
~/.aws/config file or your
AWS_PROFILE environment variable.
Okay, we have tags, now what?
You guessed it, more Serverless!!
We will need to give the lambda a specific role to be able to execute all of the describing we need. Luckily, serverless will allow us to create the IAM role we need for each of the lambdas.
ce:*to the policy
costs.pyfunction that will fire starts off a process that will do the following:
- Go through each of the services we wish to tag and break down their costs by the team tag. Report the spending of the service within the past day.
- For each team that comes back from the spending of that service, check the percentage change for the spending. If the change has doubled, send a slack notification and an SNS message.
The lambda fires and then uses each of the tags we allocated within the tagging lambda to pull the costs per resource and per team. This will write to a JSON file within an s3 bucket so we can hold the state of the last set of tags.
The current lambda we have set here looks to see if the change in the slope of the line between the two points is three times the original value. Depending on how tight your budget is, you can either increase or decrease this number.
Within your project directory, run the following
Serverless will then deploy both of the functions within a CloudFormation Stack with logging, an S3 bucket for the artifact and take care of the new versioning as we deploy (awesome!).
Many third-party tools will do some of the work we have here with nice UI’s and further cool features, but I find it a tad odd to spend money on things that are overlooking spending. With the Serverless Framework, we can turn the lambdas on and off as we see fit. Maybe there is a specific time in which we know there are some big changes to infrastructure. Maybe there is a rollout, our auto-scaling is fixed, but we want to know when resources are being spent past what we believe is reasonable. I like the ability to have a tool in a holster, rather than in a contract.
Further Reading and Docs:
Serverless Docs: https://serverless.com/