silhouettes of taggers drawing graffiti
silhouettes of taggers drawing graffiti

The k8s-aws-ebs-tagger brings tagging to the AWS EBS volumes created by Kubernetes PersistentVolumeClaims (PVC). This new utility enables you to set arbitrary tags on the EBS volume so that you can better categorize and report on the state of your AWS resources. Having proper cost control tags can help you keep a handle on your AWS billing and resource utilization.

Let’s dive into how to install and use it.

Install the k8s-aws-ebs-tagger

The container images are released both on DockerHub and GitHub Container Registry and are built for both linux/amd64 and linux/arm64.

The first thing needed is an AWS IAM Role that is allowed to add & delete tags from EBS volumes. I recommend using kube2iam for assigning the role to the Pod(s) instead of using AWS access key/secrets. …

When dealing with AWS API rate-limiting there are a few tips & tricks that I find helpful. If your environment is like mine and you have a lot of code interacting with the AWS APIs, sometimes poorly, handling the default rate-limiting without errors is important.

Image for post
Image for post
Top AWS API calls in a typical hour

Python’s Tenacity

I’ve found that Tenacity for Python is a life saver. Tenacity is a general purpose library that automates retry logic. By decorating your functions Tenacity will automatically retry, with behavior determined by the decoration, when an exception is raised. …

I had an interesting conversation with a coworker in another business unit the other week where we were talking about instance types and planning for unknown workload sizes in our Kubernetes clusters. They asked what memory-to-cpu ratios my team used to decide the instance types to run for our clusters. I had to call timeout and talk about why ratios didn’t matter because I was using the cluster-autoscaler. I realized we needed to take a step back and go over the philosophy I use when running workloads on Kubernetes. …

graphical version of the blog title
graphical version of the blog title

Recently I ran into a situation where we had an IP conflict with another team’s Kubernetes cluster where they had a pod network CIDR block that conflicted with the CIDR block of the VPC my cluster (as well as legacy EC2 instances) was in. My team’s cluster could talk to their cluster over VPC peering but they couldn’t talk to me the same way. We didn’t want to put any of the application ingresses on the public internet and for internal limitations we couldn’t extended my VPC’s CIDR block. The only solution that could be found was to setup a VPC with a different CIDR block. This is easy enough to handle for the EC2 instances outside the Kubernetes cluster but live migrating a cluster without downtime was a bit of a challenge. Due to the application deployment pipeline the clusters have become pets to the engineering teams. That introduces a set of problems where spinning up & migrating to a different cluster isn’t possible without a significant time investment across many teams. …

For a variety of reasons people in the US don’t like talking about mental health. There’s a fear that there will be negative reactions and impact from talking about it. From the simple “I’m stressed out today” to the complex “I’m feeling super depressed today” it is all something that we don’t talk about. I think that’s a load of BS and I would love to see it changed. Mental health is no different from a broken arm or a twisted knee. It’s all about your body’s overall health and there are ways in which a medical professional can help you deal with it or adapt to it. …

Recently I had the opportunity to install Kubecost on several of the AWS clusters I manage. The tldr is that it was a very helpful and useful system. But to be honest, my initial thoughts were leaning towards the negative until I got it all setup. IMO, like a lot of start-up products, the documentation isn’t the greatest. I felt kind of overwhelmed by what needed to be done and the names of the project vs the docs vs the Github repo didn’t exactly match up (kubecost vs cost-model vs cost-analyzer).

BUT, and this is pretty huge, the Kubecost team was great to work with. They got on a video call, walked us through what I was doing wrong, and helped bridge the gap around what I didn’t understand. And they have a Slack channel to help as well. Once I understood how the components worked together I was good to go. I sent them my feedback about the documentation and hopefully that’ll help the next person who comes along. …

A bank vault door opening
A bank vault door opening

Two of my favorite pieces of technology have a marriage made in heaven with the vault-secrets-operator from @rico_berger. It allows, in a GitOps manner, the syncing of Vault secrets into Kubernetes secrets. While syncing between one secrets store and another sounds like a waste of bits it can make a lot of sense in some situations. Where I work we run multiple Kubernetes clusters across several regions, both on-premise and in AWS. Getting secrets securely to all of the necessary applications & platforms can be difficult without a good toolset. Vault is an incredible tool that provides a simple, yet powerful, secure method of storing and managing secrets. …

Kuberhealthy from Comcast is an incredible tool for doing synthetic checks against your Kubernetes cluster. I’ve been using it for several years now and was extremely happy when v2 came out a while back (seems like forever now). Since then there was a great blog on Kubernetes about using Kuberhealthy as a way to track your KPIs as well. As my adoption of Kuberhealthy increased I started relying on it more & more to track the overall health of the clusters I was responsible for.

Recently a co-worker and I wanted to start learning Go. I’ve dabbled with it in the past but never really learned about what I was doing. We figured that the best way to learn was to take a problem we had and try to solve it from the ground up using Go. We decided to take an open source approach to it and bring that code into our jobs instead of the other way around. While our employer embraces and supports open source projects we felt that this was something we should do on our own to better support the community. My coworker wrote the kuberhealthy-aws-iam-role-check tool and I wrote the kuberhealthy-ami-exists-check tool. …

Hey all, I’m torching the old and replacing it with this blog on Medium. I’m not sure if I’ll keep this one any more up-to-date than the previous but I’m going to try.

My focus here will primarily be Kubernetes, Programming and general Technology but there’s be a little of everything else mixed in as well. Hopefully you’ll find it interesting enough to stick around for a little while.

As always, my thoughts are my own and remember, your reality is all in your head…


Mike Tougeron

Lead SRE @Adobe , #kubernetes fan & gamer (board & video). he/him. Remember, reality is all in your head…

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store