Kubernetes, 1 year in production

It’s been a little more than a year since we’ve moved all of the core infrastructure behind Mic.com and related services to Docker and Kubernetes. We thought we’d share some of our experiences and lessons learned.

In a few words, Kubernetes (or just k8s) is a deployment automation system that manages containers in distributed environments. It simplifies common tasks like deployment, scaling, configuration, versioning, log management and a lot more. Overall, it can greatly simplify devops and allow a dev team (especially a small one) to focus on the app itself.

Disclaimer: This post is pretty technical and assumes some familiarity with devops concepts. It also reflects our personal experiences, which may be different from yours depending on your use-cases.

1. A cluster deployment process can be complicated, but tools automate a lot of it

There are a lot of moving parts in making a Kubernetes cluster work. Master nodes, worker nodes, network configuration, registry access keys, ssh keys, certificates and DNS are just some. We use AWS as our cloud provider, so at least VPC, routing and security groups need to be configured. It makes sense to automate all of this.

We use kops for cluster deployment and management on AWS. What’s great is that kops is built on the same principles as k8s. Configuration is declarative, which means when we edit a config file, kops figures out what infrastructure changes need to be made.

Overall, kops is amazing! The only issue we’ve encountered is that cluster upgrades across minor versions seem too dangerous to apply to the live production cluster. For example, upgrading kops from 1.6 to 1.7 can result in downtime. This is probably an issue with k8s rather than with kops, but still something to be aware of.

We decided to launch a new cluster and gradually transfer configs and deployments to it instead. To be fair, this could be a one time issue. In any case, it’s a good idea to test changes like this on a backup (staging) cluster first.

2. Using a separate staging cluster is a good idea

It’s important to test code in an environment that closely resembles production, but that is not actually production. With k8s we can:

• use a single cluster for both production and staging pods, keeping them isolated from each other using namespaces and resource limits
• use the second cluster for staging pods

We decided to do the latter. This turned out to be a good idea because pods’ limits and isolation are tricky to get right, and we don’t want to run pre-production code with full access to production resources (via AWS security groups).

Using a separate cluster also allows us to test and experiment with cluster configuration changes without affecting production and risking downtime.

3. Blue-green deployments make application updates painless

k8s makes it easy to run multiple copies (replicas) of a container, but replication makes code rollout a little more difficult. To avoid downtime, we update one container at a time (rolling update strategy), but this makes it possible for two versions of an app to be in production at the same time while the update is in progress. This presents a couple issues:

• Unless you use “sessionAffinity” (aka sticky sessions), requests from the same client can be routed to any of the two running versions. This can be a problem for web apps, where a new version of the frontend tries to access newly added backend route, but gets 404 from the old backend that is still running.
• Asset caching: if CloudFront tries to update an asset expecting a new version, a previous version may be returned and wrongly re-cached.

To mitigate these issues, we use blue-green deployment. k8s (as of December 2017) doesn’t support this out of the box, so we wrote a custom script that implements it using two parallel k8s deployments.

This works great. We use liveness probes before flipping the switch to make sure rollout is successful.

4. Integration with AWS is pretty good

We use AWS CodeBuild to build and upload container images to our private ECR registry. This way we don’t need to maintain a constantly running build server and registry server.

It’s great to have every recent version of our apps stored in the registry, though we have to delete older versions periodically because of AWS storage limits.

As for the ELB, the Classic Load Balancer integration works perfectly, but the Application Load Balancer isn’t supported yet. As a workaround, it can be configured manually using a NodePort target.

In staging, we use NGINX Ingress Controller to route traffic from a single ELB to multiple backends.

We use a Slack bot script (Hubot) to invoke builds and update deployments and ingress rules through the k8s API.

5. Monitoring

Heapster, InfluxDB and Grafana were pretty easy to set up using example yaml specs. This provides nice graphs (shown above) and the ability to set up alerts. InfluxDB deployment uses “nodeAffinity” to attach to the EBS volume in the correct AWS zone and region. This works well, but we aren’t sure yet whether to deploy a mission-critical database to k8s. For now, we’re pretty happy with RDS as our main database.

We also use Stackdriver to collect per-instance metrics, although we haven’t figured out a nice way to install Stackdriver agent onto Kubernetes nodes. It would be perfect to be able to do this with kops, without extra tools.

6. Autoscaling

Autoscaling is where Kubernetes shines. I think this is the most exciting feature. It allowed us to greatly reduce the number of running instances. We’re still experimenting with thresholds.

We implemented the cluster autoscaler and horizontal pod autoscaler:

• The Cluster autoscaler is a separate thing that takes care of resizing EC2 autoscaling group as needed to fit all the pods.
• The Horizontal pod autoscaler is the component that drives everything by adding or removing pods, depending on the pod’s CPU usage.

Autoscaling requires carefully adjusted pod requests and limits configs. It also requires making sure node IPs aren’t hard-coded anywhere. Autoscaler pretty much takes care of the rest.

Conclusion

Overall, we think Kubernetes is an amazing tool, and our experience with it has been extremely positive. We are a small team and don’t have a dedicated devops engineer, so k8s makes our lives a lot easier.

We definitely have things to improve in our setup, such as better monitoring and proper Stackdriver setup; better pod resource limits to run fewer instances at night; additional automation for security groups configs; and maybe some other things and gotchas we aren’t aware of yet. It’s worth noting that our workload isn’t very large, so we aren’t pushing k8s to the limit.

We’d like to thank all the Kubernetes contributors. We’re excited about the future!