How to speed up local Kubernetes development by proxying Helm charts
At Paxos, we’ve been running our services on Kubernetes for a while, but had trouble managing multiple environments until we adopted Helm and ChartMuseum. While these tools introduced a few troubles with local development at first, we made ChartProxy to address these woes, and the deployment process has never been smoother.
A Brief Primer on Helm and ChartMuseum
(Feel free to skip to the next section if you’re already familiar with these technologies.)
To write Kubernetes resource files for multiple environments, ConfigMaps will be your primary option to assign environment variables, set properties, and generally configure your resources. However, this is a fairly limiting approach, and you may find yourself copying entire files with minor changes. Helm is a package manager for Kubernetes, and it’s designed to alleviate these problems. It uses Golang’s template package with sprig functions to fill in variables, conditionally include code and even fill in entire resources (or portions thereof). Once you’ve written your templates, you can use Helm to package your resources into a single deployable chart, which can depend on other charts. You can then deploy this chart to your Kubernetes cluster and easily manage upgrades and rollbacks.
ChartMuseum is a chart repository that easily integrates with Helm. It manages your chart versions and allows your team to share charts with each other and with your production environment. Crucially, for our purposes, ChartMuseum exposes a simple REST API that’s easy to work with.
This barely scratches the surface of the use cases of Helm; if you want to learn more, check out the official documentation. Just be sure to come back here to increase your team’s local development productivity!
Problems with Local Development
We ran into problems with chart development nearly as soon as we began to convert our deploys to Helm. Because we were using ChartMuseum, we noticed that our deploy scripts contained many references to our chart repository (e.g. `paxos-charts/my-chart`). This meant that any time we ran our deployment, Helm would only use the chart in our shared repository. While this is the desired behavior when deploying to production, it’s the exact opposite of what we wanted to develop a chart and deploy it locally.
Initially, we attempted two strategies to get around this, but neither worked very well. In our first attempt, we simply pushed our in-progress charts to our shared repo. This approach worked perfectly when one engineer was working on the conversion, but as you can guess, it quickly fell apart when we started adding more engineers to the project. We found that an individual’s deploy could be easily broken by development changes made by another engineer to a different chart.
Afterwards, we made local changes to our deployment scripts to reference our local charts instead of the ones stored in ChartMuseum. This involved changing `paxos-charts/my-chart` to `/path/to/my/local/chart`, and then changing it back before committing. This process addressed the chaos of free-for all chart publishing of our first attempt, but was annoying, slow, and prone to error.
Enter ChartProxy
What we really wanted was a solution that allowed us to freely publish charts without interfering with each other. Fortunately, this isn’t a new problem, and the world of package management offers plenty of analogous solutions, such as `mvn install` and `npm link`. With this in mind, we made ChartProxy, a simple Bash script that manages a local ChartMuseum and provides access to our shared charts.
Working with ChartProxy is super simple and only requires adding a few commands to our process. First, `chartproxy start` starts the local ChartMuseum and re-directs all calls bound for our remote repo to our local one instead. This works seamlessly for both publishing and retrieving charts, so none of our deployment scripts need to be changed. Next, `chartproxy update` grabs a chart that it doesn’t already have from our shared repo and adds them locally. This way, we’re able to fully use our latest production charts in conjunction with whatever chart we happen to be working on in development. Finally, `chartproxy stop` stops the local ChartMuseum and points helm back to our shared repo. With this in place, we only need to update the Helm `requirements.lock` file before committing.
After rolling out ChartProxy, we realized that we had the potential to solve another problem. While switching to Helm, we were simultaneously modifying our AWS VPCs. This occasionally resulted in engineers losing access to our remote ChartMuseum. Fortunately, ChartProxy pulls down a full copy of the remote repo, allowing an engineer with the proper files to tar them up and send them to whomever needed them. We ended up doing this enough times that we added `pack` and `unpack` commands to simplify the process and prevent any mistakes.
One of the best parts of being a software engineer is the ability to create your own tools. Now, with ChartProxy, we can use the full powers of Helm and ChartMuseum for local and remote deploys without fiddling with temporary code changes or interfering with each other’s development. Give ChartProxy a try if you’re struggling to unify your chart deployments across development and production.
The Paxos Engineering team is always creating clever solutions to solve problems big and small. Interested in joining a team that values ingenuity, impact and cold brew coffee? Join us!
Tags: engineering, innovation
Originally published at www.paxos.com on September 27, 2018.
