Automatic Validation of Kubernetes Config in Azure DevOps
Tired of your releases failing because of indentation errors in the YAML files? Wish syntactic errors where caught by the build pipeline like with other code? Then this might be the solution for you!
TL;DR
- You can validate your k8s config as part of the build pipeline using Kubeval.
- Add this and this other file to a folder named Kubeval in your repo, these two tasks to the pipeline definition, and you’re done!*
*I’m assuming that you’re already ignoring all bin folders in your .gitignore, and are using a Linux-based agent in your build pipeline.
Tell me more!
Remember --dry-run?
Kubectl can validate k8s config locally, without actually deploying anything, by using the --dry-run flag. It’s nice, and a great option for validating configuration on your own machine, but it can be tricky to get Kubectl to work as part of the build pipeline. So when you forget to validate the config before committing, or you just can’t be bothered to find the correct combination of commands and arguments that’ll just validate, and not complain about something else, you’re stuck in a situation where everything looks okay until the release fails 💣
Enter Kubeval
Kubeval is a tool dedicated to validating Kubernetes config locally or in a CI pipeline. It’s open source software, supports multiple Kubernetes versions, and does a decent job of spotting all the small mistakes we all make when editing deployments, services, ingresses, and the like.
Limitations
Even if Kubeval validation succeeds, the release can still fail, since some errors can only be found when executing the configuration on a real cluster. For instance, if you’re referencing a ConfigMap that doesn’t exist on the cluster, Kubeval won’t be able to catch the error. Still, it’s pretty good at getting all the “did not find expected key“, “wrong indentation”, and “invalid type” mistakes.
Adding Kubeval to your build pipeline
Getting hold of Kubeval
Kubeval distributed as a compressed binary, available for different platforms. Given that we use a Linux-based pipeline agent, you can use a script to download, validate and decompress the kubeval executable.
Create a new folder named Kubeval, and add a file named use-kubeval.sh, with the contents shown below.
Since we’re downloading and running an executable from the internet, we should validate that we’ve actually got the executable we wanted, and not something strange from a hacker or a man-in-the-middle attacker. This can be done with the Linux utility sha256sum and a checksum file, so in addition to use-kubeval.sh, we need to add a checksum file to the Kubeval folder, named kubeval-checksum with the contents shown below.
When used as part of the use-kubeval.sh script, “sha256sum -c kubeval-checksum” will generate a SHA-256 hash for the file we downloaded, and compare it to a previously generated hash stored in kubeval-checksum. You should evaluate if this validation strategy is secure enough for your build system. New checksums can be generated for new versions of Kubeval with “sha256sum — tag bin/kubeval-linux-amd64.tar.gz”. If you want to use a longer checksum hash, you can use “sha512sum”.
Adding validation to the build pipeline
Now we need to add two steps to the azure-pipeline.yml pipeline definition. The first one executes use-kubeval.sh in the Kubeval folder, and the second one performs a strict validation of all YAML files in the CI folder.
Since we’re using Linux commands to download, verify and decompress the kubeval executable, you need to be using a Linux based vmImage like “ubuntu-latest” in your build pipeline.
Depending on whether you use Docker and where you put the Kubeval-steps in your pipeline, you might want to add the Kubeval folder to your .dockerignore, to avoid copying the contents into your Docker images.
A bit about security and upgrading Kubeval
As stated in the use-kubeval.sh script, you should check if there’s a newer version of Kubeval available, and upgrade if there is one. Using version v0.14.0 for all foreseeable future is probably a very bad idea 😅
Remember to generate a new checksum with “sha256sum --tag bin/kubeval-linux-amd64.tar.gz” for the new version of Kubeval.
