Easily Protect Your Kustomize Deployments With Spectral CLI

Recently, we started deploying our apps with ArgoCD and kustomize.
With our learning curve using a new system, we had our issues with failed sync attempts due to broken YAML configuration files or inconsistent deployments.

We looked for a tool to use as a rail guard and ended with Spectral.

What is spectral?
Spectral is an open source tool to monitor, classify, and protect your code, assets, and infrastructure for exposed API keys, tokens, credentials, and high-risk security misconfigurations in a simple way, without noise.

First time using spectral (source)

Getting started

1. You will need to register to spectral and get your DSN.
2. Export your DSN as an environment variable.
3. CD to the path with the YAML you want to scan.
4. Run Spectral.
5. If you don’t want to upload the result of the scan to spectral.ops (their SaaS) use the ‘ — nosend’ flag.

Spectral and kustomize

Spectral does not have built-in support for kustomize, but it is still possible to use kustomize with Spectral. To do this, you will need to run kustomize separately before running Spectral. This will allow you to use kustomize to create our k8s YAML files, and then run Spectral to validate and lint those files. Essentially, you will need to perform two separate steps in order to use kustomize and Spectral together: first running kustomize, and then running Spectral to validate the resulting configuration files.

source

Spectral and kustomize

Basic CI/CD

In our CI/CD pipeline, we have a test step that uses kustomize to convert kustomization files into k8s YAML files. After the kustomization files have been converted, we then run spectral on the resulting YAML files to validate them. This is our primary script for running spectral on the kustomize-converted files. Essentially, we are using kustomize to create our k8s YAML files, and then running spectral to ensure that the resulting files are well-formed and adhere to best practices.
Source

Main Ci script

Important note:

If you have a large number of kustomize files that need to be converted, the process of converting them may take a significant amount of time. One way to streamline this process is to create a single kustomization file that combines or aggregates the other kustomization files into a single file. This will allow you to more easily and efficiently manage and convert the configuration for all of your resources.

kustomization.yaml aggregator

We did it per region.

.
└── Source/
    ├── eu-west-1/
    │   ├── kustomization.yaml # aggregator kustomization.yaml
    │   ├── service-1/
    │   │   ├── deploy.yaml
    │   │   ├── hpa.yaml
    │   │   └── kustomization.yaml
    │   ├── service-2
    │   └── service-3
    └── eu-west-2

Improving CI/CD performance

n order to make running spectral in our CI/CD pipeline as efficient as possible, we have created a custom Docker image that includes all of the necessary dependencies for both spectral and kustomize. This allows us to easily and quickly run spectral on our kustomization files without having to manually install or configure any dependencies on the runner.

By creating this custom image, we have reduced the time the pipeline ran from 2 minutes to 30 seconds(!), so consider creating your own image with all the needed tools.

A base image can look like this:
source

Dockerfile

Important note:

Do NOT use alpine image as your base docker image because spectral has issues running on it.

Our job step
Source

Job step

Summary

By quickly adding Spectral to our CI/CD pipeline, we were able to protect from misconfiguration issues of our Kustomize files. Hope you’ll find this manual helpful.

Feel free to drop off comments here for further questions.

Best,

Isar