Shift-Left validation with Verrazzano
Verrazzano Installation Profiles are files written using YAML language following a specific install.verrazzanio.io api version linked to a schema. This schema by itself is a reference on how to declare the constraints and relationships between the resources that will be created by Verrazzano Operator on a Kubernetes Cluster. Because of the nature of YAML language, one might argue that YAML files can be painful to edit. This difficulty grows as the rules in a schema gets larger, which can lead to misconfigured sections. These sections can be hard to find and diagnose. This blog will illustrate a common practice, known as shift-left static validation. Shift-left validation ensures your YAML manifest files are valid before deploying to Kubernetes.
Kubernetes community has developed several tools to do static validation. I am going to walk you through how to use Datree and kubeconform tools using the Verrazzano Installation Profiles as input. Note that this is just a subset of all the validation tools that are available in the open-source community. Whichever tool you choose, it is important the tool supports Custom Resource Definitions.
Let’s go over the list of pre-reqs:
- Download the latest Verrazzano Custom Resource Definition (CRD) from the Verrazzano public repository at Verrazzano.io: install.verrazzano.io_verrazzanos.yaml
$> curl -LO https://raw.githubusercontent.com/verrazzano/verrazzano/v1.4.2/platform-operator/helm_config/charts/verrazzano-platform-operator/crds/install.verrazzano.io_verrazzanos.yaml- Most static validation tools require openAPI specs to be in json format to do their magic. We will need -enters… openapi2jsonschema.py - script from kubeconform to transform Verrazzano specs to json format. This is developed in python, so, guess what? we need python as a pre-req also.
$> wget https://github.com/yannh/kubeconform/blob/master/scripts/openapi2jsonschema.py
$> python --version
Python 3.9.13
$> python openapi2jsonschema.py
- Finally, the already famous kubectl . check the version that is compatible with the Kubernetes cluster your are planning to install Verrazzano on.
Note: I am a Linux user. Steps detailed in this blog post were tested on my Linux distro, but you should be able to make it work with other OS too.
Ready? Let’s start with kubeconform
[ Kubeconform ]
As their team describes it
“a Kubernetes manifest validation tool” that “enables validating Kubernetes custom resources (CRDs) and offline validation capabilities” .
Sounds exactly what we are looking for. Proceeding to install it.
$> wget https://github.com/yannh/kubeconform/releases/download/v0.5.0/kubeconform-linux-amd64.tar.gz
$> tar -C ~/tools/bin -xzf kubeconform-linux-amd64.tar.gz
$> kubeconform -v -verbose
v0.5.0Next Step, I need to transform install.verrazzano.io_verrazzanos.yaml spec file listed in the pre-reqs section to json format. The resulting conversion will create two json files, each file corresponding to a different Verrazzano API version.
$> python openapi2jsonschema.py install.verrazzano.io_verrazzanos.yaml
$> cd /tmp
$> ls -l
total 324
-rw-r--r--. 1 opc opc 321534 Feb 3 20:15 install.verrazzano.io_verrazzanos.yaml
-rw-rw-r--. 1 opc opc 6746 Feb 3 20:16 openapi2jsonschema.py
$> python openapi2jsonschema.py install.verrazzano.io_verrazzanos.yaml
JSON schema written to verrazzano_v1alpha1.json
JSON schema written to verrazzano_v1beta1.jsonSuper! we have json files generated. verrazzano_v1alpha1.json and verrazzano_v1beta1.json
Back to Verrazzano Installation Profile. There are at least 15+ components you can customize in Verrazzano from the get-go. For example, opensearch’s number of replicas, index state management policies, jaeger, DNS, prometheus, istio, authproxy, keycloak, etc. and each component has its own set of attributes that can be updated based on a given environment. If you want to install a Dev environment, you could go with a basic installation profile, but if you are thinking about production environment, your mind could take you to configure them for High Availability, certain policies, and so on, leading to a fully customized Installation Profile.
All those possible permutations on a YAML format is error-prone business. Luckily, I will be customizing DNS and Certificates components only to illustrate how to spot errors in an Installation profile as follows:
- DNS: Verrazzano supports seamless integration with Oracle Cloud Infrastructure DNS service to create DNS records automatically for each accessible component, including deployed apps. I will configure ociConfigSecret, dnsCompartmentOCID, dnsZoneOCID, and dnsZoneName
- Certificates: Verrazzano supports generating SSL certificates automatically with LetsEncrypt, among other options. For this use case, one needs to provide a value for emailAddress property to receive a challenge question.
My Verrazzano installation Profile file named install_custom_dns_certificate.yaml is taking shape. Again, more examples can be found here
$> cat install_custom_dns_certificate.yaml
apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
name: example-verrazzano
spec:
environmentName: env
profile: prod
components:
certManager:
certificate:
acme:
provider: letsEncrypt
emailAddress: emailAddress@example.com
dns:
oci:
ociConfigSecret: oci
dnsZoneCompartmentOCID: dnsZoneCompartmentOcid
dnsZoneOCID: dnsZoneOcid
dnsZoneName: my.dns.zone.name
ingressNGINX:
type: LoadBalancer…we are getting closer to the action. I promise.
We have all inputs required: a valid install file and json spec file. Now it is time for Kubeconform to tell me if it is valid or not.
Kubeconform can validate files against Kubernetes API cluster and/or offline CDRs. In this case, I will be enabling the offline option by using — schema-location flag and specifying the spec file location.
Here we go… Let’s validate our install file.
$> INSTALL_FILE=install_custom_dns_certificate.yaml
$> SPEC_FILE=verrazzano_v1beta1.json
$> kubeconform -summary -verbose -debug -schema-location $SPEC_FILE $INSTALL_FILE
2023/02/04 00:40:46 using schema found at verrazzano_v1beta1.json
install_custom_dns_certificate.yaml - Verrazzano example-verrazzano is valid
Summary: 1 resource found in 1 file - Valid: 1, Invalid: 0, Errors: 0, Skipped: 0Is a valid file.. Phew!
Now, create a copy of the install_custom_dns_certificate.yaml and replace the word ingressNGINX to ingress -something that someone would possibly mistyped — and saved it with a new name “install_custom_dns_certificate_error.yaml”, and run it again.
$> kubeconform -debug -verbose -schema-location verrazzano_v1beta1.json install_custom_dns_certificate_error.yaml
2023/02/04 00:43:58 using schema found at verrazzano_v1beta1.json
install_custom_dns_certificate_error.yaml
- Verrazzano example-verrazzano is invalid:
For field spec.components: Additional property ingress is not allowed
Summary: 1 resource found in 1 file - Valid: 0, Invalid: 1, Errors: 0, Skipped: 0There it is… Kubeconform not only list it as an invalid YAML file, but also pointed exactly to the property that triggered the error. “Additional property ingress is not allowed”.
[ Datree ]
Another good alternative to do static validations is open source community tool Datree.
Datree secures your Kubernetes by blocking the deployment of misconfigured resources.
Steps to install it are quite simple. Download the latest binary and it will self install in your current path.
$>curl https://get.datree.io | /bin/bash
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2218 100 2218 0 0 28075 0 --:--:-- --:--:-- --:--:-- 28075
Installing Datree...
[V] Downloaded Datree
[V] Finished Installation
Usage: $ datree test ~/.datree/k8s-demo.yaml
Using Helm? => https://github.com/datreeio/helm-datree
Using Kustomize? => https://hub.datree.io/kustomize-support
Run 'datree completion -h' to learn how to generate shell autocompletions
$>datree version
1.8.21Tool ready. Now I am going to start with the Install Verrazzano file containing mistyped “ingress”. But this time, I will introduce another error under certManager section, to mistype the acme LetsEncrypt protocol name to acmeERROR. Note that this is two levels down into component name certManager, as opposed to first level error for the ingress name.
apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
name: example-verrazzano
spec:
environmentName: env
profile: prod
components:
certManager:
certificate:
acmeError:
provider: letsEncrypt
emailAddress: emailAddress@example.com
dns:
oci:
ociConfigSecret: oci
dnsZoneCompartmentOCID: dnsZoneCompartmentOcid
dnsZoneOCID: dnsZoneOcid
dnsZoneName: my.dns.zone.name
ingress:
type: LoadBalancerGreat, so Datree has multiple options to run validations. It can pull yaml files directly from a repository, use another community tool -kustomize- to render new patched yaml files, and test flag, which is what will be focused on to do static validations. Similar to kubeconform, it uses — schema-location flag to provide an offline path to specific schemas.
A first run on the Verrazzano install file (the one with errors) reported correctly two errors.
datree test --schema-location verrazzano_v1beta1.json install_custom_dns_certificate_error.yaml
>> File: /tmp/install_validation/install_custom_dns_certificate_error.yaml
[V] YAML validation
[X] Kubernetes schema validation
❌ k8s schema validation error: For field spec.components: Additional property ingress is not allowed
❌ k8s schema validation error: For field spec.components.certManager.certificate: Additional property acmeERROR is not allowed
[?] Policy check didn't run for this file
(Summary)
- Passing YAML validation: 1/1
- Passing Kubernetes (1.20.0) schema validation: 0/1
- Passing policy check: 0/1
+-----------------------------------+------------------------------------------------------+
| Enabled rules in policy "Default" | 21 |
| Configs tested against policy | 0 |
| Total rules evaluated | 0 |
| Total rules skipped | 0 |
| Total rules failed | 0 |
| Total rules passed | 0 |
| See all rules in policy | https://app.datree.io/login?t=bPTwhTQfzZFfm4m9TaNFea |
+-----------------------------------+------------------------------------------------------+It nicely displays a break down of the polices ran — some out of the box- and a Kubernetes schema validation section marked with an X, indicating a schema validation test has failed. This is an expected result. But I was surprise that it didn’t stop on the first error and exited. It completely validated the file and reported the second error. Nice Job!
For our final act, let’s validate our first Installation Profile.
datree test --schema-location verrazzano_v1beta1.json install_custom_dns_certificate.yaml
(Summary)
- Passing YAML validation: 1/1
- Passing Kubernetes (1.20.0) schema validation: 1/1
- Passing policy check: 1/1
+-----------------------------------+------------------------------------------------------+
| Enabled rules in policy "Default" | 21 |
| Configs tested against policy | 1 |
| Total rules evaluated | 21 |
| Total rules skipped | 0 |
| Total rules failed | 0 |
| Total rules passed | 21 |
| See all rules in policy | https://app.datree.io/login?t=bPTwhTQfzZFfm4m9TaNFea |
+-----------------------------------+------------------------------------------------------+Assessment: Valid YAML file. ;-)
I personally like datree better over kubeconform, just because the development team has done a better job of printing this sort of results.
What is Next?
Both kubeconform and Datree are great tools to help validate Verrazzano Install Profile files before you attempt to install Verrazzano. These basic tests can be enhanced further depending on your rules, requirements and corporate policies. But as long as you can run static validation at an early stage in your workflow, it will help you catch simple typos, save you time, and lot’s of hair pulling when dealing with long YAML files.
You are unblocked now! Go ahead and install Verrazzano. But before you proceed, I suggest reading Introducing Verrazzano CLI blog -authored by great friend- and we will meet again on my next post on How to validate applications YAML files based on Open Application Model specification adopted by Verrazzano.
