Automate K8s policy checks using Datree, AWS CodeBuild, and Terraform
Introduction
Gitops is one of the best strategies to optimize the lifecycle of applications in our Kubernetes clusters. Its basis consists in defining, declaratively, the desired state of applications and cluster configurations in a Git repository. Adopting a deployment workflow based on GitOps practices will allow us to have benefits including version control, more straightforward automation of security, audit, and compliance controls, stronger stability and consistency, faster deployments, and promotes collaborative contribution, to mention a few.
“What happens in manual configurations, stays in manual configurations.”
“Everything as code is the key point and GitOps practices the solution.”
Depending on the case, the versioned code in the repository could be Kubernetes applications defined in Kustomize, Helm Charts, or simply manifests in YAML files which will ideally be the source of GitOps continuous delivery tools focused on automating deployments to ensure the desired declared state. ArgoCD and FluxCD, for instance, support a variety of interesting multi-cluster and multi-environment deployment strategies based on branches, tags, folders, and more.
How to ensure high code quality before integrating changes to the main branch? That high code quality includes complying with security policies as well as syntax and semantic validations.
Well, we could just trust… or better, we could trust, but verify in an automated way. 🚀
Let’s explore one of the infinite alternatives to automate these validations through GitOps practices and of course using some GitOps tools.
The following sections will be reviewed in this article:
- Overview of the Architecture
- Prerequisites
- Implementation
- Considerations
- Conclusions
Overview
The magic alternative satisfies the following requirements:
- A Git repository with versioned K8s manifests in the main branch.
- An automated way to validate new changes before integrating them into the main branch.
- If the automatic validations fail, the new changes cannot be integrated into the main branch.
- Plus: A way to easily replicate automation resources across multiple K8s repositories
As illustrated, the flow consists of the following steps:
- Teams of developers or operators create feature branches in the Git repository based on the main branch, with the necessary contribution to the K8s manifests. The main branch is a protected branch, so they must create a Pull Request from the feature to the main.
- When the PR is created to the main branch, an AWS CodeBuild project is triggered which analyzes the code from the feature branch using the Datree CLI.
- The details of the analysis result are sent to the Datree cloud backend and displayed in a policy check dashboard. Or you can simply view them in the CodeBuild project logs.
- If the Datree policy checks pass then AWS CodeBuild will return a successful check on the PR, otherwise, a failed check will be returned, and the PR cannot be merged.
- All automation resources provisioned in AWS as the CodeBuild project have been abstracted into a Terraform module, which will allow us to reuse and replicate these configurations for any K8s repository.
With this in mind, let’s clarify a bit, some GitOps tools and services used.
Datree
Datree is a tool focused on validating K8s manifests and combined with its CLI allows us to automate policy checks in our K8s Pipelines. The validation process consists of three phases: YAML Validation, Schema Validation, and Policy Check. In addition, the Datree concept of Centralized policy enables the option to validate policies with built-in rules and even our own custom rules.
Some similar alternatives are Kube-score, Kubeval, Conftest, and Trivy.
AWS CodeBuild
AWS CodeBuild has been the AWS DevOps service chosen to execute Datree validations when a Pull Request is created to the main protected branch through a webhook between the CodeBuild project and the git repository.
Depending on whether the result of the build is successful or failed, CodeBuild natively supports returning the check to the Pull Request in GitHub.
Terraform
As part of the “Everything as code” concept and in favor of the famous DRY principle, all resources provisioned on AWS have been abstracted into a Terraform module. Therefore, we could replicate CodeBuild Projects for all the K8s repositories we have. As an IaC tool, you will now get Pipelines as code. 🚀
Git repository
Up to this point, only the K8s manifests repository has been mentioned. This is a GitHub repository. However, you could use any source supported by AWS CodeBuild (Such as GitHub, GitHub Enterprise, AWS CodeCommit, or BitBucket)
Equally important, the Terraform module for provisioning AWS resources mainly the CodeBuild project to execute Datree policy checks, has its own GitHub repository. Thus, there are two GitHub repositories:
Prerequisites
- Terraform v1.0 or earlier
- AWS Cloud Account
- A Git repository with K8S Manifests. A GitHub repo in this case.
- Configure GitHub Authentication for AWS CodeBuild
- Datree account. Take a look at the official Datree Getting started or create the account directly.
Let’s start!
The alternative solution will be covered in the following 5 steps:
- Create the K8s Manifests repository.
- Datree pre-configuration.
- Terraform module structure and provisioning.
- Final GitHub Configurations.
- 1,2,3… Action! 🎬 Result of policy checks.
Step 1: Create the K8s Manifests repository
As you can see in the K8s manifest repository structure, there are two Kubernetes applications with their manifests declared as YAML files.
apps
├── demo-app
│ ├── base
│ │ ├── deployment.yaml
│ │ ├── ingress.yaml
│ │ ├── kustomization.yaml
│ │ ├── namespace.yaml
│ │ └── service.yaml
│ └── overlays
│ └── develop
│ ├── config.json
│ └── kustomization.yaml
└── lab-app
├── base
│ ├── deployment.yaml
│ ├── ingress.yaml
│ ├── kustomization.yaml
│ ├── namespace.yaml
│ └── service.yaml
└── overlays
└── develop
├── config.json
└── kustomization.yamlSo, the idea is to analyze these existing and future manifests when any developer or operator wants to contribute via pull request.
Step 2: Datree preconfiguration
Once you have created the Datree account and finished the official Datree Getting started detailed in the Prerequisites section, a good security practice is to create a new read token which the CodeBuild project will use later to access the Datree backend.
Go to the Datree Web Console > Settings > Token Management and then click on Create Token. Make sure the token type is Read.
To store this token securely and ensure that the CodeBuild project can take it, load the value as a SecureString in SSM Parameter Store.
/k8s-apps-example/datree/APP_TOKEN 
To interact with Datree’s Centralized policy console, let’s create a new policy for this example. Go to the Policies tab and then create a new policy named k8s_apps_example. Please note this name as it will be used later for the CodeBuild project.
Now it’s time to activate the rules that will be automatically validated in the CodeBuild project by the policy check. As illustrated above, 22 rules out of 53 have been activated, some of which belong to the Security, Argo, and Containers categories.
Step 3: Terraform module structure and provisioning
First, clone the Terraform module repository and take a look at the resources that will be provisioned.
- CodeBuild Project
- CodeBuild Webhook
- Webhook SSM Parameters (Payload URL and Secret)
- CodeStar Notification
In addition to the resources created by the module, the following resources are required as input to the module invocation.
In the main.tf file of the complete example, both the required resources and the invocation of the Terraform module are declared.
The module supports many input variables; however, the most relevant variables are a prefix for the resource names, the repository, and branch to be analyzed, the Datree policy name, and the parameter store path where the Datree token was previously created.
With the repo cloned, set up the AWS credentials, and let’s move on to the terminal and see the Terraform magic.
cd examples/complete
terraform init
terraform apply Expected output after entering yes in terraform apply:
Finally, the AWS CodeBuild project has been provisioned. 🎉
The buildspec.yml file is built from a terraform template, the final result is shown below. As you can see, first all YAML files are listed recursively, and then the datree testcommand is executed including the --only-k8s-filesflag to validate all the YAML files related to K8s manifests recursively from the repository root.
Similarly, the CodeBuild webhook has also been added to the K8s manifests repository.
Step 4: Final GitHub Configurations
To be sure that developers or operations team can only contribute to the main branch via Pull Request and CodeBuild check must pass before feature branches can be merged into the main branch. In the K8s manifests repo, create a protected branch rule associated with the main branch and select the CodeBuild check in the Status checklist.
Step 5: 1,2,3… Action! 🎬 Result of policy checks
Finally, if we create a Pull Request from a feature branch to the main branch with some changes, automatically a CodeBuild is triggered to analyze the code of the feature branch.
Depending on the status of CodeBuild execution, AWS will return to the PR the status of a failed or successful check. In this case, as the check failed, then merge is not allowed.
To detail the build status, just click on the Details label, and it will redirect you to the CodeBuild logs on AWS or directly access the run history of the CodeBuild project.
In the CodeBuild logs, you can detail the following:
- Rules that have not been passed based on the rules enabled in the Datree policy created earlier.
- A summary of the 3 consecutive phases, YAML Validation, Schema Validation, and Policy Check.
- The total result of rules evaluated, skipped, failed, and passed in the policy check phase.
In the same way, if you access the History tab in Datree, you can easily track all the different executions of policy checks.
Keep in mind
- It is not necessary to create a Datree account to run the Datree CLI, however, to access the Centralized policy features the datree account is required.
- Datree’s free model supports only GitHub & Google SSO users so it is best to create a new automation user or consider the enterprise model.
- As mentioned before, in this example all k8s manifests in the repository are being validated. However, Datree supports the validation of applications in Helm Charts and Kustomize.
- The terraform module supports the creation of CodeStar Notifications to send status to slack channels. This requires creating a chatbot notification and adding it as an input variable.
- When it is a Github.com repository the webhook is created automatically by Codebuild through the personal access token. However, when it is a GitHub Enterprise repository it must be created separately. For this reason, the Payload URL and Secret of the webhook are stored in SSM Parameter Store.
Conclusion
Throughout this article, we were able to explore a solution based on GitOps practices to automate the analysis of the manifests of our K8s applications declared in a git repository before they are deployed. The implementation relied on some GitOps tools such as Datree to validate K8s misconfigurations through policy checks, AWS Codebuild to automatically trigger checks, Terraform Modules for abstraction and provisioning of infrastructure related to this solution, and lastly, some configurations in GitHub to ensure a Pull Request-based flow.
And finally, I would like to express my deepest appreciation to Yorman and Don Oscarín for their valuable support.
