Planning Your Declarative Repository Structure with K8s
In the journey towards adopting GitOps, Devops/Release engineers need to be considerate of their software requirements(for both people and services). Knowing the needs of Software Engineers should be the driving force for GitOps adoption; at the end of the day, Devops/Release engineers are trying to increase the velocity of software delivery as well as decrease the Time to Resolve (TTR) incidents and their blast radius. Seems like an impossible task, but we should always be innovating and pushing the boundaries of developing/releasing software…
The Repository Structure
GitOps focuses heavily on creating a common language for Software Engineers and Release Engineers. Being able to make changes to an application that are explicit and visible to all team members makes Git the perfect solution — we can see exactly when changes are made, what changes were made, who made them, and who approved them.
As mentioned before, focusing on the needs of Software Engineers should always be our driving force for adopting new techniques to improve the Software Development Life Cycle (SDLC). Some considerations — Are services deployed in a single region or multi-region? What does the domain ownership model look like? Where/how do we deploy services? The answers to these questions will differ by company, however they still need to be considered!
Throughout this article, we will imagine that we are a business that delivers a web chat service, and we have two main services being stream pub and stream sub, which consists of Kubernetes Deployment, Service, and ConfigMaps for each. There will also be an application.yml for each service, which is a pointer to application specific YAML files (such as K8s manifests) as well as other application specific metadata
Pro-Tip: Do not bind your declarative repository with your code repository. There are significant performance implications to doing this and it makes webhook management overly difficult as CI pipelines often trigger as a result of a base trunk (ie main) mutating.
Domain-Focused Structure
In a domain-focused structure, we need to acknowledge ownership models and promote autonomous releasing. For this example we will assume that stream-pub and stream-sub are in the same domain called “chat”, and we will introduce a new domain called “notifications” which will have two services being stream-pub and stream-sub (same name was intentional as domains should operate autonomously). The below can be a structure of this when leveraging a monorepo:
/<domain>/<service>/application.yml
/<domain>/<service>/config-map.yml
/<domain>/<service>/deployment.yml
/<domain>/<service>/service.yml
CODEOWNERSIf we embraced a multi-repo strategy, then we could consider a repository naming convention that is reflective of each domain and flatten the above — but lets keep on the monorepo path. Let’s consider some domain-shared configurations, such as pod toleration's for nodes with specific taints
/chat/_base/patch/tolerations.yml
/chat/stream-pub/application.yml
/chat/stream-pub/config-map.yml
/chat/stream-pub/deployment.yml
/chat/stream-pub/service.yml
/chat/stream-sub/application.yml
/chat/stream-sub/config-map.yml
/chat/stream-sub/deployment.yml
/chat/stream-sub/service.yml
/notifications/_base/patch/tolerations.yml
/notifications/stream-pub/application.yml
/notifications/stream-pub/config-map.yml
/notifications/stream-pub/deployment.yml
/notifications/stream-pub/service.yml
/notifications/stream-sub/application.yml
/notifications/stream-sub/config-map.yml
/notifications/stream-sub/deployment.yml
/notifications/stream-sub/service.yml
CODEOWNERSOn first take, lots of engineers begin to panic because of ALL the YAML files they have to manage (hence why I call myself a YAML Janitor). 18 YAML files are needed to be kept up with, but notice how easily we could structure our CODEOWNERS for code/config ownership if this is a monorepo…it could reflect something like the below:
/chat/ @my-company/domain-chat
/notifications/ @my-company/domain-notificationsNote: I am not an advocate for monorepo
Multi-Region Structure
Let’s look at a repository structure that supports multi-region deployments to Kubernetes (K8s). We may have configurations for our services that are different per region, or we want to do update regions one at a time. The below could be a valid declarative repository structure:
/<service>/<region>/application.yml
/<service>/<region>/config-map.yml
/<service>/<region>/deployment.yml
/<service>/<region>/service.yml
CODEOWNERSImagine that we have 2 regions in AWS, for example us-east-1 and us-west-2. Let’s consider there are properties that will always be common between regions. Our repository could reflect the below:
/stream-pub/_base/common.yml
/stream-pub/us-east-1/application.yml
/stream-pub/us-east-1/config-map.yml
/stream-pub/us-east-1/deployment.yml
/stream-pub/us-east-1/service.yml
/stream-pub/us-west-2/application.yml
/stream-pub/us-west-2/config-map.yml
/stream-pub/us-west-2/deployment.yml
/stream-pub/us-west-2/service.yml
/stream-sub/_base/common.yml
/stream-sub/us-east-1/application.yml
/stream-sub/us-east-1/config-map.yml
/stream-sub/us-east-1/deployment.yml
/stream-sub/us-east-1/service.yml
/stream-sub/us-west-2/application.yml
/stream-sub/us-west-2/config-map.yml
/stream-sub/us-west-2/deployment.yml
/stream-sub/us-west-2/service.yml
CODEOWNERS2 services across 2 regions equates to 18 files in this example. There is beauty in this repetition though — for example if we wanted to make a change to ONLY the stream-pub service in us-east-1, then we can a Git Pull Request to only modify /stream-pub/us-east-1/deployment.yml. It’s obvious from the contents of this PR that we will ONLY be forcing a change to this single region and service, allowing for easy reviews for software engineers and it enables some interesting release tactics so we don’t cause a global outage with one deployment change, thus minimizing the blast radius from a deployment 😎
Multi-Region, Domain-Focused within a Monorepo
Quite the mouthful for a subsection, but now lets combine the two previous sections and see what yields:
/chat/_base/patch/tolerations.yml
/chat/stream-pub/_base/common.yml
/chat/stream-pub/us-east-1/application.yml
/chat/stream-pub/us-east-1/config-map.yml
/chat/stream-pub/us-east-1/deployment.yml
/chat/stream-pub/us-east-1/service.yml
/chat/stream-pub/us-west-2/application.yml
/chat/stream-pub/us-west-2/config-map.yml
/chat/stream-pub/us-west-2/deployment.yml
/chat/stream-pub/us-west-2/service.yml
/chat/stream-sub/_base/common.yml
/chat/stream-sub/us-east-1/application.yml
/chat/stream-sub/us-east-1/config-map.yml
/chat/stream-sub/us-east-1/deployment.yml
/chat/stream-sub/us-east-1/service.yml
/chat/stream-sub/us-west-2/application.yml
/chat/stream-sub/us-west-2/config-map.yml
/chat/stream-sub/us-west-2/deployment.yml
/chat/stream-sub/us-west-2/service.yml
/notifications/_base/patch/tolerations.yml
/notifications/stream-pub/_base/common.yml
/notifications/stream-pub/us-east-1/application.yml
/notifications/stream-pub/us-east-1/config-map.yml
/notifications/stream-pub/us-east-1/deployment.yml
/notifications/stream-pub/us-east-1/service.yml
/notifications/stream-pub/us-west-2/application.yml
/notifications/stream-pub/us-west-2/config-map.yml
/notifications/stream-pub/us-west-2/deployment.yml
/notifications/stream-pub/us-west-2/service.yml
/notifications/stream-pub/_base/common.yml
/notifications/stream-sub/us-east-1/application.yml
/notifications/stream-sub/us-east-1/config-map.yml
/notifications/stream-sub/us-east-1/deployment.yml
/notifications/stream-sub/us-east-1/service.yml
/notifications/stream-sub/us-west-2/application.yml
/notifications/stream-sub/us-west-2/config-map.yml
/notifications/stream-sub/us-west-2/deployment.yml
/notifications/stream-sub/us-west-2/service.yml
CODEOWNERSOh, hi YAML…all 38 files (39 less CODEOWNERS)! This structure, again, seems daunting. There are only 4 services across 2 domains and 2 regions. In a domain focused model, you can consider deploying a K8s cluster per domain, and multi-region doesn’t mean 1 cluster per region, resulting in the addition of another overlay for “k8s-context” — that’s even MORE YAML!
The benefit of the above structure is that we can easily identify when changes are targeted for a specific domain (ownership ✅) and we can reduce our blast radius of changes by targeting specific regions (granularity ✅). It’s obvious to the pull request reviewer what the impact is going to be, which helps to drive confidence and speed!
Wrapping Up
There are so many permutations to constructing a declarative repository and that’s why its incredibly important to take a moment to think about what your business needs. Too much YAML to manage may cost more, but not having enough YAML means there may not be enough flexibility to fit your business needs; the acronyms WYSIWYG (“What you see is what you get” — explicit) versus DRY (“Don’t repeat yourself” — implicit) are always at play when structuring your declarative repository strategically. My biggest rule of thumb for structuring declarative repositories, in an effort to increase Software Engineer’s release velocity, all the while minimizing impact from incidents is to be explicit. As momma always said, “It’s better to have and not need, rather than need and not have”. Good luck with your YAML!!!
