GitOps: A Brief Introduction
Introduction
Have you ever fixed a random misconfiguration issue in production? Let me guess: it was super urgent, so you issued some commands to make things work as quickly as possible. Only for it to up again on re-deploy because you forgot to persist the fix? Fear not, you’re not alone! There is a method that streamlines deployments, boosts collaboration and cuts down on errors, and it does so by moving away from an imperative approach to making changes, to a declarative one: GitOps.
In this article, we will look at what GitOps is as a practice and define its core concepts. We will explore how GitOps enhances development lifecycle, and how each of the core principles of GitOps maps to different stages of the GitOps Flow. Finally, we will consider both the benefits and common challenges of implementing GitOps. By the end of this article, readers should be able to both understand the concepts and also start evaluating how well GitOps fits within their team or organization.
GitOps Defined
GitOps is a modern approach to continuous delivery best suitable for cloud-native environments built around Kubernetes. It leverages Git repositories (the hint was in the name, right?) to define, store and manage infrastructure and application definitions and configurations. As such, it is applicable to all major aspects of a typical software delivery ecosystem. From Infrastructure-as-code (IaC), configuration management, CI/CD pipelines, to actual application definitions and manifests, GitOps can be viewed as a universal, structured way of reasoning about systems. Once we have a clear mechanism for describing all aspects of our systems, we can treat (nearly) everything as code, and store the desired state as a collection of files within a Git repository. This notion is often referred to as “Everything-as-Code”, and is at the core of GitOps.
GitOps Principles
As the name suggests, GitOps is all about the automation of operational tasks and processes based on the declarative state stored in Git.
There are four key GitOps principles. Two focus on utilizing Git for declarative state management, while the other two center around automating operational tasks and processes based on that state.
Let us define and discuss the core GitOps principles. A team or organization looking to implement GitOps should focus on the following four pillars:
- Declarative Configuration
- Versioning and Immutability
- Desired State Automation
- Continuous Reconciliation
Declarative Configuration mandates defining the system’s desired state via files stored in one or more Git repositories. This includes infrastructure definitions, configurations, application code, and deployment manifests. Declarative configuration replaces the imperative model of issuing commands to define and update our system, ensuring consistency and reproducibility.
Using Git for storing the desired state naturally leads to the second pillar of GitOps: Versioning and Immutability. Git’s version control capabilities capture changes for historical and audit trail purposes. Git also facilitates reviewing and approval of changes to the desired state. Similarly, Git can also be used to revert any erroneous or unwanted changes introduced to the state of our system.
Git repositories act as the single source of truth about our system, enhancing transparency and accountability. Git-driven controls and automations ensure consistent changes, thus reducing errors and time-consuming manual interventions.
The real strength of using Git repositories as the single source of truth is in automating the propagation of changes to target environments. This introduces the third pillar of GitOps: Desired State Automation. Since GitOps is a natural fit for cloud-native environment setups on top of Kubernetes, we can use Kubernetes itself to automatically apply the desired state changes. To do this, we utilize specialized controllers that act as GitOps agents watching the GitOps repository and comparing the desired state in the repository with the actual state of the cluster. Some of the most popular open-source GitOps agents include ArgoCD and Flux.
GitOps agents not only automate application of desired state, but also continuously reconcile desired and actual states. Through a Continuous Reconciliation loop, these agents compare repository changes with the cluster’s state. This reconciliation process can be further enhanced with web hooks that can notify the agents of GitOps repository changes almost immediately. This makes reconciliation more responsive and minimizes the amount of time the cluster is in a potentially unwanted or drifted state.
Continuous reconciliation ensures automatic undoing of changes introduced via imperative commands, thus promoting immutability and enforcing structured change management through Git review and approval processes. In case of errors introduced manually or imperatively, the reconciliation loop acts as a self-healing mechanism, restoring the cluster to the desired, working state.
GitOps Workflow
With the theory behind the GitOps principles out of the way, let us now delve into how their implementation enhances a typical Develop → Build + Test → Deploy lifecycle. The relationship between GitOps principles and the GitOps workflow is illustrated below.
Details worth pointing out include:
- Commits to application repositories trigger an automated build + test CI/CD pipeline. Upon successful test execution, a container image is built, tagged and pushed to the container registry.
- CI/CD system does NOT imperatively deploy the built application, but rather commits a configuration change to the GitOps repository, updating the image tag in the deployment manifest with the one that was just built.
- GitOps Agent is configured to watch for changes within the GitOps repository and apply them automatically. It will detect that the desired state in the GitOps repository has changed and no longer matches the actual state within the cluster, since the running image tag does not match the desired (newly built) image tag.
- GitOps Agent then figures out what all needs to be changed (in this case, just the image tag reference of the application deployment, and then applies the changes to the underlying Kubernetes resources.
- Kubernetes will pull the specified image the from the container registry and replace the running instance of the application with a new version.
- GitOps Agent will continuously monitor for drift between the actual and desired state of the cluster and run the reconciliation loop to ensure the state of the cluster matches the state declared within the GitOps repository.
A similar flow kicks in when the GitOps repository is directly updated by committing changes to other configuration-as-code or infrastructure-as-code files. Pipeline implementation will differ, of course, but the changes can still be applied automatically by an agent, running either remotely or within the cluster.
While the GitOps agent commonly runs inside the Kubernetes cluster it operates on, it is also possible to configure a single agent to communicate with remote clusters. Such setup allows for centralized management and orchestration of changes across multiple clusters and environments.
GitOps Benefits
Implementing a solid GitOps practice has many benefits for the entire organization, albeit most notably for dev, ops, and platform teams. While many were either mentioned or hinted at while discussing individual GitOps principles, here are some additional benefits worth highlighting:
- Familiarity: Git and related flows and processes are familiar to most technical personnel, regardless of the technology stack in question
- Technology stack agnostic: if it can run on Kubernetes and be expressed declaratively, GitOps can be used. Whether we’re dealing with microservices or monoliths, Java, Ruby or Python, Pulumi, Terraform or something else entirely — GitOps setups are inherently agnostic to the technology stack of choice.
- Vendor-neutral: because Git, a very popular open-source version control system, is at the core of GitOps, GitOps setups are also often vendor-neutral and flexible. If at some point we wish to move from BitBucket to GitLab, for example, the migration path is often much more straightforward with GitOps than with an imperative-driven setup, where we would need to worry about vendor-specific configurations, pipeline syntax and more.
- Support for multiple Kubernetes-oriented tools: GitOps agents typically support more than one Kubernetes-oriented tool; for example, in addition to kubectl, one can easily utilize Helm, Kustomize, Jsonnet, or even make use of custom tooling, like SOPS or Hashicorp Vault, all in a GitOps-friendly manner.
- Improved security and reliability: In an imperative approach, CI/CD system often has direct access to the underlying cluster or cloud environment, which can be problematic from a security standpoint. With GitOps, an agent typically runs within the cluster and thus the risk of exposure is smaller. At the same time, because all changes are reviewed and approved before actual rollout, risk of mistakes is often reduced. As mentioned before, undoing mistakes can be done as simply and as quickly as reverting a problematic Git commit. All of this helps improve system reliability over time.
- Enhanced developer experience, speed & frequency of deployments: because GitOps relies on automation at every possible step in the lifecycle, it often gives way to noticeable productivity gains, leading to more frequent deployments and better developer experience overall.
- Increased visibility & collaboration: storing nearly everything in Git increases visibility, which in turn empowers team members to better comprehend the system, suggest improvements through pull requests and reviews, and scrutinize the overall system more thoroughly than they could otherwise.
GitOps Challenges
Like any technology-related concept, GitOps also comes with its own set of challenges. Here are some drawbacks and challenges worth noting:
- Structuring GitOps repo(s): storing everything in a single repository, while potentially appealing at first, is almost always a bad idea. On the other hand, being too granular can lead to too many GitOps repos at play. Additionally, planning repository structure for multiple environments, clusters, regions or cloud providers can be difficult and requires careful considerations
- Git conflicts: Automating nearly everything can sometimes lead to Git conflicts. When multiple pipelines write to the same repository simultaneously, Git conflicts can disrupt the whole process. In such scenarios, pipelines or other mechanisms need to be able to either avoid or resolve conflicts seamlessly.
- Secrets management: Managing secrets can be a significant challenge with the GitOps approach, since it mandates storing everything, including confidential information, as files in Git. This in turn requires encrypting sensitive files before storing them in Git. Consequently, the chosen GitOps agent must be able to decrypt these files dynamically in order to create Kubernetes secrets efficiently.
- Complexity at scale: In addition to structure of the GitOps repositories, scaling out GitOps practices to multiple environments, clusters, regions and/or cloud providers introduces a unique set of challenges from an orchestration point of view
- Cost: An overly complex GitOps setup, whether due to excessive tooling, infrastructure, or maintenance overhead, can result in increases in costs over time. While this is typically not directly related to GitOps in and of itself, it’s crucial to note that adopting GitOps practices does not inherently shield against cost escalation.
Conclusion
GitOps is a natural fit for cloud-native, Kubernetes-oriented environments. It offers many benefits over other alternatives, such as advanced degrees of automation, drift detection and reconciliation, easy audit trail and history of changes, quicker rollbacks, enforcement of code reviews and approvals, and faster and more frequent deployments. While not without its challenges, a carefully considered and continually improving GitOps practice has a lot to offer to both technical and business teams within any organization embracing cloud-native and Kubernetes.
Useful Links
Interested in learning more about GitOps and related concepts? Check out these links to explore more:
