Is GitOps actually useful?

I was thinking about the end-to-end process of applying configuration to Kubernetes in early 2017, around the time that Alexis Richardson and WeaveWorks were creating the concept of GitOps. I worked with Alexis on the CNCF Technical Oversight Committee, so he gave me a heads up that they were working on this. I loved the concept from the beginning.

Since those early days of GitOps, I’ve seen a lot questions regarding how GitOps is different from CI/CD (or “CIOps”), what the value of GitOps is, and where it makes sense to use GitOps. In many cases, any git-triggered deployment mechanism is characterized as GitOps, which is understandable from the early description as “operations by pull request.”

So what is GitOps? GitOps is a design pattern for continuous reconciliation of desired state and live state using version control to manage declarative specification of the desired state and automation to make that desire a reality. It extends the scope of declarative management to encompass the continuous deployment process, enabling operators to focus on WHAT the state should be as opposed to HOW pipelines, scripts, and deployment tools are changing the state. It also provides continuous drift detection (and optionally remediation) between the desired state and live state, and appropriate breadcrumbs can make it easy to trace back to the source of truth for the desired state. GitOps is similar in several ways to the Annealing intent-based automation system used within Google. It is also kind of like Puppet, but for Kubernetes clusters.

OpenGitops defines the key characteristics of GitOps as: (1) declarative, (2) versioned and immutable, (3) pulled automatically, and (4) continuously reconciled. Note that “git” is not mentioned. One could use other versioned storage. I’m a fan of OCI (container) images. Git workflow is unnecessarily tedious. “GitOps” is a catchier name than “StoreOps” or “ImageOps”, though. But I digress.

Let’s focus on properties 1, 3, and 4. Those familiar with Kubernetes will recognize this as the controller pattern. It’s how Kubernetes itself works and has proven to be robust, resilient, and widely applicable. GitOps adds another source of truth, in addition to Etcd.

Let’s also start with the last question, where GitOps makes sense. The most obvious scenario is the retail edge use case popularized by Chick-fil-A: hundreds or thousands of deployment targets on separate private networks with sometimes low network bandwidth and not-so-high network and runtime availability. Because targets are so numerous, deployments need to be parallelized. Because some targets are likely to be unavailable during deployments, deployments need to be retried, performed out of order, left running in the background, and so on. Fixed-stage pipelines are too brittle in this situation, but the controller pattern is perfect. Additionally, pulling the configuration is simpler from a networking and credentials / permissions management perspectives — the targets just need read-only access to the single source of truth.

Are there benefits from using GitOps for other use cases? I’m admittedly not a user, but what I see as the main benefit is simplifying the deployment model. The reconciliation process is fairly constrained compared to open-ended pipelines and deployment orchestrators.

A consequence of this constrained model is that more complex processes are shifted left, to manipulations of the desired state in storage, and/or shifted right, to other controllers in the cluster, similar to the Kubernetes Deployment controller, which moved rolling update functionality from the command-line tool to the control plane in Kubernetes 1.2, both of which are more introspectable than pipelines. I can observe, validate, query, and analyze the desired state in storage, and see what the reconciliation status is overall and drill down to a granular level if I need to. APIs are easier to build automation around than scripts and log files.

I don’t think the potential of GitOps has been fully realized yet, though. It’s partly hampered by complex declarative configuration formats. Some users render WET configuration back into storage to mitigate this issue, but that doesn’t make edits easier. Information about failures and obstacles encountered during the deployment process also need to be surfaced more consistently to humans and/or higher-level processes.

GitOps doesn’t solve all deployment problems or even cover the entire deployment process, but it’s a solid foundational building block.

What do you think? Do you find GitOps to be useful? What do you see as its disadvantages?

Feel free to reply here, or send me a message on LinkedIn or X/Twitter, where I plan to crosspost this.

If you found this interesting, you may be interested in other posts in my Infrastructure as Code and Declarative Configuration series.