The GitOps Pipeline — Part 2

Alexis Richardson, CEO & Founder, Weaveworks is the original author of this blog

This is my second post on GitOps.

GitOps is short for “Git Ops”. It is a name for a set of Ops practices using Git. The spirit of the name is to be like DevOps — something that anyone can do rather than being “a product”. Read my first blog post in this series to understand more about “GitOps — Operations by pull request”.

What is GitOps

GitOps is a way to do Continuous Delivery. It works by using Git as a source of truth for declarative infrastructure and applications. Automated delivery pipeline automatically roll out changes to your infrastructure when changes are made to Git. But the idea goes further — using tools to look at the actual production state, and tell you when what’s source code doesn’t match the real world, giving you the ability to spot the differences and fix problems accordingly. In other words GitOps extends pipelines with a feedback loop for observing and controlling the system .

At Weaveworks, we implement these principles in our product, Weave Cloud. This not only helps customers ship apps faster, it also helps us run our own cloud native stack. I want to present how we do this, identify best practices and tools, and showcase some of Weaveworks’ use cases. In this post I shall describe the GitOps pipeline. In a later post I shall talk about the GitOps observability, feedback and control.

Wait, but why GitOps?

GitOps empowers developers to do operations.

To do this, GitOps aims to make developers more productive by applying familiar tools to the hard things: operations management and monitoring. Every developer can use Git and make pull requests; now they can use Git to accelerate and simplify operational tasks for Kubernetes etc. The benefits are far reaching:

• a model for cloud native CICD pipelines;
• faster mean time to deployment and mean time to recovery,
• actionable alerting,
• stable rollbacks (ie., revert/rollback/fork as per Git);
• and an overall coherent approach to understanding, observing and managing apps.

GitOps is Continuous Delivery meets Cloud Native

No grand claims for novelty are being made. GitOps uses ideas drawn from DevOps and Site Reliability Engineering, and probably start with Martin Fowler’s superb Continuous Integration overview in 2006. In some sense we think GitOps is just the latest state of the art.

We are also building on the move towards immutable infrastructure and declarative container orchestration. We’ll say more about this later and in the meantime you can read a good intro in this blog post.

TL;DR we want to minimize the risk of change after deployment, whether intended or via “configuration drift”. And we want our whole system’s desired state to be properly described in Git. Containers and cloud native tools (list of CNCF projects) provide most of what we need for that. At any rate: it is working for us. Weaveworks have been using GitOps with our own production Kubernetes stacks to prove out these ways of working for nearly two years.

Let’s turn to Weaveworks tooling and how we use pipelines. Note: as background it helps to have at least glanced through some of the links above, starting with Jez Humble’s description of the Foundations of Continuous Delivery, for we are building on those principles.

GitOps: Pipelines intro

I want to talk about what we do at Weaveworks, and how we roll out a change to the Weave Cloud deployment on AWS. Our end to end pipeline is wired directly into our clusters and encompasses every step in our build, deploy, manage, monitor lifecycle. We only deploy containers and all our workloads are containerized (more on that later).

Weaveworks’ pipeline is a little different from what people typically do. To understand why, let’s look at typical delivery pipeline.

• CI runs tests; output is delivered to a container image repository
• CD systems deploy a container automatically (or on request, i.e. manually)

In diagram form this might look like this:

Choose your own CI — and other pipeline tools

At Weaveworks, we also use the components shown above, but with some important tweaks that I shall explain in the next section.

Our current toolchain is GitHub for version control, CircleCI for our CI builds of Weave Cloud, Quay.io for our internal image repo, Weave Flux for deployment, Kubernetes for orchestration. Attached to this are data storage and an observability stack. A simplified picture of this is shown below.

Good news! You can use any CI system for GitOps. Eg: for our customer tutorials we use GitLab which bundles code & build. There’s no reason you can’t use alternative CI tools — we like Jenkins, TravisCI, Shippable and Google Container Builder.

GitOps: Automated git → cluster synchronisation

This next part is really important. Deployment and release automation are fundamental to how we do GitOps and make the delivery pipeline work. For this, we created a new tool, Weave Flux, that enables GitOps deployment and natively understands how to manage deployments on Kubernetes. Weave Flux is a tool that automates the staging and release of containers to Kubernetes, as well as services, deployments, network policies and even Istio routing rules. Flux is therefore ‘glue’ that maps all the relationships between code, services and running clusters. It can be seen as a leaner, Kubernetes-native alternative to Spinnaker.

Recall that in GitOps, any changes in your production environment, from (virtual) infrastructure to application deployments, are done via pull request. It helps if you are using a tool which supports Git-cluster synchronisation, and so is designed for version-controlled systems and declarative application stacks.

Flux’s main feature is automated synchronization which ensures that if you make any changes to your repository, those changes are automatically deployed to your cluster. As the intro states: this is a simple, but dramatic improvement on current state of the art.

The GitOps Pipeline

Here’s a new diagram, which shows that everything upstream of deployment is anchored by Git. The deploy component is now split into two parts: a Flux automator that observes new builds and updates configurations to create a new release, and a Flux synchronizer that ensures the orchestrator has the correct state. Note the differences with the previous ‘simplified’ diagram. To be clear: this diagram, below, shows our real pipeline.

Delivery Orchestration: why GitOps is not CI-Ops

Sidebar!

GitOps changes the relative importance of CI for delivery orchestration, i.e. the processes that own and drive the code-to-production pipeline.

Mainstream continuous delivery puts CI at the design centre of the pipeline and treats Git and CD as servant components. In this picture, Git is a service, used by CI to obtain an input for the build. By the same token, CD is an implementation extension powered by CI to promote build artifacts to production. For example the CI tool might use scripts embedded in CI jobs. Therefore in the mainstream view we can say: CI owns delivery orchestration.

In the GitOps pipeline model, Git is the design centre. It plays the central role of “source of truth for everything in the system” — code, config and the full stack. CI, build and test services are necessary for constructing deployable artefacts. But in the GitOps pipeline, the overall orchestration of delivery is coordinated by the deployment and release automation system — triggered by updates to repos. In summary, CD owns delivery orchestration, not CI.

I don’t want to oversell this distinction. It is a subtle shift in how pipelines work, that any CI provider could adopt.

GitOps: Only deploy containers & config

We recommend that your build system generates container images and that you deploy these to your cluster from a container image repository.

Why are containers so important? At Weaveworks, we use the immutable server pattern. Once code has been committed in Git, we don’t want anything else to change. We want to minimise risk of divergence from the system’s intended state. Thus it helps if deployment artifacts are the same in dev & prod. And containers are one way to guarantee this. Using VMs is ok, but using containers just makes all of this easier and less likely to go wrong.

At Weaveworks, internally, we use Docker containers, and Quay.io for our image repository. Our Weave Cloud product works with any container repo: Docker Hub, GCR, and so on.

GitOps recommendations for container orchestration

In GitOps you want your whole system to be expressed declaratively. This is where cloud native tools really help:

• Orchestration: As you gain momentum and your applications grow, you will find it easier to use declarative platforms like Kubernetes or Docker Swarm, to automate orchestration and systematise it, instead of doing everything using ‘config snowflakes’.
• Idempotent installation: Use tools like kubeadm (or kops or kubicorn) that provide idempotent Kubernetes installation which is a prerequisite for GitOps. Or use a cloud service like GKE, ECS, ACS. At Weaveworks, we have been able to use Ansible for idempotent installation.
• Resources: As explained in the previous post, we use Terraform to provision resources.

Selecting a Git provider

You need Git to manage and track system changes, roll them out, or roll them back. Git provides these features via a versioned source of truth with an audit trail including comments. But Git on its own is not enough; you want a Git provider so that:

• Your source of truth is reliable. Your current state — code, config, comments — are available, your change logs will not be lost.
• User friendly social features like user names, auth, permissions, teams, private and public repos and pull request collaboration. This makes Git more intelligible to humans and enables teamwork.
• Standard web APIs to integrate Git into a larger workflow involving build and CI, deployment management and image repos, maybe permissions and signatures.

We use GitHub. This supports developers working in teams, being social and empathetic to customers and to each other. GitHub is more than a source code repository: it is a social tool for managing change. So are Bitbucket and GitLab. Pick one that works for you.

Summary

I hope I have described how:

• Git is the source of truth for the code plus all the config and associated stack. This only works if we have declarative infrastructure.
• There is a GitOps pipeline that is triggered by pull requests and orchestrated by the CD tooling (in our case, Weave Flux). CI and Container repos play a supporting role.
• Containers are the unit of deployment, for immutability. Note that GitOps is similar to PaaS but we push containers and declarative config, not code, and any architecture that supports containers may benefit from end to end automated pipelines.

As a result a developer can update some code in GitHub and release into production as a pull request. Deployment is a sync from Git to the running (orchestrated) cluster. Is there a reverse direction? We’ll look into this in the next blog post (Observability & Control). *Maybe* we can get to a place where the following diagram is realistic.

Weave Cloud

Weave Cloud empowers developers with the ingredients of GitOps: deployment, automation, monitoring and ops management. Here is a short video showing a developer making changes to a Kubernetes app via Weave Cloud and just using Git.

GitOps and App Dev Culture

GitOps inherits from best practices going back 10–15 years. Cloud native is making these more relevant today. To quote from Adrian Cockroft’s piece on CNCF and AWS:

Cloud native architectures take full advantage of on-demand delivery, global deployment, elasticity, and higher-level services. They enable huge improvements in developer productivity, business agility, scalability, availability, utilization, and cost savings.

This means developers have a direct business impact by adopting these practices. And, with team/social practices around Git, the modern application developer is a manager of change. Business and technology are in constant flux (haha) and may be described as “fluid”.

This is driving business to require developers to own end to end delivery. With that comes operational ownership. More and more until everyone will do it. To quote Matt Klein from Lyft, “No one wants to do ops. They HAVE to do ops.”. So here’s to GitOps. 🍻

Next time: GitOps — Part 3: Observability.

— alexis

Alexis is the co-founder and CEO of Weaveworks. He is also the chairman of the TOC for CNCF, and the co-founder of the Coed:Code meetups. Previously he was at Pivotal, as head of products for Spring, RabbitMQ, Redis, Apache Tomcat and vFabric. Alexis was responsible for resetting the product direction of Spring and transitioning the vFabric business from VMware. Alexis co-founded RabbitMQ, and was CEO of the Rabbit company acquired by VMware in 2010, where he worked on numerous cloud platforms. Rumours persist that he co-founded several other software companies including Cohesive Networks, after a career as a prop trader in fixed income derivatives, and a misspent youth studying and teaching mathematical logic.

Originally published at www.weave.works.