GitOps — Comparison Pull and Push
Introduction to GitOps
In the last year, a new approach was added to the possible deployment options for Kubernetes. This approach is named GitOps and is founded on the core concept, that every deployment is versioned secure inside a git repository.
The main benefits of this approach are:
- Versioned Deployments & Change history
All your cluster state is inside a git repository, and only by committing the users are able to update deployments. Also, every change can be monitored through the git commit history.
- Rollbacks by using git primitives
A simple git reset can revert changes of deployments and the old state is always available.
- Out of the box access control
A git system is often a place containing a lot of sensitive data, so it is in most firms properly secured. The same grade of security affects now also deployment operations.
- Policies for deployment
Out of the box, most git systems support policies on branches, for example, that only Pull Requests can update the master branch and another member of the team has to review and accept the deployment. As with access control, the same policies apply to your deployment updates.
As you can see, a lot of benefits are waiting on the GitOps site. In the last year, two approaches are rising up, the first one is the push-based one and the second is the pull-based one. Before we analyze them both, we will first take a look at how typical Kubernetes deployments look like.
In Kubernetes different deployment methods and tools have settled over the past years.
- Raw Kubernetes Template Deployments / Kustomize
This is the most basic way to deploy applications to kubernetes. The developer creates basic yaml files and applies them to the cluster. To reduce rewriting the same templates again, and again, Kustomize was born, that allows modularizing Kubernetes templates.
- Helm Charts
The Helm Charts are an option, to create a bundle of templates, init Containers, Sidecars etc., that are used to deploy one application with more customization than the template approach. In the core, there are heavily templated yaml files, that are filled in with values by the Helm tool and then send to a cluster component called tiller that deploys them to the cluster and allows upgrading and rollbacks. Important is the fact, that under the hood Helm just fills in the template values and then applies the templates as in the traditional approach. There is a variety of helm charts available, spanning a broad range of popular applications.
- Alternative tools
There are tons of alternative tools available, all have in common, that they process some kind of templated files into kubernetes understandable yaml files and then submit them.
At my clients, we regularly use Helm charts for tools we depend on (because they are very solid and reduce work a lot) and raw Kubernetes yaml files to deploy our own applications.
Pull & Push
In one of my last blog posts, I introduced a tool called Weave Flux, that allows committing templates to a git repository and updates the deployment after each commit or container push. As I have seen, this tool is still one of the main drivers for the pull approach so I will often reference it. If you want more details on how to use it, here is a link to the other blog post. After this is done, we will examine the push approach in detail.
All benefits of using a GitOps process are valid for both approaches.
The pull approach is based on the fact, that all changes are applied from inside the cluster. There is an operator inside the cluster, that regularly scans the associated git repositories and docker registries and if a change occurs, the cluster state will be updated from inside. This process is generally noted as very secure, due to the fact, that no external client has credentials with admin access to the cluster.
- No external client has the right to publish to the cluster, all updates are coming from inside.
- Some tools allow also syncing helm chart updates and can sync them to the cluster.
- Docker registries can be scanned for new versions and if a new image is pushed, the git repo and the deployment are updated with the new version.
- Pull-based tools can be distributed in different namespaces, with different git repositories and access rights and through this, a multi-tenant model can be applied. Team A can use namespace A, Team B can use namespace B and Infrastructure Team can have a global one.
- The tools are regularly very lightweight.
- Combined with a tool like Bitnami Sealed Secrets operator, secrets can be placed encrypted inside the git repository and are extracted inside the cluster.
- No coupling on the CD pipelines, because deployments are done cluster internally.
- Managing secrets of Helm Chart deployments is harder than normal because secret values have to be first created as for example sealed secrets, then be decrypted by an internal operator and then made available to the pull tool. After this has happened, a helm release can be executed with the values in the secrets that are already deployed. The easiest way would be to create a secret of all helm values used for deployment, decrypt it and then commit it to git.
- Through the pull approach, you are bound to the tools executing the pull. This shrinks the possibility to customize the process of applying deployments to the cluster. For example, the usage of Kustomize is harder, because it has to be executed before the result templates are pushed to git. I don’t say that separate tools can’t be used, but it is harder to implement them to the deployment flow.
In the push approach, an external system (mostly CD pipelines) are triggering deployments to the cluster, after a git repository was committed to or a previous CI pipeline was executed successfully. In this approach, the pipeline system has access to the cluster.
- Security is determined by the git repository and the build pipeline.
- Deploying Helm Charts can be done easier and with the support of helm plugins.
- Secret management is easier because secrets can be applied to pipelines and also stored encrypted in git, as preferred by the user.
- Not bound to the specific tool, because all kind of tools can be used.
- Container version updates can be injected by build pipeline.
- Cluster credentials are inside the build system
- Updating the containers of deployments is still easier with a pull-based process
- Strongly coupled to the CD system, because pipelines have to be created that are maybe in the language of Gitlab Runners, and then later on the team decides to migrate to Azure DevOps or Jenkins and a lot of the build pipelines have to be migrated.
Summary — Push or Pull
I think it is as always, both approaches have their pros and cons and some tasks are easy with the one and harder with the other. At first, I did deployments manually by hand, but after I stumbled over some blog posts of WeaveFlux, it convinced me to implement GitOps processes for all projects. To implement it for all basic templates was an easy process, but then I started to stumble over issues with my helm charts. At this time Weave Flux did only provide a very early version of the Helm Chart Operator and even now, some tasks are harder, because of manually pre-creating secrets and applying them. Now you could argue, that the pull approach is way more secure because cluster credentials aren’t available outside the cluster and this would increase security so much, that the additional effort is worth it.
I was surprised, that after I rotated this thought a little bit, I came to the conclusion that this is not true. If you think about the components that should be most secure in your organization, this list will include the secret stores and the CICD systems/git repos. The information that is inside them is very sensitive and has to be protected very well. Additionally, if someone manages to break into your git repository and is able to push things there, he can deploy whatever he wants (regardless of push/pull) and infiltrate your cluster systems. So the most critical component to protect is your git repository / CICD systems and not your cluster credentials here. If you have good policies and security measures on this kind of systems and the cluster credentials are only pulled as secrets into the pipelines, the additional security of a pull approach is maybe not as valuable as originally thought.
Additionally, if someone manages to break into your git repository and is able to push things there, he can deploy whatever he wants (regardless of push/pull) and infiltrate your cluster systems
So if the effort is higher implementing pull and security is not increased, wouldn’t it be better to use just push-based approach? On the other hand, someone could argue that you are coupling very strongly with the CD system and it is maybe better to not do that, to allow easier migrations later on.
In my opinion, it is as always, use what is easier for your use case or combine everything. Personally, I use both approaches, WeaveFlux for pull-based deployments, that includes mostly our self-written services, and a push-based approach with Helm and plugins, that makes it very easy to apply Helm Charts to the cluster without stumbling over issues while creating secrets. I guess there will be never a fit it all solution because these topics are always very complex and use case dependent. What I can recommend deeply is using GitOps, because it makes operations life way easier and a lot more secure.
I hope my personal opinion on this topic makes it easier for you to decide, which approach is suitable for your type of deployments and I would be happy to hear your opinions.
Join our community Slack and read our weekly Faun topics ⬇