Gating Flux Source Reconciliation for Release Management

In this blog post, I will try to explain how release management can be achieved with FluxCD, which is one of the GitOps tools that enable the automatic deployment of infrastructure code stored in version control systems to a Kubernetes cluster.

How Continuous Delivery and Continuous Deployment mechanisms can be implemented will be discussed.

When bootstrapping Flux, you can specify which repository, branch, and path on GitHub will be used to deploy the infrastructure code. Let’s go through an example using a sample repository.

First, let me explain the structure of the files within the repository. Typically, in a repository, you can have a structure similar to the following:

Directory Structure of Repository

The YAML files related to the Custom Resource Definitions (CRDs) and the Deployments of the Source, Helm, and Kustomization controllers under the flux-system directory is automatically committed to the repository when the flux bootstrap the command is executed.

In the following file, we specify the VCS URL, branch, and path for the Kustomization objects that Flux will deploy. Here, we instruct Flux to monitor the YAML file in the infra directory of the repository.

infra/flux-system/gotk-sync.yaml

# This manifest was generated by flux. DO NOT EDIT.
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 10m0s
  ref:
    branch: master
  secretRef:
    name: flux-system
  url: YOUR_VCS_URL
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: flux-system
  namespace: flux-system
spec:
  path: ./infra
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

Flux will look at the contents of the app-kustomization.yaml file located under the infra directory. Here, we specify the path and interval value for our Flux Kustomization objects. The path defines the directory where the Kustomization file is located, and the

Interval determines the frequency at which Flux will check for any configuration drift between the branch's commit and the Kubernetes cluster.

infra/app-kustomization.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
    name: flux-demo
    namespace: flux-system
spec:
    interval: 10m0s
    retryInterval: 1m0s
    path: ./infra/app
    prune: true
    wait: true
    force: true
    timeout: 10m0s
    sourceRef:
        kind: GitRepository
        name: flux-system

In the following file, we specify our Kustomization objects:

infra/app/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy-flux-demo.yml

We are maintaining the Deployment workload for our application.

infra/app/deploy-flux-demo.yml

apiVersion: apps/v1
kind: Deployment
metadata:
    name: flux-demo
    namespace: default
    labels:
        app: flux-demo
spec:
    replicas: 1
    selector:
        matchLabels:
            app: flux-demo
    template:
        metadata:
            labels:
                app: flux-demo
        spec:
            containers:
                -   name: flux-demo
                    image: uzumlukek/local-firestore:testing

RELEASE MANAGEMENT

In the file structure shared, Flux will perform a configuration drift check for the application with the specified 10-minutes interval against the Deployment state present in the VCS repository. This allows Flux to detect any differences between the desired state defined in the repository and the actual state of the application running in the cluster.

PROBLEM: The potential issue here is if an accidental commit is made to the branch that Flux is checking. ( Some junior problems :D )

Instead of immediately reverting the commit, efforts can be made to resolve the issue before the next interval. However, if the problem cannot be resolved before the next interval, the clusters will be updated, potentially resulting in an incorrect release that impacts your production environment.

As a solution, Continuous Delivery and Continuous Deployment can be implemented.

It is important to understand the difference between Continuous Delivery and Continuous Deployment as release management approaches.

Continuous integration vs. delivery vs. deployment

Continuous Delivery

Once the approval process is completed, it is important to perform a manual release to the production environment.

In our example, even if a commit has been made to the Master branch, Flux will not update the Kubernetes cluster without a manual release.

This ensures that Flux only applies changes to the cluster that have gone through the designated approval process and have been manually released.

Flux CLI is required for implementation.

Flux CLI

The Flux CLI tool indeed supports a subcommand called “suspend”. After a stable release, you can use the following command to prevent Flux’s Source Controller Pod from checking for configuration drift in the source.

 flux suspend source git flux-system

After making sure of our tests and committing to the Master branch, we need to run the following command “resume” subcommand to check the Source for Drift.

flux resume source git flux-system

With these 2 commands, we have deployed, and after making sure that it is released properly, we stop the system with the suspend command.

Scheduled Continuous Deployment

Photo by Content Pixie on Unsplash

It is updating the production environment either continuously or according to a certain schedule, and exiting the release in a way that will affect the end user. Again, in our example, in mechanisms that commit to the Master branch and receive scheduled updates, Flux notices the configuration drift in the cluster and updates the application with the new version.

For Scheduled Continuous Deployment, it is necessary to schedule the same structure as we did on Continuous Delivery.

However, Flux does not support this structure in a built-in way, according to the following issue. With the Time Based Gating structure, it cannot check if there is an update only at a certain time.

[RFC] Gating Flux reconciliation by stefanprodan · Pull Request #3158 · fluxcd/flux2

We can implement different implementations to create this scheduled structure ourselves. For example;

• Cronjob
• Systemd Service Trigger by Timer
• Scheduled Github Action Job

These structures can be expanded. It is up to your architectural structure and creativity.

Here, I will show you how to Trigger Systemd Service with Timer and resume + suspend with this Systemd service.

We create the Timer that will trigger the Release Systemd service at 3 am.

cat <<EOF >/etc/systemd/system/release.timer
[Unit]
Description=*-*-* 3:00:00
[Timer]
OnCalendar=*-*-* 3:00:00
Persistent=true
[Install]
WantedBy=timers.target
EOF

We specify the script to run on the machine with the Release Systemd service. The commands that will work here are the resume and suspend operations, respectively.

cat <<EOF >/etc/systemd/system/release.service
[Unit]
Description=Release Service
[Service]
Type=simple
ExecStart=/bin/bash -c 'flux resume source git flux-system && flux suspend source git flux-system'
TimeoutStopSec=86400s
[Install]
WantedBy=multi-user.target
EOF

Finally, if there is a new commit in our Master branch at 3 am every night, we perform the Continuous Deployment process on a scheduled basis.

While managing Release with Flux, a Gitops tool, we can perform Continuous Delivery and Continuous Deployment operations in this way.

Conclusion

If you want to use FluxCD and apply release management, you can implement Continuous Delivery and Continuous Deployment with Flux resume and suspend features.

Thanks for reading. If you have questions or comments regarding this article, please feel free to leave a comment below.

Would like to get in touch? Reach me out on LinkedIn:

https://www.linkedin.com/in/muhammedsaidkaya/