Zero-downtime migration to Helm chart

Background

Beekeeper made use of a custom, legacy script to deploy our microservices to K8s. This was introduced way before GitOps has reached today’s maturity, and the script’s nature and vision are synonymous with GitOps’ design principle, to have a single source of truth for our application deployments in K8s. However, if we compare it with nowadays available tools, this script have little to no added-benefit for various reasons:

• Reinvent the wheel. The script provides templating for common patterns in deploying our k8s-based applications, basically achieving the same functionalities as other tools for managing complex K8s manifests, such as Helm, Kustomize, Jsonnet.
• Cost. The maintenance of the script becomes expensive with time as the code grows larger and opinionated. Hence, this gives a much slower learning curve for both maintainers and users (i.e. our SDE team) of the script.

As we are moving towards GitOps at Beekeeper, Helm suits our need for managing our complex or repeated K8s manifests since it integrates well with FluxCD (as HelmRelease) and Kustomize. We pick Kustomize to benefit from the ability to patch YAML on demand, thus giving a higher degree of flexibility to our developers. Then, equipped with FluxCD, we want to deliver a true GitOps approach with flexibility and consistent pattern for all our standard and custom applications.

Challenges

During the early discovery phase, we realized that FluxCD and Helm handle things differently:

• FluxCD
  All drifts (i.e. changes made manually or differ from what committed in Git repo) of K8s objects will be simply overridden by FluxCD accordingly, reconciling from the latest commit in Git. For example, if you have such an object in K8s:

# Current object in K8s
apiVersion: v1
kind: ConfigMap
metadata:
  name: bonjour
  namespace: default
  labels:
    custom: "manual"
data:
  KEY: VALUE

and then you have such definition in Git for the same object:

# This is what you specified in Git 
# file is under a path incorporated by FluxCD's Kustomization
apiVersion: v1
kind: ConfigMap
metadata:
  name: bonjour
  namespace: default
  labels:
    custom: "from_git"  
data:
  KEY: VALUE

• FluxCD will simply update the object to the latter, plus adding some labels. The final object in K8s will look like after FluxCD reconciliation has been successful:

# Resulting object in k8s 
apiVersion: v1
kind: ConfigMap
metadata:
  name: bonjour
  namespace: default
  labels:
    custom: "from_git"
    kustomize.toolkit.fluxcd.io/name: kustomization-name
    kustomize.toolkit.fluxcd.io/namespace: default
data:
  KEY: VALUE

In short, FluxCD’s behavior is “I don’t care what’s there, I’ll apply whatever is written in Git”. In fact, in order to achieve true GitOps, it is generally desirable to have this behaviour and we favour it for most of our use cases.

• Helm
  Firstly, unlike FluxCD, Helm will simply refuse to recreate or manage the same object (i.e. identical name and kind within the same namespace). Helm’s approach is “I do care what’s there, I’ll apply only if it is explicitly under my control”. While it prevents the unexpected, this officially kicks off some required migration steps to the Helm chart at Beekeeper.
  Secondly, since Helm 3, Helm has implemented this 3-way Strategic Merge Patches. Under normal circumstances — except during first time deployment, helm compares the proposed chart’s manifests with the most recent chart’s manifests to determine the difference it needs to apply. However in our case as the application objects are already in K8s, Helm will not have a record of the most recent charts’ manifests. Consequently, this presents us with some difficulties: if the live states differ from the rendered chart’s manifests, Helm will attempt to apply only the difference.

Reflecting on the above limitations and challenges we may face with Helm, we, the DevOps team at Beekeeper propose a way to streamline our pathway to Helm.

Avoiding Downtime

Existing solutions all proposed to delete all the objects that will be redeployed by helm. With this clean startline, helm will not complain about the presence of application objects in K8s and it will redeploy completely the rendered manifests as expected.

However, this means downtime and we would like to avoid it. Surely, one could argue that opting to perform migration during a maintenance window would be an ideal solution. Yet, at Beekeeper, our users are everywhere around the world and even our regular maintenance windows during midnight often resulted in small service disruptions somewhere. Furthermore, we have to potentially perform this delete-and-redeploy for hundreds of our microservices, which seems tedious!

We emphasize the importance of availability of our platform. In other words, we try to avoid relying on maintenance windows (i.e. increasing the frequency of possible downtime) and so we need a solution that works best for our use-cases.

Our solution: Migrate to Helm without Downtime

Part I: Tell Helm “hey these objects are under your control now”

This step is pretty straightforward. Basically all objects rendered and applied by Helm (e.g. via Helm cli or helm-controller) will have this label, telling that they are managed by Helm:

"app.kubernetes.io/managed-by": “Helm”

and these annotations telling which helm release are they part of:

"meta.helm.sh/release-name": “app-name”
"meta.helm.sh/release-namespace": “namespace”

Hence, for this part, we only need to the following:

kubectl annotate $kind $name -n $namespace "meta.helm.sh/release-name"=$releasename
kubectl annotate $kind $name -n $namespace "meta.helm.sh/release-namespace"=$namespace
kubectl label $kind $name -n $namespace "app.kubernetes.io/managed-by"=Helm

Part II: Avoid Likely Failure of K8s Deployment

Considering Helm’s design, telling Helm to start managing the K8s objects is not enough. During the first migration, helm-controller executes helm install and here we encountered a problem we have mentioned earlier: if the rendered charts' manifests differ from the actual live states, it will only apply the difference. Although we have certain custom in-house helm charts, covering each unique case is unrealistic, given the different nature of every microservice. As a consequence, our generic, custom helm charts always diverge during the Helm template and only the difference would be taken into account during the Helm install step. Unfortunately, this will likely cause K8s deployment to fail.

For instance, imagine if you have this deployment:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ramen-app
  namespace: food
  labels:
    app: ramen-app
spec:
  selector:
    matchLabels:
      app: ramen-app
  template:
    metadata:
      labels:
        app: ramen-app
    spec:
      containers:
        - image: beekeeper-ramen-app:1.2.3
          name: ramen-app
          volumeMounts:
            - name: ramen-recipe-config
              mountPath: /opt/config/

and, for some reasons the referenced volume name changed to vegie-ramen-recipe-config, hence the proposed helm chart manifests is:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ramen-app
  namespace: food
  labels:
    app: ramen-app
spec:
  selector:
    matchLabels:
      app: ramen-app
  template:
    metadata:
      labels:
        app: ramen-app
    spec:
      containers:
        - image: beekeeper-ramen-app:1.2.3
          name: ramen-app
          volumeMounts:
            - name: vegie-ramen-recipe-config
              mountPath: /opt/config/

Then the deployment become like this:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ramen-app
  namespace: food
  labels:
    app: ramen-app
spec:
  selector:
    matchLabels:
      app: ramen-app
  template:
    metadata:
      labels:
        app: ramen-app
    spec:
      containers:
        - image: beekeeper-ramen-app:1.2.3
          name: ramen-app
          volumeMounts:
            - name: ramen-recipe-config
              mountPath: /opt/config/
            - name: vegie-ramen-recipe-config
              mountPath: /opt/config/

For which K8s marked the deployment as failed, as both volumes conflict on the same path.

Part III: Analyze The Secret of Helm Release Object

To overcome the issue, we need to trick Helm to believe that:

• the current state has been produced by Helm
• the latest manifests produced by Helm are identical to the current live state.

Helm keeps track of each release (i.e. helm installation/upgrade) via a K8s secret object. To illustrate better, here we give an example as follow:

---
apiVersion: v1
data:
  release: SDRzSUNCV3ZrV01BQTJSbFkym.........
kind: Secret
metadata:
  labels:
    name: name
    owner: helm
    status: deployed
    version: "1"
  name: sh.helm.release.v1.name.v1
  namespace: namespace
type: helm.sh/release.v1

The previous manifests recorded in each release is, in fact, written in an encoded format and we can run the following command to decode it:

echo SDRzSUNCV3ZrV01BQTJSbFkym......... | base64 -d | base64 -d | gunzip -c

Now we have the following one-liner JSON:

{"name":"release-name","info":{"first_deployed":"2022-10-27T14:15:11.089565672Z","last_deployed":"2022-11-10T16:45:56.812501717Z","deleted":"","description":"Upgrade complete","status":"deployed"},"chart":{"metadata":{"name":"some-helm-chart","version":"0.0.0","description":"some description","apiVersion":"v2","appVersion":"1.0.0","type":"application"},"lock":{"generated":"2022-10-24T15:43:36.703446516Z","digest":"sha256:b1ca0d3f0777..."},"templates":[{"name":"templates/deployment.yaml","data":"LS0tCiMgU291cmNlOiBnbG9vL3RlbXBsYXRlcy81LWdhdGV3Y....."}],"values":null,"schema":null,"files":null},"config":{},"manifest":"---\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  annotations:\n    generation: \"1\"\n.....","version":1,"namespace":"release-namespace"}

In prettier format:

{
  "name": "release-name",
  "info": {
    "first_deployed": "2022-10-27T14:15:11.089565672Z",
    "last_deployed": "2022-11-10T16:45:56.812501717Z",
    "deleted": "",
    "description": "Upgrade complete",
    "status": "deployed"
  },
  "chart": {
    "metadata": {
      "name": "some-helm-chart",
      "version": "0.0.0",
      "description": "some description",
      "apiVersion": "v2",
      "appVersion": "1.0.0",
      "type": "application"
    },
    "lock": {
      "generated": "2022-10-24T15:43:36.703446516Z",
      "digest": "sha256:b1ca0d3f0777..."
    },
    "templates": [
      {
        "name": "templates/deployment.yaml",
        "data": "LS0tCiMgU291cmNlOiBnbG9vL3RlbXBsYXRlcy81LWdhdGV3Y....."
      }
    ],
    "values": null,
    "schema": null,
    "files": null
  },
  "config": {},
  "manifest": "---\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  annotations:\n    generation: \"1\"\n.....",
  "version": 1,
  "namespace": "release-namespace"
}

Note that this is just an example of Helm release configuration.

As you can observe from above json, the latest rendered manifests are recorded as long string values of “manifest”. We have now located where and how the previous manifests are stored by Helm.

Part IV: Injecting a hacky Helm Release Secret Object

Note that for the sake of the reading experience, we write this step after the labeling and annotations (Part I). During our actual migration, we perform this step first before doing Part I, so that we do not need to clean the label and annotations at the 3rd step below.

Next, we just need to assemble our flow. For each microservice, we do:

1. First, gather all the K8s objects associated with this application. This is feasible thanks to our deployment setup, which requires every manifest to be committed to git. At least we guarantee the object’s Name, Kind, and Namespace are correct.
2. Next, to better detect drift that possibly occurred because of manual intervention, we can get the current state of every object and dump its YAML to a file. Example script in bash:

touch /tmp/dump
for i in "${!names[@]}"; do
do
    local kind=${kinds[$i]}
    local name=${names[$i]}
    local namespace=${namespaces[$i]}

    kubectl get $kind $name -n $namespace && \
      kubectl get $kind $name -n $namespace -o yaml >> /tmp/dump
    echo "---" >> /tmp/dump
done

3. Example of, using yq, sanitizing the YAML from any server-side generated values:

yq -i 'del(.metadata.annotations."kubectl.kubernetes.io/last-applied-configuration")' /tmp/dump
yq -i 'del(.metadata.creationTimestamp)' /tmp/dump
yq -i 'del(.metadata.generation)' /tmp/dump
yq -i 'del(.metadata.resourceVersion)' /tmp/dump
yq -i 'del(.metadata.uid)' /tmp/dump
yq -i 'del(.status)' /tmp/dump

sed -i 's/\"/\\\"/g' /tmp/dump

4. Compile the JSON manifest.

Create a one-liner YAML from compiled object YAMLs:

cat /tmp/dump | awk '{printf "%s\\n", $0}'

put the one-liner YAML accordingly:

{
  "name": "release-name",
  "info": {
    ...
  },
  "chart": {
    ...
  },
  "config": {},
  "manifest": "PUT HERE",
  "version": 1,
  "namespace": "release-namespace"
}

Using jq, create a one-liner JSON:

cat /tmp/dump-json | jq -c > /tmp/dump-json-one-liner

5. Encode the JSON

helm_release=$(cat /tmp/dump-json-one-liner | gzip -c -k | base64 | base64 | awk '{print}' ORS='')

6. Create the secret object in K8s

cat << EOF | kubectl create -f-
apiVersion: v1
data:
  release: ${helm_release}
kind: Secret
metadata:
  labels:
    name: name
    owner: helm
    status: deployed
    version: "1"
  name: sh.helm.release.v1.name.v1
  namespace: namespace
type: helm.sh/release.v1

EOF

Done! Here we have officially registered the live states of our application as Helm release.

Part V: Test and observe

We do not have many things to explore here at this step. Just merge your PR and let FluxCD do the work:

kubectl get helmrelease
NAME       AGE    READY   STATUS
some-app   15m    True    Release reconciliation succeede

Conclusion

Et voilà, our testing has confirmed the successful migration to Helm charts as intended. We have effectively demonstrated a methodology for transitioning to Helm charts without any interruption in service.

To further streamline the migration process, we have developed a custom script for automation. This tool is highly valued by our software development engineers, as it allows them to efficiently manage and migrate microservices to Helm charts.

As the implementation of this automation script is straightforward, we have chosen not to include it in this post.

We look forward to sharing our next update with you.

References

• https://helm.sh/
• https://fluxcd.io/