Rethinking Helm and ArgoCD at OneFootball: The steps to Automation and Standardization

Published in

OneFootball Tech

5 min readOct 31, 2024

Managing Kubernetes infrastructure for millions of football fans worldwide at OneFootball demands a seamless, automated approach. As we continue evolving our platform, we rely on key tools like ArgoCD and Helm to manage our workloads efficiently. However, as our setup grew, so did its complexity, leading us to rethink our approach.

In this article, we’ll walk through how we revamped our Helm chart and ArgoCD setup to improve automation and standardization while reducing manual upkeep. Our goal is better isolation of concerns and more efficient automation, ensuring charts remain up-to-date with minimal effort.

To get an higher picture of the setup and the process read the previous blog post on this topic: “Rethinking Helm and ArgoCD at OneFootball: Our Journey to Automation and Standardization”. If you’re ready to start keep on reading for more technical details.

Considerations

Before starting it’s worth considering that we have two types of charts to apply in our clusters. Public and Private charts. Although all the third-party charts are public, the ones we manage (in-house charts) and the ones we extend come from Onefootball GitHub private repositories and we don’t want them to be publicly available. This adds an extra requirement, argocd needs to be able to fetch private helm repositories from the onefootball organization in github. It turned out this is quite simple, the repository requires an index.yaml defining the chart metadata for each available version.

Let’s start playing

The initial folder structure looks like this:

./charts
    /argocd-connect
       ...
    /karpenter
    /loki
       ...
./clusters
    /prod
       /argocd-connector
          /helmfile.yaml
          /values.yaml
       /chart-values
              ...
          karpenter.yamls
          loki.yaml
              ...
       /eks
            ...
         /main.tf
            ...
       /manifests
            ...
    /staging

Create Dedicated Repositories for in-house and customized Charts

We need to extract the charts into dedicated GitHub repositories, there are a few things to do:

Create and init the repository
Move the code
Create the release process

Since this process will be repeated for several charts, we’ll automate it with a simple script to avoid human error.

#!/bin/bash

init() {
  chart="$1"
  repo="helm-chart-${chart}"
  repo_with_owner="motain/helm-chart-${chart}"

  gh repo create "$repo_with_owner" --private
  gh repo clone "$repo_with_owner"

  cd "./$repo/"

  git checkout -b main
  git commit --allow-empty -am "chore(setup): Initial commit to create main branch"
  git push -u origin main
}

move_chart() {
  chart="$1"

  repo="helm-chart-${chart}"
  repo_with_owner="motain/helm-chart-${chart}"

  cd "./$repo/"
  git checkout -b "<ISSUE-ID>/extract-of-charts-to-specific-repositories-${chart}"

  # Copy the .github folder from the another repository hosting an helm chart project
  cp -r ../helm-chart-xxx/.github .
  git add .
  git commit -sam 'chore(release): add release workflow' -m 'use standard workflow to release helm chart'
  mkdir -p "./charts/"
  cp -r "../cluster-monorepo/charts/${chart}" ./charts/
  mv "charts/${chart}/.helmignore" "charts/${chart}/README.md" .

  git add .
  git commit -sam "feat(${chart}): populate ${chart} helm chart repository" -m "copy chart definition from cluster-monorepo/charts/karpenter"
  git push origin "<ISSUE-ID>/extract-of-charts-to-dedicated-repositories-${chart}"
  gh pr create --fill-verbose;
  gh pr merge --squash --auto --delete-branch
}

# List of charts to migrate
charts="karpenter ... rollout-analysis"
for chart in $charts; do
  (init "$chart")
  (move_chart "$chart")
done

We use GitHub Actions to automate the release of these Helm charts. The action packages and uploads the chart artifact, managing the index.yaml file used by Helm. We'll ensure these repositories remain private by specifying that pages_branch is the main branch.

- name: Install Helm
  uses: azure/setup-helm@v4
  env:
    GITHUB_TOKEN: "${{ github.token }}"
- name: Run chart-releaser
  uses: helm/chart-releaser-action@v1.6.0
  with:
    pages_branch: main
  env:
    CR_TOKEN: "${{ github.token }}"

I invite you to check the helm/chart-releaser-action and the chart-releaser projects to gain more confidence with the release process.

Refactor ArgoCD Templates

We update ArgoCD’s application definitions to support multiple sources from different locations. In detail, we fetch the chart from an upstream repository and apply values from the cluster repository with the convention that values files are located in valuesDir and are named after the chart they’re redefining (localChartName).

For each item in the list remote.components we create an application in Argocd. The values source defines the host where values files are located. Note the keyword ref, this allows us to reference the content from this source as $values.

In the Helm source, we differentiate between public and private charts (Onefootball’s charts). We reference the public charts with the keyword chart while charts from GitHub repositories are referenced using the keyword path. To simplify the interface the keyword is chosen based on the remoteChartURL property; if it contains “github.com/motain” it’s a private repository and we use the keyword path.

{{- range $component := .Values.project.remote.components }}
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: {{ $component.localChartName }}
  namespace: {{ $.Values.namespace }}

...

spec:
  destination:
    namespace: {{ $component.namespace }}
  
  ...

  sources:
  - helm:
      releaseName: {{ $component.localChartName }}
      valueFiles:
        - $values/{{ $.Values.project.remote.valuesDir }}/{{ $component.localChartName }}.yaml
    repoURL: {{ $component.remoteChartURL }}
    {{- if contains "github.com/motain" $component.remoteChartURL }}
    path: "charts/{{ $component.remoteChartName }}"
    {{- else }}
    chart: {{ $component.remoteChartName }}
    {{- end }}
    targetRevision: {{ $component.remoteChartVersion }}
  - ref: values
    repoURL: https://github.com/motain/cluster-monorepo.git
    targetRevision: main
  syncPolicy:

  ...

To populate the applications we pass a list of remote components in the format:

- localChartName: prometheus-remote
  remoteChartName: kube-prometheus-stack
  remoteChartVersion: x.y.z
  remoteChartURL: https://prometheus-community.github.io/helm-charts
  namespace: prometheus-namespace
- localChartName: rbac
  remoteChartName: rbac
  remoteChartVersion: rbac-x.y.z
  remoteChartURL: https://github.com/motain/helm-chart-rbac.git
  namespace: rbac-namespace

Reorganize the Repository

We can now remove the charts from the charts folder and populate the remote components list.

We could have update the applications in place but we opted for creating a new cluster with the new definition and shift the load. If you are courius about how to manage blue/green deployment with EKS I strongly recommend you to read our post: “From Blue to Green: Optimizing AWS EKS Clusters Upgrade with Blue/Green Tactic”.

After the refactor, the repository looks like this:

./charts
    /argocd-connect
./clusters
    /prod
       /argocd-connector
          /helmfile.yaml
          /values.yaml
       /chart-values
              ...
          karpenter.yamls
          loki.yaml
              ...
       /eks
            ...
         /main.tf
            ...
       /manifests
            ...
    /staging

Automate Chart Updates

By enabling Renovate in our new repositories, we automate chart updates. Renovate will monitor for new chart versions and automatically open pull requests to apply updates.

Automate Update of Cluster Repository

We also configure Renovate to update the cluster repository whenever a new version of any chart is released — whether it’s an external, customized, or in-house chart. Updating the cluster is not as linear as the previous step. Our definition of the charts to install is not standard. But nothing is lost. Renovate allows you to specify custom ways to look into your code searching for packages to update. We have to instruct it to look for files matching the regular expression (^|/)clusters\/.*\/argogcd-connector\/.*values\\.ya?ml$ and look for the sequence:

- localChartName: chart-xyz
  remoteChartName: <depName>
  remoteChartName: currentValue>
  remoteChartURL: <registryUrl>
  namespace: namespace-xyz

This is an extract of our configuration:

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": [
    "config:base"
  ],
  
      ...

"customManagers": [
    {
      "customType": "regex",
      "fileMatch": [
        "(^|/)clusters\/.*\/argocd-connector\/.*values\\.ya?ml$"
      ],
      "matchStrings": [
        "remoteChartName: (?<depName>.*)[^\\w]*remoteChartVersion: (?<currentValue>.*)[^\\w]*remoteChartURL: (?<registryUrl>.*)"
      ],
      "datasourceTemplate": "helm"
    }
  ]
      ...
}

Notice that each remote component item must respect this structure to ensure the charts get updated automatically.

Streamline the Diff Process

We want to validate changes before applying them in the cluster. For this scope, we have a dedicated action that looks for changes in clusters/**/argocd-connector/** and clusters/**/chart-values/** and run the Argocd diff command.

argocd app diff "${{ inputs.app-name }}" \
  --grpc-web \
  --refresh \
  --revisions "${{ inputs.remote-version }}" \
  --source-positions 1 \
  --revisions "${{ inputs.local-version }}" \
  --source-positions 2 \
  --server "${{ inputs.argocd-server }}" \
  --auth-token "${{ inputs.argocd-token }}"

The action grabs the output and publishes it as a comment in the pull request to be reviewed.

Conclusions

By adopting these changes, we aim to reduce our team's cognitive load, automate repetitive tasks, and improve our setup's maintainability. This revamped approach made our system more robust and freed up time for us to focus on strategic (and more interesting) improvements.