Step up your GitOps game using External Secrets Operator

An Introduction to GitOps

GitOps — What a trendy term it is these days (well, at least at the time I’m writing this article — it is). It allows you to use your Git repository as your ‘single source of truth’.

GitOps is an excellent approach to adopt, especially if you manage a large set of Kubernetes clusters and applications. This was the case for us at BioCatch, the pioneer of behavioral biometrics and a global leader in fraud detection. We had to scale up more and more both our clusters and applications, and it got to a point where we had to find a better way to manage our deployments.

We started doing some research and discovered that switching our strategy and adopting GitOps will work great for us. The right tool for the mission is ArgoCD — a powerful GitOps-oriented tool.

The purpose of this article is not to convince you that GitOps is a great pattern to follow but to present a solution and bump your GitOps approach to a whole other level.

A solution? What is the problem?

Let me explain. In short, following the GitOps principles, you store your Kubernetes manifests in a Git repository, and basically - that’s it, right? They are pulled and applied to your Kubernetes cluster, where they live happily.

That sounds convenient. But what happens when you have this little piece of data that you really, and I mean it, really do not want to commit?

This can be your credentials to the database, certificates, or anything else you consider to be confidential — you name it. In the native Kubernetes approach, it lives in a Kubernetes resource — the Secret.

So, what’s all this noise made about this one small resource?

The Kubernetes Secret needs to be kept… as a secret. The issue is that it is not encrypted, all the data is base64 encoded, meaning that it can be decoded by everyone, so it is not such great to save these sensitive information manifests in a Git repository.

Of course, we could create the secret manually using kubectl, and avoid using Git for storing secrets, but we don’t want to do this manual work, right?

Luckily for us, there is a fantastic tool that will help us avoid all these frustrating manual processes on the one hand, and keep it secure on the other hand.

Taking advantage of Vault, alongside External Secrets Operator

As we already understand, Kubernetes doesn’t provide us with an ‘out of the box’ solution to keep our data encrypted. However, there are some solutions for this issue, like the Vault. There is a high chance you’re already using it to store your most sensitive data in a key-value format, so why not leverage it with GitOps?

How does it work? It is pretty simple…

There is this great tool called ‘External Secrets Operator’, or ESO. It can use Vault as a ‘provider’ — a backend for the ESO, and with some reasonable steps such as installing it, creating some manifests, and configuring it, you’re good to go and start using its capabilities.

Authentication to Vault

We want to gain access to our secure Vault. There are numerous methods for authenticating, we will take advantage of one named ‘approle’. It is a set of policies to access Vault, which is used by machines and services, and that suits us perfectly.

I highly encourage you to follow a few steps in the official documentation and allow using this method.

At this point, we have a way to authenticate with Vault, which is great, but the obvious question that comes now is what do I do with it?

Some ESO hands-on

We will start to play with the External Secrets Operator, finally!

The Operator consists of a set of CRD’s which splits into two main parts; let’s break it down.

The first part is used for authentication to Vault:
SecretStore: A namespaced resource that allows access to Vault only to resources inside the namespace.
ClusterSecretStore: Does the same as the SecretStore, but with a slight difference. As the name implies, it serves the entire cluster and lives on a cluster-level.

You, of course, will choose whichever solution best works for you. I chose to use ClusterSecretStore, and here is a YAML representation of it.

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend-approle
spec:
  provider:
    vault:
      server: "https://my-vault-server.com"
      path: "secret_path" # path in vault we want to mount
      version: "v2" # version of the Vault engine
      auth:
        appRole:
          path: "approle"
          roleId: "xxxxx-xxxxx-xxxxx-xxxxx-xxxxx"
          secretRef:
            name: "vault-secret-id" # kind: Secret name
            key: "secret-id"
            namespace: external-secrets

This ‘vault-secret-id’ Secret holds onto your secret-id to authenticate with Vault, and… it will be the last manually created Kubernetes Secret.

Now, we have arrived at our second part: The creation of the Secret resource, which we will do using either one of:

ExternalSecret: A namespaced resource that we’ll use to create the Kubernetes Secret resource.

ClusterExternalSecret: A cluster-scoped resource that can be used to push an ExternalSecret to specific namespaces using a namespaceSelector.

Here as well, you shall choose whichever solution works best for your environment.

The ExternalSecret resource is awesome. As you can (not) read below, in the YAML representation, there is no secret data specified in it, meaning you can feel guiltless pushing it to your Git repository and that way, accomplish a full GitOps cycle.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: external-secret-myapp
  namespace: myapp # same namespace as of the application
spec:
  refreshInterval: "30s" # pull latest config from Vault every 30 seconds
  secretStoreRef:
    name: vault-backend-approle
    kind: ClusterSecretStore
  target:
    name: myapp-secret # kind: Secret name to be created
    creationPolicy: Owner
  data:
  - secretKey: password # key in secret to be created
    remoteRef:
      key: secret/path/to/myapp # vault path
      property: password # The actual key in the vault path

I wrote my own ExternalSecret, what’s next?

So, now all there is left to do is to push the manifest to your Git repository, along with your other manifests and as soon as it is applied to your cluster, it’ll magically create a Kubernetes Secret resource with the correct values you pulled from Vault.

That’s about it — you’ve pushed a secret to Git, without pushing a secret to it. Hope that makes sense.

If you got here, I hope you enjoyed reading the article. I encourage you to read the official documentation to optimize your journey and customization options with ESO.

Stay tuned to the next posts regarding Secrets, GitOps, and whatever in between.

About BioCatch

We are the leader in behavioral biometric technology which analyzes an online user’s physical and cognitive digital behavior and helps to protect individuals and their financial accounts from frauds. We analyze more than 2,000 parameters, such as the way people move their mouse, the angle in which they hold their phone, the speed of typing their personal details, etc, and based on the personal profile we create, we can identify when an imposter is trying to use their account.

Thanks for reading!

Authored by Yoni Katz, DevOps Engineer, BioCatch