Using the External Secrets Operator with OCI Kubernetes and OCI Vault

Published in

Oracle Developers

6 min readJun 1, 2022

A customer recently pointed me to the External Secrets Operator (ESO) project and I was pleased to see it already has integration with OCI Vault. In this article, we’ll take ESO for a spin and see how we can use it with OKE.

Installing the External Secrets Operator on OKE

First, create an OKE Cluster either using the Quick Create in the OCI Console or using https://github.com/oracle-terraform-modules/terraform-oci-oke. You can now use Cloud Shell or the operator host in the terraform-oci-oke module to run kubectl or helm commands.

Let’s use the helm chart to install the ESO:

helm repo add external-secrets https://charts.external-secrets.iohelm install external-secrets \
  external-secrets/external-secrets \
  -n external-secrets \
  --create-namespace

Configuring ESO for use with OCI Vault

If you are not using instance_principal, create a Kubernetes secret in a yaml file to hold the the private key and the fingerprint:

apiVersion: v1
kind: Secret
metadata:
  name: pk-fp
  labels:
    type: oracle
type: Opaque
stringData:
  privateKey: |
    -----BEGIN RSA PRIVATE KEY-----
    abcdefg1234567890/+-
    -----END RSA PRIVATE KEY-----
  fingerprint: ab:cd:ef:gh:12:34

And create the secret:

kubectl create -f pk-fp.yaml

You can now create the SecretStore:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: alivault
spec:
  provider:
    oracle:
      vault: # The vault OCID
      region: # The vault region
      auth:
        user: # A user OCID
        tenancy: # A user's tenancy
        secretRef:
          privatekey:
            name: pk-fp
            key: privateKey
          fingerprint:
            name: pk-fp
            key: fingerprint

N.B. If you use instance_principals, remove the section from auth to the end. You also need to create a dynamic group and add the worker nodes to them. Finally, you also need to create a policy to allow the dynamic group to use the key which you’ll be creating in OCI Vault.

kubectl create -f oci-secret-store.yaml

Create an ExternalSecret

Let’s now create an ExternalSecret using ESO:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: oci-secret
spec:
  refreshInterval: 0.03m
  secretStoreRef:
    kind: SecretStore
    name: alivault # Must match SecretStore on the cluster
  target:
    name: my-oke-secret # Name for the secret on the cluster
    creationPolicy: Owner
  data:
  - secretKey: key
    remoteRef:
      key: my-eso-secret

my-eso-secret is the name of the OCI Secret in Vault:

In this case, it just has a plain content (“aSecretInSiV”) so we use data as above and we can create the ExternalSecret:

kubectl create -f ext-secret.yaml

Get the external secrets:

kubectl get es
NAME         STORE      REFRESH INTERVAL   STATUS
oci-secret   alivault   0.03m              SecretSynced

We can see that the secret has been synced. Let’s list the secrets in this namespace:

kubectl get secrets
NAME                  TYPE                                DATA   AGE
default-token-mnpsm   kubernetes.io/service-account-token   3    41m
my-oke-secret         Opaque                                1   108s
pk-fp                 Opaque                                2    10m

We can see that the my-oke-secret has been created. Let’s have a peek inside the secret:

kubectl get secret my-oke-secret -o yaml
apiVersion: v1
data:
  key: YVNlY3JldEluU2lW
immutable: false
kind: Secret
metadata:
  annotations:
    reconcile.external-secrets.io/data-hash: cff53a4bddfc2d04ea50e6d0da2c2868
  creationTimestamp: "2022-06-01T02:11:43Z"
  name: my-oke-secret
  namespace: default
  ownerReferences:
  - apiVersion: external-secrets.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: ExternalSecret
    name: oci-secret
    uid: 0039c7ab-91c2-4e15-acc7-f8de87ba1254
  resourceVersion: "9117"
  uid: 3074d172-078f-4214-9f21-a6bc69aae182
type: Opaque

We can see that the key has been created. Let’s extract it to see if it matches what we created in the OCI Secret:

kubectl get secret my-oke-secret -o json | jq -r ."data.key" | base64 -d
aSecretInSiV

Excellent. This matches!

Updating the secret

Let’s now see if it can handle updates. In the OCI Secret, let’s create a new version:

Wait for version 2 to be active and current and run the command above again to extract the secret:

kubectl get secret my-oke-secret -o json | jq -r ."data.key" | base64 -d
SecretVersion2

The value in the Kubernetes Secret has been updated.

Using the ESO with another application in the cluster

Let’s see how we can use ESO to inject secrets into other applications running in the cluster. In this case, we’ll try to inject a password into Grafana.

Add the Grafana helm repo:

helm repo add grafana https://grafana.github.io/helm-chartshelm show values grafana/grafana > grafana.yaml

Edit the grafana.yaml and locate the Administrator credentials section:

admin:
  ## Name of the secret. Can be templated.
  existingSecret: "grafana-in-eso"
  userKey: admin-user
  passwordKey: admin-password

Change the existingSecret to as above (or a value of your choice).

Next, create a secret in OCI Vault as follows:

We can now create an ExternalSecret using the above:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: oci-secret
spec:
  refreshInterval: 15m
  secretStoreRef:
    kind: SecretStore
    name: alivault
  target:
    name: grafana-in-eso
    creationPolicy: Owner
  dataFrom:
  - extract:
      key: grafana

Let’s now deploy Grafana:

helm install grafana grafana/grafana -f grafana.yaml

Port-forward to the Grafana pod:

export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")kubectl --namespace default port-forward $POD_NAME 3000

Notice that we are now using the values we created in OCI Vault instead. We are able to login.

Let’s say we now want to change the admin username and password. Create a new version in OCI Secret:

Once it’s active, login again:

This will fail. The reason is that when the Grafana pod is created, it reads the secret and uses the admin-username and admin-password values from the secret and stores it in config file. So right now, even though the values in OCI Secret and the Kubernetes secret have changed, Grafana is not aware of this yet.

In order to use the new value, we have to update of this file. We can use Grafana’s REST API but I don’t know if Grafana has a listening mechanism to watch for secret changes (something to investigate in the future). For now, the easiest way to force a reload of the username and password is by deleting the Grafana pod to force its recreation by Kubernetes:

kubectl delete pod $POD_NAME

You’ll notice a new pod being created:

kubectl get pods
NAME                       READY   STATUS              RESTARTS  AGE
grafana-788976bc4c-q2swm   0/1     ContainerCreating   0         8s

Wait for it to be READY and port-forward to it with the values you used in version 2 of the secret. This time, you will be able to login to Grafana.

Note: this is not the recommended way of updating Grafana username and password. The point of this example is to show reloading of the secret from OCI Vault via ESO in an application.

Summary

The External Secrets Operator is an excellent way of integrating and using OCI OKE with OCI Vault. Both OKE and the applications running in OKE can use ESO to retrieve sensitive data from OCI Vault.

I hope you find this post useful.

Want to discuss? Join our public Slack channel!