Oracle Developers
Published in

Oracle Developers

Using the External Secrets Operator with OCI Kubernetes and OCI Vault

A customer recently pointed me to the External Secrets Operator (ESO) project and I was pleased to see it already has integration with OCI Vault. In this article, we’ll take ESO for a spin and see how we can use it with OKE.

Installing the External Secrets Operator on OKE

First, create an OKE Cluster either using the Quick Create in the OCI Console or using https://github.com/oracle-terraform-modules/terraform-oci-oke. You can now use Cloud Shell or the operator host in the terraform-oci-oke module to run kubectl or helm commands.

Let’s use the helm chart to install the ESO:

helm repo add external-secrets https://charts.external-secrets.iohelm install external-secrets \
external-secrets/external-secrets \
-n external-secrets \
--create-namespace

Configuring ESO for use with OCI Vault

If you are not using instance_principal, create a Kubernetes secret in a yaml file to hold the the private key and the fingerprint:

apiVersion: v1
kind: Secret
metadata:
name: pk-fp
labels:
type: oracle
type: Opaque
stringData:
privateKey: |
-----BEGIN RSA PRIVATE KEY-----
abcdefg1234567890/+-
-----END RSA PRIVATE KEY-----
fingerprint: ab:cd:ef:gh:12:34

And create the secret:

kubectl create -f pk-fp.yaml

You can now create the SecretStore:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: alivault
spec:
provider:
oracle:
vault: # The vault OCID
region: # The vault region
auth:
user: # A user OCID
tenancy: # A user's tenancy
secretRef:
privatekey:
name: pk-fp
key: privateKey
fingerprint:
name: pk-fp
key: fingerprint

N.B. If you use instance_principals, remove the section from auth to the end. You also need to create a dynamic group and add the worker nodes to them. Finally, you also need to create a policy to allow the dynamic group to use the key which you’ll be creating in OCI Vault.

kubectl create -f oci-secret-store.yaml

Create an ExternalSecret

Let’s now create an ExternalSecret using ESO:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: oci-secret
spec:
refreshInterval: 0.03m
secretStoreRef:
kind: SecretStore
name: alivault # Must match SecretStore on the cluster
target:
name: my-oke-secret # Name for the secret on the cluster
creationPolicy: Owner
data:
- secretKey: key
remoteRef:
key: my-eso-secret

my-eso-secret is the name of the OCI Secret in Vault:

In this case, it just has a plain content (“aSecretInSiV”) so we use data as above and we can create the ExternalSecret:

kubectl create -f ext-secret.yaml

Get the external secrets:

kubectl get es
NAME STORE REFRESH INTERVAL STATUS
oci-secret alivault 0.03m SecretSynced

We can see that the secret has been synced. Let’s list the secrets in this namespace:

kubectl get secrets
NAME TYPE DATA AGE
default-token-mnpsm kubernetes.io/service-account-token 3 41m
my-oke-secret Opaque 1 108s
pk-fp Opaque 2 10m

We can see that the my-oke-secret has been created. Let’s have a peek inside the secret:

kubectl get secret my-oke-secret -o yaml
apiVersion: v1
data:
key: YVNlY3JldEluU2lW
immutable: false
kind: Secret
metadata:
annotations:
reconcile.external-secrets.io/data-hash: cff53a4bddfc2d04ea50e6d0da2c2868
creationTimestamp: "2022-06-01T02:11:43Z"
name: my-oke-secret
namespace: default
ownerReferences:
- apiVersion: external-secrets.io/v1beta1
blockOwnerDeletion: true
controller: true
kind: ExternalSecret
name: oci-secret
uid: 0039c7ab-91c2-4e15-acc7-f8de87ba1254
resourceVersion: "9117"
uid: 3074d172-078f-4214-9f21-a6bc69aae182
type: Opaque

We can see that the key has been created. Let’s extract it to see if it matches what we created in the OCI Secret:

kubectl get secret my-oke-secret -o json | jq -r ."data.key" | base64 -d
aSecretInSiV

Excellent. This matches!

Updating the secret

Let’s now see if it can handle updates. In the OCI Secret, let’s create a new version:

Wait for version 2 to be active and current and run the command above again to extract the secret:

kubectl get secret my-oke-secret -o json | jq -r ."data.key" | base64 -d
SecretVersion2

The value in the Kubernetes Secret has been updated.

Using the ESO with another application in the cluster

Let’s see how we can use ESO to inject secrets into other applications running in the cluster. In this case, we’ll try to inject a password into Grafana.

Add the Grafana helm repo:

helm repo add grafana https://grafana.github.io/helm-chartshelm show values grafana/grafana > grafana.yaml

Edit the grafana.yaml and locate the Administrator credentials section:

admin:
## Name of the secret. Can be templated.
existingSecret: "grafana-in-eso"
userKey: admin-user
passwordKey: admin-password

Change the existingSecret to as above (or a value of your choice).

Next, create a secret in OCI Vault as follows:

We can now create an ExternalSecret using the above:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: oci-secret
spec:
refreshInterval: 0.03m
secretStoreRef:
kind: SecretStore
name: alivault
target:
name: grafana-in-eso
creationPolicy: Owner
dataFrom:
- extract:
key: grafana

Let’s now deploy Grafana:

helm install grafana grafana/grafana -f grafana.yaml

Port-forward to the Grafana pod:

export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")kubectl --namespace default port-forward $POD_NAME 3000

Notice that we are now using the values we created in OCI Vault instead. We are able to login.

Let’s say we now want to change the admin username and password. Create a new version in OCI Secret:

Once it’s active, login again:

This will fail. The reason is that when the Grafana pod is created, it reads the secret and uses the admin-username and admin-password values from the secret and stores it in config file. So right now, even though the values in OCI Secret and the Kubernetes secret have changed, Grafana is not aware of this yet.

In order to use the new value, we have to update of this file. We can use Grafana’s REST API but I don’t know if Grafana has a listening mechanism to watch for secret changes (something to investigate in the future). For now, the easiest way to force a reload of the username and password is by deleting the Grafana pod to force its recreation by Kubernetes:

kubectl delete pod $POD_NAME

You’ll notice a new pod being created:

kubectl get pods
NAME READY STATUS RESTARTS AGE
grafana-788976bc4c-q2swm 0/1 ContainerCreating 0 8s

Wait for it to be READY and port-forward to it with the values you used in version 2 of the secret. This time, you will be able to login to Grafana.

Note: this is not the recommended way of updating Grafana username and password. The point of this example is to show reloading of the secret from OCI Vault via ESO in an application.

Summary

The External Secrets Operator is an excellent way of integrating and using OCI OKE with OCI Vault. Both OKE and the applications running in OKE can use ESO to retrieve sensitive data from OCI Vault.

I hope you find this post useful.

Want to discuss? Join our public Slack channel!

--

--

--

Aggregation of articles from Oracle engineers, Groundbreaker Ambassadors, Oracle ACEs, and Java Champions on all things Oracle technology. The views expressed are those of the authors and not necessarily of Oracle.

Recommended from Medium

We’re shutting down Lisn

Configure LoadBalaner (HA-Proxy) & Webserver using Ansible on AWS

1 https://worldnewsguru.us/business/food-delivery-mobile-application-market-to-reach-62-83-bn-global

Top 3 Reasons Why You Should Sign-up for a Hackathon Today!

鬼滅之刃劇場版 無限列車篇~線上看小鴨完整高清电影[HD-2020]~BLURAY

Cloud ERP — Thinking Smarter!

Testing Principles 101

Why should you learn Go?

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ali Mukadam

Ali Mukadam

More from Medium

Introduction to drone.io CI/CD platform Part 2

Choosing the Best Kubernetes Cluster and Application Deployment Strategies

Demystifying Kubernetes networking using tcpdump

Load Balancing with MetalLB on bare metal Kubernetes