Updating Secrets from a Kubernetes Pod
Sometimes you might want to update secrets from within a Kubernetes Pod. For example, if you have the access token and refresh token for an OAuth application, you might need to update these every time you refresh them. I’m sure there’s a few different ways to go about this, and there’s always the option of just storing these in a DB. In my case, I needed to refresh my tokens for Fitbit and wanted to keep them as secrets in Kubernetes. After reading some stuff online, I found the kubectl patch
command, which seemed to be what I wanted. The full command would be something like:
kubectl patch secret <secret> -p='{"data":{"<key>": "<base64_encoded_val>"}}
I wanted to do this in Python, so I wrote the following functions, which worked great from my machine:
from base64 import b64encode
from subprocess import rundef b64encodestr(string):
return b64encode(string.encode("utf-8")).decode()def update_secrets(secret, key, val):
b64val = b64encodestr(val)
cmd = f"""kubectl patch secret {secret} -p='{{"data":{{"{key}": "{b64val}"}}}}'"""
return run(cmd, shell=True)
Then I created a simple DAG using the KubernetesPodOperator that would update my airflow
secret by updating TEST_SECRET
with the current timestamp every minute. To use this method, you first need to make sure you have kubectl
installed in the image that you’re using. When I first ran it, I got the following error:
Error from server (Forbidden): secrets "airflow" is forbidden: User "system:serviceaccount:default:default" cannot get resource "secrets" in API group "" in the namespace "default"
After reading a bit, I found this was due to roles and permissions. In order to fix it, I first needed to create a role that would allow updating secrets. I did so with the following command (note that you’ll need the Kubernetes Engine Admin
role assigned from GCP to do run these):
kubectl create role update-secrets --verb=get,patch --resource=secrets
Once I’d created the role, I had to assign it to my default
service account on the default
namespace (which is the default setup in pods spun up by KubernetesPodOperator). You can do so with the following:
kubectl create rolebinding --role=update-secrets default-update-secrets --serviceaccount=default:default
After this, everything is working as expected! Now I can update my access and refresh tokens for my fitbit account when pulling data each day!
NOTE: If you’re just using Python, you could also use the kubernetes
python library and the patch_namespaced_secret
function. For the purpose of this article I decided to use the most “portable” option.