Encrypting Secret Data at Rest with Kops

Steering you towards the cliffs since 1981

Configuring Kubernetes experimental encryption provider config in Kops is pretty straight-forward.

The functionality was added in September 2017, but googling for how to set it up doesn’t necessarily land you in success land, but you’re a few easy steps away!

First, you’ll need to base64 encoded secret key to encrypt your other secrets with. I keep this little script handy in ~/bin/encryptionconfig to generate them.

My secrets are all “bananas”

1. Generate a config: ~/bin/encryptionconfig > conf.yaml
2. Upload the config: kops create secret encryptionconfig -f ./conf.yaml

You can verify the file was uploaded to your kops state store:

aws s3 ls ${KOPS_STATE_STORE/${KOPS_CLUSTER_NAME}/secrets/encryptionconfig

You can read more about the support encryption types here.

3. Set spec.encryptionConfig: true in the cluster’s config. Run kops edit cluster and add the following:

spec:
  encryptionConfig: true

This sets the --experimental-encryption-provider-config flag on the kube-apiserver for you to the correct value for kops which is /srv/kubernetes/encryptionconfig.yaml

4. Update that cluster!

kops update cluster --yes
kops rolling-update cluster --yes

You’re done!

You can SSH into one of your masters to verify the file is present at /srv/kubernetes/encryptionconfig.yaml

To verify that your secrets are in fact being encrypted. Lets generate a new one:

kubectl create secret generic secret1 -n default --from-literal=mykey=mydata

Now from one of your masters and run:

ETCDCTL_API=3 etcdctl get /registry/secrets/default/secret1 ETCD_FLAGS_HERE | hexdump -C

The secret should be prefixed with k8s:enc:YOUR_SELECTED_ENCRYPTION_SCHEME_HERE:v1: .

Sources:

• https://github.com/kubernetes/kops/blob/master/docs/cli/kops_create_secret_encryptionconfig.md
• https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/
• https://github.com/kubernetes/kops/blob/master/nodeup/pkg/model/kube_apiserver.go#L61
• https://github.com/georgebuckerfield/kops/blob/master/pkg/apis/kops/cluster.go#L162