Amazon KMS for a Kops Kuberenetes Cluster

Gayatri S Ajith
Jul 1 · 6 min read

When using Amazon EKS you can configure a AWS KMS key to encrypt etcd. How to do that for a cluster created using kops? It took me a few hours to understand the whole flow. So, here goes.

What is Kops?

Pre-requisites

  • The AWS CLI Amazon AWS user that was used to create the cluster should be given permissions to manage Amazon KMS Keys. I gave my kops Amazon AWS user AWSKeyManagementServicePowerUser permissions. (I am sure its not safe to give such elevated permissions but, I didn’t have the patience or need to explore further for my development cluster. Do leave a comment if you find out the exact permissions needed.)

Step 1: Create the Amazon KMS master key

You can create this Key via the Amazon AWS Web Console or if your AWS CLI user has enough permissions you can create it via AWS CLI too.

KEY_ID=$(aws kms create-key --query KeyMetadata.KeyId --output text)
aws kms describe-key --key-id $KEY_ID

You will get a long JSON output that will looks like this. Note the ARN value — we will need it.

{
"KeyMetadata": {
"Origin": "AWS_KMS",
"KeyId": "1234abcd-12ab-34cd-56ef-1234567890ab",
"Description": "",
"KeyManager": "CUSTOMER",
"Enabled": true,
"KeyUsage": "ENCRYPT_DECRYPT",
"KeyState": "Enabled",
"CreationDate": 1403010355.345,
"Arn": "arn:aws:kms:ap-south-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
"AWSAccountId": "111122223333"
}
}

Step 2: Create the KMS Provider Docker Image

There is a ready provider program for Amazon KMS here: https://github.com/kubernetes-sigs/aws-encryption-provider

Clone the project, build the docker image and we will the imageto create the container which will run the Amazon KMS provider program.

git clone https://github.com/kubernetes-sigs/aws-encryption-provider
cd aws-encryption-provider/
docker build -t <the name for your Docker Image> ./
[DONT FORGET TO PUSH YOUR DOCKER IMAGE TO A REPO LIKE DOCKER HUB]

Step 3: Add a label to all Master nodes

# List all the instance groups
kops get ig --name <CLUSTER NAME> --state <STATE S3 BUCKET>
# Edit the master instance group
kops edit ig <MASTER IG NAME> --name <CLUSTER NAME> --state <STATE S3 BUCKET>
kind: InstanceGroup
spec:
nodeLabels:
dedicated: master # this is the label
role: Master

Step 4: Create the Amazon KMS Provider container on the Master node

On the master nodes, manifest files for static pods are found in /etc/kubernetes/manifests. So we need to create our Amazon KMS Provider’s static pod manifest file in this path too.

How do we create static pods in a kops managed cluster? Using the fileAssets key in the Cluster spec. fileAssets allows you to create files (using inline file contents) on nodes in the cluster. You can use roles to decide which type of nodes you want this file to be created on. This ensures that even if you have multiple master, the provider static pod manifest file will be created on all of them.

# Edit the cluster object spec
kops edit cluster --name <CLUSTER NAME> --state <STATE S3 BUCKET>
kind: Cluster
spec:
fileAssets:
- name: aws-encryption-provider.yaml # name of the file we want to create
path: /etc/kubernetes/manifests/aws-encryption-provider.yaml
roles:
- Master # want to create this on Master nodes
content: | # contents of the file
apiVersion: v1
kind: Pod
metadata:
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ""
labels:
k8s-app: aws-encryption-provider
name: aws-encryption-provider
namespace: kube-system
spec:
containers:
- image: <THE AWS PROVIDER DOCKER IMAGE YOU BUILT>
name: aws-encryption-provider
command:
- /aws-encryption-provider
- --key=<ARN OF THE KEY YOU CREATED>
- --region=<THE AWS REGION OF THE KEY>
- --listen=/srv/kubernetes/socket.sock
- --health-port=:8083
ports:
- containerPort: 8083
protocol: TCP
livenessProbe:
httpGet:
path: /healthz
port: 8083
volumeMounts:
- mountPath: /srv/kubernetes
name: kmsplugin
hostNetwork: true
priorityClassName: system-cluster-critical
volumes:
- name: kmsplugin
hostPath:
path: /srv/kubernetes
type: DirectoryOrCreate

Save the file and when you exit the Cluster Objects specs will be saved. To update the cluster and roll the changes to all the nodes in the cluster execute these. Give the cluster 10–15 mins to get ready.

kops update cluster --name <CLUSTER NAME> --state <STATE S3 BUCKET> --yes --admin
kops rolling-update cluster --name <CLUSTER NAME> --state <STATE S3 BUCKET> --yes

Step 5: Give Master node’s IAM Role permissions to manage KMS

# Edit the cluster object spec
kops edit cluster --name <CLUSTER NAME> --state <STATE S3 BUCKET>
kind: Cluster
spec:
additionalPolicies:
master: |
[
{
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:Encrypt"
],
"Resource": [
"<ARN OF THE KEY YOU CREATED>"
]
}
]
kops update cluster --name <CLUSTER NAME> --state <STATE S3 BUCKET> --yes --admin

Save the file and when you exit the Cluster Objects specs will be saved. To update the cluster and roll the changes to all the nodes in the cluster execute these. Give the cluster 10–15 mins to get ready.

kops update cluster --name <CLUSTER NAME> --state <STATE S3 BUCKET> --yes --admin
kops rolling-update cluster --name <CLUSTER NAME> --state <STATE S3 BUCKET> --yes

STEP 6: Create the encryption configuration file

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- kms:
name: aws-encryption-provider
endpoint: unix:///srv/kubernetes/socket.sock
cachesize: 1000
timeout: 3s
- identity: {}

This encryption configuration file is telling K8s to use the KMS provider. And we are giving the path to the socket that will be called when the cluster needs to handle encryption. This socket is created/managed by our static pod. Look the socket path. It’s the same that is mounted into our static pod.

Note: There are different types of encryption providers available. KMS is just one of them. This link has more info about the other types of providers: https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#understanding-the-encryption-at-rest-configuration

STEP 7: Tell Kops about the encryption configuration file

kops create secret encryptionconfig -f <PATH TO ENCRYPTION CONFIG FILE> --name <CLUSTER NAME> --state <STATE S3 BUCKET>
kops update cluster <CLUSTER NAME> --state <STATE S3 BUCKET> --yes --admin

STEP 8: Enable encryption configuration in the cluster object

# Edit the cluster object spec
kops edit cluster --name <CLUSTER NAME> --state <STATE S3 BUCKET>
kind: Cluster
spec:
encryptionConfig: true

Save the file and when you exit the Cluster Objects specs will be saved. To update the cluster and roll the changes to all the nodes in the cluster execute these. Give the cluster 10–15 mins to get ready.

kops update cluster --name <CLUSTER NAME> --state <STATE S3 BUCKET> --yes --admin
kops rolling-update cluster --name <CLUSTER NAME> --state <STATE S3 BUCKET> --yes

And, that’s it…

# Create the secret object
kubectl create secret generic secret1 -n default --from-literal=mykey=allgood
# Enter the ETCD-Server
kubectl exec -it -n kube-system $(kubectl get pods -n kube-system | grep etcd-manager-main | awk '{print $1}') sh
# Query ETCD
cd /opt/etcd-v3.3.10-linux-amd64/
ETCDCTL_API=3 ./etcdctl \
--key /rootfs/etc/kubernetes/pki/kube-apiserver/etcd-client.key \
--cert /rootfs/etc/kubernetes/pki/kube-apiserver/etcd-client.crt \
--cacert /rootfs/etc/kubernetes/pki/kube-apiserver/etcd-ca.crt \
--endpoints "https://127.0.0.1:4001" get /registry/secrets/default/secret1

Your output will look something like this. The value should be encrypted and you will notice the provider name in the output.

/registry/secrets/default/secret10o0m0hc:`?He.0:aws-encryption-provider:?1xf???Ԥ????ʭ?i?$?I?M7?ۜ??c??(!?_B5 ?p?ؙ~0| *?H??ka?H|C?[?OS?????٫?e??_!??ݥqx7f? E?`?{K{?t???;}??m??m?????|s??N^i1???,?b;?&?`l[{R4&c??Թ}?????)?`?{????-?T??˽4??|??T?+?$年g<VHAĎ????Ǧo>? ]W???:?% ??Z7:?t?u?B??ء?w?G?3??7??@)\ x?????o?_???'?"?3F??K? "ͤ6???.??????5?+?????H?\~YF????j???:$ϯOI8????????.0?

Remember, technically, you can make all the changes together and update the cluster once. I’m sharing the steps as I executed it!

Reference Links

Geek Culture

Proud to geek out. Follow to join our +1.5M monthly readers.