Manage Kubernetes Secrets with Crossplane and External Secrets
How to easily and securely store your Kubernetes secrets in AWS Secret Manager with Argo CD, Crossplane and External Secrets Operator.
Note: Though this article and the demo in it are heavily focused on AWS and secret management specifically, the principles presented here can be easily applied on any major cloud provider and for any resources provided by it.
Preamble
In HiredScore, we, as many other companies do, run the lion’s share of our infrastructure on Kubernetes. One problem we encountered, though admittedly not a very complicated one, is secret management. We wanted to keep all our secrets in a safe and not Kubernetes reliant service.
AWS Secret Manager
As we already run on AWS EKS, choosing their Secret Manager was pretty easy. This service allows full secret lifecycle management through the comfort of the AWS ecosystem.
External Secrets Operator
We then needed to decide on how to create AWS Secret Manager backed secrets inside our Kubernetes clusters. First we considered Secrets Store CSI driver (“SSCSID”), but we finally chose External Secrets Operator (“ESO”) as it looked (and still looks) as the best solution for our needs. An in-depth comparison between the two can be found here.
When using ESO for AWS Secret Manager, you need to deploy two types of objects.
SecretStorethat points to AWS Secrets Manager in a certain account within a defined region.ExternalSecretsthat define the KubernetesSecretsand which AWS secrets they rely on.
Crossplane
Next, we encountered the final obstacle in our secret management quest - permissions. ESO needs AWS permissions to access the secrets. We considered several solutions, including Terraform and eksctl. But they all increased the complexity of our deployments and made the DevOps involvement in the procedure higher from the desired none.
Crossplane (“XP”) is a CNCF (at the time of writing) incubating project that allows using the Kubernetes control plane for management of resources outside it. In our use-case, we used the Upbound provider-aws-iam provider to provision the IAM Role and an inline Policy that will be used to access the AWS secrets in conjunction with the consuming resources themselves.
This way the chart and its external AWS dependencies (in this case it’s just IAM Roles and Policies, but it can be anything) are deployed together using GitOps principles by Argo CD, minimizing developer input and DevOps involvement.
How Does it Work?
The architecture is pretty simple, we provision all the resources (except the AWS Secret Manager secret) through Argo CD with a Helm Chart. Using a chart dedicated for those resources allows us to keep everything DRY and easily extend existing charts with ours as a dependency or any differently managed Kubernetes resources (e.g. with Kustomize).
The secret needs to be created manually beforehand, as it’s assumed its contents are not randomly generated but hold some meaning (e.g. OIDC configuration). Otherwise, we could simply generate the secret every time.
The access flow is roughly as following:
- The
SecretStoreuses theServiceAccountto Access AWS for the Kubernetes secret creation. - The
ServiceAccountuses the newly provisioned IAM role attached to it with an annotation. - The IAM Role uses its inline policy to access to the AWS secret.
Demo
For the demo, we’ll skip the Argo CD part in favour of a simple helm install and mostly skip how the mentioned components need to be deployed securely in production environments for the sake of simplicity.
Create an AWS Secret Manager Secret
The fastest way to create one is with aws CLI.
aws secretsmanager create-secret \
--name aws-test-secret \
--secret-string "{\"user\":\"eso\",\"password\":\"crossplane\"}"Create an AWS EKS Cluster with OIDC
The fastest way to deploy is with eksctl, or to bring your own.
brew install eksctl # or any other package manager
eksctl create cluster crossplane-eso-demo --with-oidc
aws eks describe-cluster --name crossplane-eso-demo --query "cluster.identity.oidc.issuer" --output text | cut -d/ -f5Make note of the cluster ID, we’ll need it later.
Install External Secrets Operator
The fastest way to deploy is with Helm.
helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm install external-secrets external-secrets/external-secrets \
-n external-secrets --create-namespaceInstall Crossplane and the provider
XP too can be deployed easily with Helm.
helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update
helm install crossplane crossplane-stable/crossplane \
-n crossplane-system --create-namespaceWe then need to create an AWS role for the provider to use in AWS resource creation. This can be done with following script.
#!/usr/bin/env bash
set -e
ACCOUNT_ID='<aws-account-id>'
REGION='<aws-region>'
EKS_CLUSTER_NAME='crossplane-eso-demo'
EKS_CLUSTER_ID=$(aws eks describe-cluster --region "$REGION" --name "$EKS_CLUSTER_NAME" --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)
cat <<-EOF > policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::$ACCOUNT_ID:oidc-provider/oidc.eks.$REGION.amazonaws.com/id/${EKS_CLUSTER_ID}"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringLike": {
"oidc.eks.$REGION.amazonaws.com/id/${EKS_CLUSTER_ID}:sub": "system:serviceaccount:crossplane-system:provider-aws-*"
}
}
}
]
}
EOF
aws iam create-role --region "$REGION" --role-name "$EKS_CLUSTER_NAME" --assume-role-policy-document file://policy.json --query "Role.Arn" --output text
aws iam attach-role-policy --region "$REGION" --role-name "$EKS_CLUSTER_NAME" --policy-arn aarn:aws:iam::aws:policy/IAMFullAccess
rm -f policy.jsonNote: in production environments it’s important to limit which XP resources can be provisioned and by whom with RBAC and a policy engine (e.g. Kyverno).
Then we need to create the DeploymentRuntimeConfig that our provider will use, this can be done with kubectl apply -f the following file.
apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
name: aws
spec:
serviceAccountTemplate:
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::<aws-account-id>:role/eks-test-role #the ARN from beforeNote: in production environments, it’s important to configure resources and replicas to the provider deployment via the spec.deploymentTemplate object.
Now we can install the provider itself, this too can be done with kubectl apply -f (the demo was tested on v1.15.0).
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
name: provider-aws-iam
spec:
package: xpkg.upbound.io/upbound/provider-aws-iam:latest
runtimeConfigRef:
name: awsAnd to finish up with XP, we need to create this ProviderConfig to configure the provider to authenticate with IRSA by, you guessed it, kubectl apply -f to the following file.
apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
name: default
spec:
credentials:
source: IRSAInstall the demo chart
After finishing with the endless prerequisites, we can finally get to business. First clone the Git repository with the Helm chart.
git clone https://github.com/alex-souslik-hs/aws-crossplane-eso-demo.git
cd aws-crossplane-eso-demo/chartAfterwards, we need to fill the values.yaml.
aws:
enabled: true
accountId: "<aws-account-id>"
clusterName: crossplane-eso-demo
clusterId: <aws-cluster-id> # the ID from before
region: <aws-region>
secretManager:
enabled: true
secrets:
- name: k8s-test-secret
data:
- secretKey: user
remoteRef:
key: aws-test-secret
property: user
- secretKey: password
remoteRef:
key: aws-test-secret
property: password
serviceAccount:
create: trueAnd then finally we can deploy with Helm.
helm install demo . -f values.yaml --waitAfter waiting a few minutes for everything to be up and ready, we can get the k8s-test-secret and make sure the contents are correct.
kubectl get secret k8s-test-secret -o json | jq '.data | map_values(@base64d)'Don’t forget to clean up!
Conclusion
So what did we see here today?
For the DevOps
- Much simpler and more automated approach to cloud dependency management (not just secrets).
- DRY modular approach of dependency charts for common resources.
For the Developer
- Less mundane stuff to do before deploying the app.
- Less reliance on the DevOps.
Final Thoughts
In HiredScore, we always aim to keep things DRY and minimize manual intervention wherever possible, this allows speeding up development cycles, making developer teams independent and maintaining a high standard in our infrastructure.
I hope you found this post helpful and informational. While this use-case is fairly simple, it can be easily extended and modified to your needs.
Interested in this type of work? We’re looking for talented people to join our team!
