Manage Kubernetes Secrets with Crossplane and External Secrets

How to easily and securely store your Kubernetes secrets in AWS Secret Manager with Argo CD, Crossplane and External Secrets Operator.

Note: Though this article and the demo in it are heavily focused on AWS and secret management specifically, the principles presented here can be easily applied on any major cloud provider and for any resources provided by it.

Preamble

In HiredScore, we, as many other companies do, run the lion’s share of our infrastructure on Kubernetes. One problem we encountered, though admittedly not a very complicated one, is secret management. We wanted to keep all our secrets in a safe and not Kubernetes reliant service.

AWS Secret Manager

As we already run on AWS EKS, choosing their Secret Manager was pretty easy. This service allows full secret lifecycle management through the comfort of the AWS ecosystem.

External Secrets Operator

We then needed to decide on how to create AWS Secret Manager backed secrets inside our Kubernetes clusters. First we considered Secrets Store CSI driver (“SSCSID”), but we finally chose External Secrets Operator (“ESO”) as it looked (and still looks) as the best solution for our needs. An in-depth comparison between the two can be found here.

When using ESO for AWS Secret Manager, you need to deploy two types of objects.

• SecretStore that points to AWS Secrets Manager in a certain account within a defined region.
• ExternalSecrets that define the Kubernetes Secrets and which AWS secrets they rely on.

Crossplane

Next we encountered the final obstacle in our secret management quest - permissions. ESO needs AWS permissions to access the secrets. We considered several solutions including Terraform and eksctl. But they all increased the complexity of our deployments and made the DevOps involvement in the procedure higher from the desired none.

Crossplane (“XP”) is a CNCF (at the time of writing) incubating project that allows using the Kubernetes control plane for management of resources outside it. In our use-case we used the Upbound provider-aws-iam provider to provision the IAM Role and an inline Policy that will be used to access the AWS secrets in conjunction with the consuming resources themselves.

This way the chart and its external AWS dependencies (in this case it’s just IAM Roles and Policies, but it can be anything) are deployed together using GitOps principles by Argo CD, minimizing developer input and DevOps involvement.

How Does it Work?

The architecture is pretty simple, we provision all the resources (except the AWS Secret Manager secret) through Argo CD with a Helm Chart. Using a chart dedicated for those resources allows us to keep everything DRY and easily extend existing charts with ours as a dependency or any differently managed Kubernetes resources (e.g. with Kustomize).

High level architecture

The secret needs to be created manually beforehand as it’s assumed its contents are not randomly generated but hold some meaning (e.g. OIDC configuration). Otherwise we could simply generate the secret every time.

The access flow is roughly as following:

• The SecretStore uses the ServiceAccount to Access AWS for the Kubernetes secret creation.
• The ServiceAccount uses the newly provisioned IAM role attached to it with an annotation.
• The IAM Role uses its inline policy to access to the AWS secret.

Demo

For the demo we’ll skip the ArgoCD part in favor of a simple helm install and mostly skip how the mentioned components need to be deployed securely in production environments for the sake of simplicity.

Create an AWS Secret Manager Secret

Fastest way to create one is with aws CLI.

aws secretsmanager create-secret \
  --name aws-test-secret \
  --secret-string "{\"user\":\"eso\",\"password\":\"crossplane\"}"

Create an AWS EKS Cluster with OIDC

Fastest way to deploy is with eksctl, or to bring your own.

brew install eksctl # or any other package manager
eksctl create cluster crossplane-eso-demo --with-oidc
aws eks describe-cluster --name crossplane-eso-demo --query "cluster.identity.oidc.issuer" --output text | cut -d/ -f5

Make note of the cluster ID, we’ll need it later.

Install External Secrets Operator

Fastest way to deploy is with Helm.

helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm install external-secrets external-secrets/external-secrets \
  -n external-secrets --create-namespace

Install Crossplane and the provider

XP too can be deployed easily with Helm.

helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update
helm install crossplane crossplane-stable/crossplane \
  -n crossplane-system --create-namespace

We then need to create an AWS role for the provider to use in AWS resource creation. This can be done with following script.

#!/usr/bin/env bash
set -e

ACCOUNT_ID='<aws-account-id>'
REGION='<aws-region>'
EKS_CLUSTER_NAME='crossplane-eso-demo'
EKS_CLUSTER_ID=$(aws eks describe-cluster --region "$REGION" --name "$EKS_CLUSTER_NAME" --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)

cat <<-EOF > policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::$ACCOUNT_ID:oidc-provider/oidc.eks.$REGION.amazonaws.com/id/${EKS_CLUSTER_ID}"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringLike": {
                    "oidc.eks.$REGION.amazonaws.com/id/${EKS_CLUSTER_ID}:sub": "system:serviceaccount:crossplane-system:provider-aws-*"
                }
            }
        }
    ]
}
EOF

aws iam create-role --region "$REGION" --role-name "$EKS_CLUSTER_NAME" --assume-role-policy-document file://policy.json --query "Role.Arn" --output text
aws iam attach-role-policy --region "$REGION" --role-name "$EKS_CLUSTER_NAME" --policy-arn aarn:aws:iam::aws:policy/IAMFullAccess
rm -f policy.json

Note: in production environments it’s important to limit which XP resources can be provisioned and by whom with RBAC and a policy engine (e.g. Kyverno).

Then we need to create the ContollerConfig that our provider will use, this can be done with kubectl apply -f to the following file.

apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
  name: irsa-controllerconfig
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<aws-account-id>:role/eks-test-role #the ARN from before
spec:

Now we can install the provider itself, this too can be done with kubectl apply -f (the demo was tested on v0.41.0).

apiVersion: pkg.crossplane.io/v1 
kind: Provider 
metadata:
  name: provider-aws-iam
spec:
  package: xpkg.upbound.io/upbound/provider-aws-iam:latest
  controllerConfigRef:
    name: irsa-controllerconfig

And to finish up with XP, we need to create this ProviderConfig to configure the provider to authenticate with IRSA by, you guessed it, kubectl apply -f to the following file.

apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: IRSA

Install the demo chart

After finishing with the endless prerequisites we can finally get to business. First clone the Git repository with the Helm chart.

git clone https://github.com/alex-souslik-hs/aws-crossplane-eso-demo.git
cd aws-crossplane-eso-demo/chart

Afterwards we need to fill the values.yaml.

aws:
  enabled: true
  accountId: "<aws-account-id>"
  clusterName: crossplane-eso-demo
  clusterId: <aws-cluster-id> # the ID from before
  region: <aws-region>

  secretManager:
    enabled: true
    secrets:
      - name: k8s-test-secret
        data:
          - secretKey: user
            remoteRef:
              key: aws-test-secret
              property: user
          - secretKey: password
            remoteRef:
              key: aws-test-secret
              property: password

serviceAccount:
  create: true

And then finally we can deploy with Helm.

helm install demo . -f values.yaml --wait

After waiting a few minutes for everything to be up and ready we can get the k8s-test-secret and make sure the contents are correct.

kubectl get secret k8s-test-secret -o json | jq '.data | map_values(@base64d)'

Don’t forget to clean up!

Conclusion

So what did we see here today?

For the DevOps

• Much simpler and more automated approach to cloud dependency management (not just secrets).
• DRY modular approach of dependency charts for common resources.

For the Developer

• Less mundane stuff to do before deploying the app.
• Less reliance on the DevOps.

Final Thoughts

In HiredScore, we always aim to keep things DRY and minimize manual intervention wherever possible, this allows speeding up development cycles, making developer teams independent and maintaining a high standard in our infrastructure.

I hope you found this post helpful and informational. While this use-case is fairly simple it can be easily extended and modified to your needs.

Interested in this type of work? We’re looking for talented people to join our team!