Secret Management in EKS using SSM Parameter Store, KMS and ESO

Introduction

When deploying applications in Kubernetes, you are often confronted to secret management challenges. From Vault, to other providers it can be difficult to choose and implement a solution.

In this article I will show you how you can leverages AWS services to safely store secrets while ensuring multi-tenancy, local work and scalability.

We will leverages:

• SSM Parameter Store to store configs and secrets
• IAM to restrict access
• KMS to encrypt/decrypt secrets
• External Secret Operator to inject secrets in our Kubernetes Cluster

We will assume that:

• Nobody can manage namespaces except the platform team. Otherwise secrets can be exfiltrated
• Nobody can update ClusterSecretStore (or basically any Cluster Resources) except the platform team
• Nobody can create resources in a team namespace if they are not part of that team (Add Role and Rolebinding)

Presentation

Before starting let’s assume we have the following teams:

• teamone
• teamtwo

In my case we have an IDP configured with SAML however any other authentication mechanism can be used.

For each of our team we will create:

• A role to assume with an IAM policy that allows Create, Update, Delete and Get Parameter under path /projects/team
• A role that will be assumed by a service account deployed in external-secrets namespace (IRSA)
• A ClusterSecretStore with a condition on namespaces using label selector my-org/secret-store:

Here is the high level diagram:

As we can see here, for teamone, we have a condition on the namespace label my-org/secret-store=teamone-secret-store. Therefore if a user in teamtwo tries to reference the ClusterSecretStore teamone-ssm it will be rejected.

Implementation

IAM role my-org-teamone

Here is the trust relationship. This role can be assumed using SAML endpoint. It is up to your organization to set the role that can be assumed by your team

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Teamone"
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::000000000000:saml-provider/My-Saml"
            },
            "Action": "sts:AssumeRoleWithSAML",
            "Condition": {
                "StringEquals": {
                    "SAML:aud": "https://signin.aws.amazon.com/saml"
                }
            }
        }
    ]
}

And the inline policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "TeamoneSSM"
            "Action": [
                "ssm:PutParameter",
                "ssm:DeleteParameter",
                "ssm:DeleteParameters",
                "ssm:UpdateParameter",
                "ssm:GetParameter*"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:ssm:us-east-1:000000000000:parameter/projects/teamone*"
        }
    ]
}

IAM role teamone-secret-store

Here is the trust relationship. This role can be assumed by a service account teamone-secret-store under namespace external-secrets

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "TeamoneSecretStoreSA",
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::000000000000:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/00000000000000000000000000000000"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringLike": {
                    "oidc.eks.us-east-1.amazonaws.com/id/00000000000000000000000000000000:sub": "system:serviceaccount:external-secrets:teamone-secret-store"
                }
            }
        }
    ]
}

And the inline policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
           "Sid": "TeamoneSecretStoreSASSM"
           "Action": [
                "ssm:GetParameter*",
                "ssm:ListTagsForResource",
                "ssm:DescribeParameters"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:ssm:us-east-1:000000000000:parameter/projects/teamone*"
        }
    ]
}

KMS Key teamone

The KMS key will be dedicated to your team. It can be used to encrypt anything they want. For example encrypt a terraform state in S3… For our use case it will be used to encrypt / decrypt secret parameters in SSM.

To simplify, we will use an alias alias/teamone. This is really important that you define a KMS key policy to restrict who can use it. For our case admins and teamone will be admin on that KMS Key and teamone-secret-store will be able to Decrypt only:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AdminKMSAll",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::000000000000:role/admin"
            },
            "Action": "kms:*",
            "Resource": "*"
        },
        {
            "Sid": "TeamoneKMSAll",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::000000000000:role/my-org-teamone"
            },
            "Action": "kms:*",
            "Resource": "*"
        },
        {
            "Sid": "TeamoneSecretStoreSADecrypt",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::000000000000:role/teamone-secret-store"
            },
            "Action": "kms:Decrypt",
            "Resource": "*"
        }
    ]
}

Kubernetes Resources

We can now create the kubernetes resources.

First the service account teamone-secret-store. We will use IRSA to assume role teamone-secret-operator

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    # Assume role teamone-secret-store
    # This is allowed with Trust RelashionShip TeamoneSecretStoreSA
    eks.amazonaws.com/role-arn: 'arn:aws:iam::000000000000:role/teamone-secret-store'
  name: teamone-secret-store
  namespace: external-secrets

Then ClusterSecretStore that is the interface to SSM ParameterStore:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: teamone-ssm
spec:
  conditions:
    - namespaceSelector:
        matchLabels:
          my-org/secret-store: teamone-secret-store
  provider:
    aws:
      auth:
        jwt:
          serviceAccountRef:
            name: teamone-secret-store
            namespace: external-secrets
      region: us-east-1
      service: ParameterStore

Let’s test it!

Create/read secret

First we want to validate that teamone is not able to decrypt secret from teamtwo:

That’s a pretty good start! Our user from teamone is not able to decode teamtwo secrets.

However teamone can access secrets under /projects/teamone and encrypt/decrypt using alias/teamone KMS key:

Inject secrets

Let’s first create a namespace teamone-monitoring with label my-org/secret-store=teamone-secret-store.

Let’s now inject our secret using an ExternalSecret

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: teamone-secret-testsecured
  namespace: teamone-monitoring
spec:
  data:
    - secretKey: testSecured
      remoteRef:
        conversionStrategy: Default
        decodingStrategy: None
        key: /projects/teamone/testsecured
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: teamone-ssm
  target:
    creationPolicy: Owner
    deletionPolicy: Retain
    name: demo-secret
    template:
      data:
        testSecured: '{{ .testSecured }}'
      engineVersion: v2

If we retrieve it we can observe following status:

status:
  binding:
    name: demo-secret
  conditions:
  - lastTransitionTime: "2023-12-05T15:07:57Z"
    message: Secret was synced
    reason: SecretSynced
    status: "True"
    type: Ready
  refreshTime: "2023-12-05T15:07:57Z"
  syncedResourceVersion: 1-4e13cb3edea6fb7bf6b161fb8a410cc5

And if we get the secret:

$ kubectl -n teamone-monitoring get secret teamone-secret-testsecured -o jsonpath='{.data.testSecured}' | base64 -d
test

Validate no secret exfiltration

Let’s now try to exfiltrate a secret from teamtwo. We will deploy in our teamone namespace an ExternalSecret that will target teamtwo-ssm ClusterSecretStore to get the teamtwo secret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: teamone-secret-testsecured
  namespace: teamone-monitoring
spec:
  data:
    - secretKey: testSecured
      remoteRef:
        conversionStrategy: Default
        decodingStrategy: None
        key: /projects/teamtwo/test
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: teamtwo-ssm       # Try to use teamtwo-ssm from a namespace prefixed by teamone
  target:
    creationPolicy: Owner
    deletionPolicy: Retain
    name: demo-secret
    template:
      data:
        testSecured: '{{ .testSecured }}'
      engineVersion: v2

If we take a look at the status:

status:
  conditions:
  - lastTransitionTime: "2023-12-05T15:15:01Z"
    message: could not get secret data from provider
    reason: SecretSyncedError
    status: "False"
    type: Ready

And in the external secrets operator logs:

using cluster store \"teamtwo-ssm\" is not allowed from namespace \"teamone-monitoring\": denied by spec.condition

Local dev

For local dev you can create a script that fetch the secret from SSM and set it in an env variable

# AWS Login required first
export MY_SECRET=$(aws ssm get-parameter --name "/projects/teamone/test-secured" --query "Parameter.Value" --output=text --with-decryption)
# start my app

Conclusion

You are now able to easily manage secrets in your EKS cluster using AWS Services without “root” secret.

You can also create ClusterExternalSecret that will create an ExternalSecret in all the team namespaces. This helps when you need to populate a secret (such as a docker config to pull image)

Last remarks:

• Ensure that nobody can schedule pods in external-secret namespace otherwise he will be able to exfiltrate secrets
• Ensure that nobody except teamone can create resources (ExternalSecret, Pod) in teamone namespaces otherwise he will be able to exfiltrate secrets
• Ensure that nobody can update ClusterSecretStore (and more globally Cluster resources) otherwise he will be able to escalate resulting in secret exfiltration (or worse)
• More generally: Before implementing a solution, ensures it follows your organisation security guidelines

Disclaimer

The content provided in this article is for informational purposes only. While efforts have been made to ensure accuracy, readers should conduct their own research before implementing any techniques discussed. The author and contributors are not liable for any losses or damages arising from the use of this information. By accessing this article, readers agree to do so at their own risk and acknowledge their responsibility for any actions taken based on its content.