Kubernetes Self-managed Cluster with AWS IAM OIDC

3 min readApr 30, 2022

Summary

This article is about setting up the pod authentication with IAM role on self-managed K8s cluster using the native AWS IAM OIDC provider. This feature also eliminates the need for third-party solutions such as kiam or kube2iam. If you are setting up for the EKS cluster, you can easily achieve by following the AWS documentation. This is part of my K8s learning journey and I would like to try it in hard way with self-managed cluster instead of using the EKS. I use Terraform to deploy the resources on AWS.

You can checkout my GitHub repo to see the full deployment.

I will skip the infrastructure deployment and will go straight to OIDC part.

First thing first enable the feature on API server

I use the below parameters to enable the service account token volume projection. Great article about how projected service account tokens work.

--feature-gates="ServiceAccountIssuerDiscovery=true"
--service-account-issuer=https://<CLUSTER-DNS>
--service-account-jwks-uri=https://<CLUSTER-DNS>/openid/v1/jwks

Create a role binding to allow fetching the public keys and validating the JWT tokens.

apiVersion: rbac.authorization.k8s.io/v1                             kind: ClusterRoleBinding                             
metadata:                               
  name: oidc-reviewer                             
roleRef:                               
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole                               
  name: system:service-account-issuer-discovery                             subjects:                               
  - kind: Group                                 
    name: system:unauthenticated

Detail in bootstrap-controllers.sh file.

Verify that the cluster can be fetched jwks.

$ curl --cacert ca.crt https://<CLUSTER-DNS>/.well-known/openid-configuration | jq
{
	"issuer": "https://<CLUSTER-DNS>",
	"jwks_uri": "https://<CLUSTER-DNS>/openid/v1/jwks",
	"response_types_supported": ["id_token"],
	"subject_types_supported": ["public"],
	"id_token_signing_alg_values_supported": ["RS256"]
}$ curl --cacert ca.crt https://<CLUSTER-DNS>/openid/v1/jwks
{
	"keys": [
		{
			"use": "sig",
			"kty": "RSA",
			"kid": "Rt3TBA31bh3rH...",
			"alg": "RS256",
			"n": "vL0tjBqLDFTyqOC...",
			"e": "AQAB"
		}
	]
}

Create OIDC provider on AWS

Once the cluster is ready, I use the below Terraform code to retrieve the cluster certificate thumbprint and create an OIDC provider on AWS.

data "tls_certificate" "oidc" {
  url          = "https://<CLUSTER-DNS>"
  verify_chain = false
}resource "aws_iam_openid_connect_provider" "oidc" {
  url             = "https://<CLUSTER-DNS>"
  client_id_list  = ["<AUDIANCE-NAME>"]
  thumbprint_list = ["${data.tls_certificate.oidc.certificates.0.sha1_fingerprint}"]
}

And create a test IAM role with ViewOnlyAccess policy.

resource "aws_iam_role" "oidc" {
  name = "<TEST-ROLE-NAME>"assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRoleWithWebIdentity"
        Effect = "Allow"
        Sid    = ""
        Principal = {
          Federated = aws_iam_openid_connect_provider.oidc.arn
        }
        Condition = {
          StringEquals = {
            "<CLUSTER-DNS>:aud" : "<AUDIANCE-NAME>"
          }
        }
      },
    ]
  })managed_policy_arns = ["arn:aws:iam::aws:policy/job-function/ViewOnlyAccess"]
}

My Terraform file for creating the above resources.

Testing with test role

Create a config map to set the environment variables for AWS SDK.

apiVersion: v1
kind: ConfigMap
metadata:
  name: oidc-iam-pod-role
data:
  AWS_DEFAULT_REGION: ap-southeast-2
  AWS_ROLE_ARN: arn:aws:iam::<AWS-ACCOUNT-ID>:role/<TEST-ROLE-NAME>
  AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/oidc-iam/serviceaccount/toke

Create a service account for pod to use the config map.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: oidc-iam-pod

Create a test pod with AWS CLI docker image.

apiVersion: v1
kind: Pod
metadata:
  name: awscli
spec:
  containers:
    - image: amazon/aws-cli
      name: awscli
      command: ["sleep"]
      args: ["3600"]
      envFrom:
        - configMapRef:
            name: oidc-iam-pod-role
      volumeMounts:
        - mountPath: /var/run/secrets/oidc-iam/serviceaccount/
          name: aws-token
  serviceAccountName: oidc-iam-pod
  volumes:
    - name: aws-token
      projected:
        sources:
        - serviceAccountToken:
            path: token
            expirationSeconds: 600
            audience: <AUDIANCE-NAME>

Exec to pod and run STS caller identity command to see the authentication is worked.

$ kubectl exec awscli -- aws sts get-caller-identity
{
    "UserId": "AROAX:botocore-session",
    "Account": "<AWS-ACCOUNT-ID>",
    "Arn": "arn:aws:sts::<AWS-ACCOUNT-ID>:assumed-role/<TEST-ROLE-NAME>/botocore-session"
}

Security warning

Please note that anyone who knows the audiance name and IAM role ARN may lead to compromise your AWS IAM security token. It is highly recommend to tighten the IAM role’s trust relationships with OIDC provider and service account when creating the IAM role as below. Creating an IAM role and policy for your service account.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:aud": "<AUDIANCE-NAME>",
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:<NAMESPACE>:<SERVICE-ACCOUNT>"
        }
      }
    }
  ]
}