Granting IAM permissions to pods in EKS using OIDC

Say you're using AWS’s managed Kubernetes platform(EKS) and want to deploy an application that requires access to AWS resources like an S3 bucket or a Kinesis stream. What are the options to enable this?

One thing is certain, you’d need an IAM role with one or more policies that grant the required permissions. But then, how‘d you link this role with your application?

There are three ways you could do this:

• AWS Access/Secret keys— This is generally a bad idea both from a security and maintenance perspective.
• Custom solutions like kube2iam, kiam, etc. — These leverage the EC2 instance metadata endpoint for generating temporary AWS credentials. It works but it comes with a maintenance overhead.
• AWS OpenID Connect(OIDC) — Oauth2 and OpenID for a zero-maintainance JWT token based authentication.

OIDC is definitely the way to go and with EKS, it almost works like magic. In this post, we’ll take a look at how we can leverage it and also peek under the hood at how EKS is able to offer this.

What’s OIDC?

OIDC — which stands for OpenID Connect is a secure communications protocol that uses OpenID for authentication, Oauth2 for authorization, and JWT tokens for sharing credentials.

The exact details on how OIDC works are out of scope for this post. But if you are curious, I highly recommended reading: Illustrated Guide to Oauth and OIDC.

Now as for how it’s implemented in AWS SDKs(eg: boto3), if you are using a version that’s newer than what’s listed here(you most likely are!), you are able to use OIDC without any code changes!

How do I use this?

1. Enable the IAM OIDC provider in EKS

This is a one-time effort. Follow this guide to set it up.

2. Create the IAM role for the application

We’d need to create an IAM role for each application with the necessary IAM policies with permissions to access AWS resources. Follow this guide for more details.

3. Add service account to trusted entities

Allow the service account(we’ll create this in the next step) to assume the IAM role using a trusted policy. Substitute the ARN for the OIDC provider from step 1, the service account name, and namespace in the following template:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/OIDC_PROVIDER"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringLike": {
          "OIDC_PROVIDER:sub": "system:serviceaccount:<service_account_namespace>:<service_account_name>"
        }
      }
    }
  ]
}

4. Create a service account and add the required annotation

Create a Kubernetes service account and set the full ARN of the role created in step 2 under an eks.amazonaws.com/role-arn annotation.

apiVersion: v1
kind: ServiceAccount
metadata:
    ...
    annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/IAM_ROLE_NAME

5. Deploy and it just works!

The Pods are now able to access AWS services without using a secret key or by querying a protected EC2 metadata endpoint!

How does it work?

So what’s happening behind the scenes in EKS to make this work?

The credit goes to eks-pod-identity-webhook, which is deployed as a type of an admission controller called MutatingAdmissionWebhook, a feature in Kubernetes that lets you modify(mutate) objects as they are created/updated.

The role of eks-pod-identity-webhook is as follows:

Inject environment variables

Variables necessary for the AWS SDKs to detect and use OIDC authentication:

• AWS_ROLE_ARN — The same ARN that was added to the annotation.
• AWS_WEB_IDENTITY_TOKEN_FILE — A path to a file containing a secret JWT ID token generated by AWS.

Configure token mount

How is the file with the token created and mounted in the path in AWS_WEB_IDENTITY_TOKEN_FILE? It’s made possible using a Kubernetes feature called ServiceAccountVolumeProjection .

A service account token is created and refreshed periodically, subsequently updating the volume. AWS SDKs are configured to listen for changes and pick up the new token.

To put it all together, on inspecting the Pod spec, you should see the following:

apiVersion: v1
kind: Pod
metadata:
  ...
spec:
   serviceAccountName: <service_account_name>
  ...
### Everything below is added by the webhook ###
    env:
    - name: AWS_DEFAULT_REGION
      value: us-west-2
    - name: AWS_REGION
      value: us-west-2
    - name: AWS_ROLE_ARN
      value: "arn:aws:iam::ACCOUNT_ID:role/IAM_ROLE_NAME"
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
    - name: AWS_STS_REGIONAL_ENDPOINTS
      value: "regional"
    volumeMounts:
    - mountPath: "/var/run/secrets/eks.amazonaws.com/serviceaccount/"
      name: aws-token
  volumes:
  - name: aws-token
    projected:
      sources:
      - serviceAccountToken:
          audience: "sts.amazonaws.com"
          expirationSeconds: 86400
          path: token

Hope you find this to be useful. Thanks for reading!