Amazon EKS Authentication with Google OpenID Connect (OIDC) via a proxy API service.

Here at WW, our cloud-first strategy has led the Platform Engineering team to maintain a multi-cloud platform environment, combining the best services that each chosen cloud provider has to offer into our global infrastructure.

Our applications are distributed among multiple cloud providers, and we have internally developed our own API services on top of this multi-cloud platform infrastructure. This provides a unique Cloud agnostic view to our development teams and partners when it comes to interacting with the whole system, such as when building and shipping codes, logging, monitoring, and authenticating.

Two key providers in our infrastructure are Amazon Web Services (AWS) and Google Cloud Platform (GCP). Our Kubernetes Clusters in Google (GKE) and in Amazon (KOPS) are all configured to use Google OpenID connect tokens for authentication. Authorization is performed using the normal Kubernetes RBAC method, but by using a different approach, we are running our own Kubernetes RBAC controller to interface the Kubernetes Cluster RBAC with the GCP Cloud IAM. This allows for a unique and reliable authorization method among all our Kubernetes clusters, either GKE- or KOPS-built.

Adding AWS Elastic Kubernetes Service (EKS) Clusters to our infrastructure required special care so that our existing users could take advantage of the new computing resources and capabilities that the EKS has to offer. Our existing users’ identity is provided by Google OIDC and EKS and, as it stands now, EKS cannot use any OIDC provider to authenticate requests to the Kubernetes API server. We had to design a solution to help our existing users seamlessly interact with our EKS clusters as we had done with our previous OIDC compatible Kubernetes clusters (GKE, KOPS).

The Problem

Amazon EKS uses a webhook token authentication method implemented by AWS IAM Authenticator to authenticate Kubernetes API server requests. This prevented us from using our existing authentication method based on Google OpenID Connect.

The Solution

With AWS IAM, we can create an IAM OpenID Connect (OIDC) identity provider (IdP) to establish trust between an OIDC-compatible IdP, such as Google OIDC, and our AWS account. This is done via AWS Web Identity Federation. This method needed to be secure, allowing a user to log in using Google IDP credentials. It also needed to grant the correct and necessary privileges to any authorized user on our AWS EKS Kubernetes Clusters.

To respond to our security concern, we chose to implement our own OIDC API service that exchanges a Google OIDC token for a secure AWS STS token, which is ready to be used by the EKS client side to generate an AWS IAM Authenticator webhook token. The OIDC API service sets the identify field in the generated AWS STS token that is being used to identify users of our EKS Kubernetes Clusters, behaving as an AWS STS proxy. The client makes a secure and authenticated REST call to the OIDC API service that returns an AWS STS token. It’s then ready to be used by the AWS IAM Authenticator client side to generate a valid Webhook token.

Putting It All Together

- Kubernetes authentication with Google OIDC:

There are many write-ups that explain how to configure Kubernetes to work with Google OIDC, and the world does not need yet another! You can easily find these on your favorite search engine, or in the Kubernetes docs.

- Setting up a trust policy on AWS for Login with Google OpenID Connect (OIDC):

Go to AWS IAM Management console and create an IAM role named “EKS-GOOGLE-AUTH” with the following trust relationship.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "prod",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ACCOUNT-ID-PROD:role/STS_PROXY"
      },
      "Action": "sts:AssumeRole"
    },    
    {
      "Sid": "nonprod",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ACCOUNT-ID-NONPROD:role/STS_PROXY"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

- Update EKS aws-auth configmap with this entry:

- rolearn: arn:aws:iam::REDACTED:role/EKS-GOOGLE-AUTH
   username: google-user:{{SessionName}}

- Install WW oidc-api service in your infrastructure:

The installation instructions are provided within the project’s repository aws-sts-proxy.

- Use a custom script to generate the EKS AWS IAM Authenticator Webhook token:

We also provide a tiny bash script to orchestrate the generation of the needed EKS AWS IAM Authenticator Webhook token using Google ID Token and the STS proxy eks-oidc-sproxy.

- Update client Kubeconfig:

apiVersion: v1
kind: Config
contexts:
- context:
 cluster: default
 user: eks-oidc-sproxy
  name: default
current-context: default
users:
- name: eks-oidc-sproxy
clusters:
- cluster:
 certificate-authority-data: REDACTED
 server: https://REDACTED.eks.amazonaws.com
  name: default
users:
- name: eks-oidc-sproxy
  user:
 exec:
   apiVersion: client.authentication.k8s.io/v1alpha1
   command: eks-oidc-sproxy
   args:
        - mycluster

- Use ‘kubectl’ to send requests to EKS Kubernetes API Server:

Display the Kubernetes version running on the client and server.

kubectl --kubeconfig mycluster.kubecfg version --short
Client Version: v1.13.4
Server Version: v1.13.7-eks-c57ff8

Print some Kubernetes node information:

kubectl --kubeconfig mycluster.kubecfg get nodes \
--output=custom-columns=NAME:.metadata.name,OS-IMAGE:.status.nodeInfo.osImage,KERNEL-VERSION:.status.nodeInfo.kernelVersion
NAME                          OS-IMAGE         KERNEL-VERSION
ip-**-**-**-**.ec2.internal   Amazon Linux 2   4.14.128-112.105.amzn2.x86_64
ip-**-**-**-**.ec2.internal   Amazon Linux 2   4.14.128-112.105.amzn2.x86_64
ip-**-**-**-**.ec2.internal   Amazon Linux 2   4.14.128-112.105.amzn2.x86_64
ip-**-**-**-**.ec2.internal   Amazon Linux 2   4.14.128-112.105.amzn2.x86_64

Other options for authenticating AWS EKS Cluster via OIDC:

I recently came across this project. However, I have not used it and cannot comment on it. The AWS STS proxy API service solution has been in use throughout all of our AWS EKS clusters for more than a year, bridging Google OIDC with AWS STS, and we have been very happy with it.

Summary

We had to design this solution for our AWS EKS-managed clusters to allow our existing Google OpenID Connect users to authenticate to those clusters as there was no solution available when we started this journey.

We are open to a better solution from the community. In the meantime, we hope this will help others who have the same needs.

Thanks for reading!

Roudy Veve, Principal Platform Engineer, WW

Interested in joining the WW team? Check out the careers page to view technology job listings as well as open positions on other teams.