Service to Service Authentication on Kubernetes

Service to service authentication is hard. Or more accurately, service to service authentication done properly and securely is hard. And if you want to support human users with SSO at the same time, well that’s extremely hard.

Luckily, Kubernetes provides functionality to make the process easier: projected service account tokens. Taking some lessons from AWS’s IAM Roles for Service Accounts (IRSA) architecture, you can morph these projected tokens to be OpenID Connect (OIDC) ready. OIDC compatibility in place, you are set to use them adjacent to your human users with an OIDC-compatible authentication proxy like OAuth2-Proxy.

Why Bother?

This may seem daunting at first glance, but implementing this across your application platform yields huge benefits.

From your developers’ point of view:

• Auto-rotating, easily accessible ID Tokens
• OIDC-compatible — ready for internal authentication and federated external authentication (e.g. to AWS)
• No need to worry about private signing key management and storage
• Support for human users via SSO as well

And if you take the time to implement this throughout your application platform, all these features are available to your developers seamlessly. They can focus on business logic instead of trying to implement authentication themselves (…which will likely have bugs and security vulnerabilities).

The Final Architecture Goal

The Final Architecture Goal

Identity on Kubernetes

On Kubernetes, the Service Account resource is the way to provide an identity to workloads running in your Pods.

Clusters provide Pods access to their identity via JSON Web Tokens (JWTs). They use a service account key to sign a JWT that contains all the needed Service Account details in the JWT’s claims. Historically, this was an unchanging “forever” token with no expiration and no audience, and it was stored as a Kubernetes Secret. This design had limitations and security concerns that the advent of projected tokens in Kubernetes v1.12 alleviated.

Service Account Token Volume Projection

Service Account token volume projection gives you a way to overcome the limitations of the Secret-held “forever” tokens.

By using a projected volume, Kubernetes has the ability to provide dynamically rotating tokens to your Pods. This allows a proper expiration on your tokens. Kubernetes automatically handles creating a new token with a fresh expiration when your existing mounted token gets to 80% of its lifespan.

You aren’t limited to one projected volume either. You can mount multiple projected tokens — allowing you to specify the distinct audience (aud) claims you need for each token. Not only does the audience claim provide security benefits, it will be a key aspect of your OIDC conversion — acting as your OIDC client ID.

Here is a sample Pod spec showing the volume projection and mount:

Any container running in that Pod can get that token to use via HTTP bearer authentication by reading it from /var/run/secrets/tokens/sample-token.

Digging into that token’s contents and Base64 decoding the payload, you would see a payload like this:

Notice your Service Account name and its Namespace are part of your token’s subject (sub) claim. Receiving applications should use this to authorize the Service Account identity after authenticating the token.

We’ll touch on the audience (aud) and issuer (iss) claims later. They are important for the OIDC compatibility of these tokens.

Everything under the complex kubernetes.io claim isn’t needed for OIDC purposes. But if you want it, that claim provides nice additional details for the receiving application. You just have to code up how to extract it yourself.

Federated Identity with OIDC

OpenID Connect (OIDC) is an authentication protocol that helps verify a user’s (or machine’s) identity. It adds an identity layer on top of the OAuth2 protocol (hence we will use OAuth2-Proxy in the final stage of this guide). There’s a lot of nitty-gritty details to the specification, you can brush up on them here if you want.

For this design, the OIDC discovery documents are the key aspect of the OIDC protocol you will need — particularly the JSON Web Key Sets (JWKS). Those store the public keys needed to verify token signatures.

Inside an OIDC ID Token, there is an issuer (iss) claim in the payload that should have a publicly accessible URL. With OIDC, you can find the OpenID configuration at the /.well-known/openid-configuration path under it.

As a sample, you can peek at Greenhouse’s OneLogin OIDC identity provider here.

There’s a lot going on in that JSON, but amidst all that configuration noise the most important setting for this guide is the jwks_uri which contains your JWKS file. As mentioned before, this houses all your public keys needed to verify OIDC ID Token signatures.

In OIDC, your public keys being publicly accessible in a consistent manner based on the OIDC discovery documents is the key to identity federation. This allows external entities to verify the authenticity of your ID Tokens and use them for authorization.

Kubernetes Tokens and OIDC

AWS IRSA History

Here at Greenhouse, we got introduced to the potential of using Kubernetes projected tokens as OIDC ID Tokens by AWS’s IRSA design. There’s a LOT of details in that introduction, but if you have the mental stamina to read through it you’ll see a Pod spec that looks very similar to our examples above.

Not only did AWS socialize the architecture, they provided the source code for tools that help ease the deployment on your Kubernetes clusters. For instance, their EKS Pod Identity Webhook project shows how to project tokens as needed into any Pod with merely an annotation using a Mutating Admission Webhook.

Most importantly, they didn’t reserve the toolkit just for their EKS offering. AWS published how to use these techniques in a self-hosted Kubernetes environment — most critically the steps for how to turn projected tokens into valid OIDC ID Tokens.

OIDC for Kubernetes

As we touched on before, the key OIDC elements you need to make your projected tokens look like bare-bones OIDC ID Tokens is the openid-configuration discovery document and a JWKS file containing the public keys needed to verify a token’s signature.

Following AWS’s guide, you can create a minimal openid-configuration file for service to service authentication that looks like this:

The next more tricky part is generating a public JWKS from your service account keys found on your Kubernetes cluster’s kube-apiserver. AWS also published the Go source code for this — just pass in your public key and it will spit out the JWKS version of it you need that matches your private signing key.

Lastly, you need to host these public documents under the /.well-known path at your projected tokens’ issuer URL. When you set up projected tokens on your cluster, you enable them by setting these kube-apiserver options, so be sure to align with the -service-account-issuer flag.

Authenticating with OAuth2-Proxy

Now that all your Pods have an identity tied to their Service Account that they can pass around via OIDC ID Tokens, you need a way to authenticate those tokens at your receiving services.

Unless you want to manually code up support into your applications (which many libraries exist to do this if you choose), an authentication proxy is the best route to easily deliver this functionality to your developers. OAuth2-Proxy stands out in the open-source auth proxy space for its ability to handle our makeshift machine OIDC identities and human users simultaneously.

Machine Users Only

OAuth2-Proxy provides a bunch of potential OAuth2 providers to plug in. But if you only want to support machine users directly, the OIDC provider is the one you need.

Here are the minimum command-line flags you need to make the OIDC provider work with your Kubernetes projected tokens that you will be sending via bearer headers:

Some notes on these flags (since some of the settings are a little confusing…):

• oidc-issuer-url must match the issuer (iss) claim in your tokens. This is where you are hosting the public OIDC discovery documents you configured.
• client-id needs to match the audience (aud) claim in your tokens. You set the audience in the Pod spec in the projected volume details.
• cookie-secret and client-secret don’t matter for machine users. But they have to be set for OAuth2-Proxy to start up. Just set them with junk data.
• skip-jwt-bearer-tokens is what allows OAuth2-Proxy to verify ID Tokens in a bearer header directly. Otherwise it would look for a session cookie for authorization purposes.
• email-domains must be * for Kubernetes machine users support. If you glance above at the decoded contents of a projected token payload, you’ll notice there’s no email claim. Hence the * is mandatory.

With Human SSO Users Too!

If this is a service to service authentication guide, why should you care about supporting SSO as well?

Inevitably, you will have an application on your platform that wants to expose an API endpoint to services and human users alike. So you should plan ahead and architect a solution that is flexible enough to support both use cases.

OAuth2-Proxy allows this split paradigm. You can reserve the bearer header for service authentication while leaving traditional requests to be handled by your SSO provider for your human users.

If the skip-jwt-bearer-tokens flag is set enabling bearer token authentication, OAuth2-Proxy supports the extra-jwt-issuers flag as well.

extra-jwt-issuers lets you provide a list of issuer=audience pairs to attempt to verify any bearer ID Tokens that come in. The proxy will automatically look at the issuer URL for an OIDC /.well-known/openid-configuration or /.well-known/jwks.json. It will then use those public keys to validate the signature of any bearer tokens that come in (assuming their audience claim matches the audience you configured as well).

The important aspect of the extra-jwt-issuers flag is it works on bearer tokens adjacent to ANY provider you configure for your standard human users.

So your normal users without bearer headers can hit the proxy and get the behavior you expect from an SSO proxy. If they have a valid session cookie, they pass through no problem. If not, they get redirected to your configured OAuth2 identity provider to sign in. Upon success, they get a session cookie from OAuth2-Proxy and are golden to talk to your backend applications.

Putting it all Together

Between Kubernetes projected tokens, OIDC federation and OAuth2-Proxy as an authentication proxy sidecar, there’s a lot of complex ideas intermingling here.

If you decide to adopt all or part of this architecture, you’ll need to figure out the best way to deploy it in your environment. Luckily you have options:

• Mutating Admission Webhooks (taking motivations from IRSA’s EKS Pod Identity Webhook)
• Directly in your Kubernetes resource specs (with something like Helm or Kustomize)
• Built into your application deployment platform

You’ll also need to decide how to deploy OAuth2-Proxy if you want that layer as well. Adding a sidecar container directly in your Pods or adding Kubernetes Ingress annotations are both popular architectures.

Admittedly, none of this is trivial. But now that it’s implemented at Greenhouse, we’ve found the final product is worth the investment. So give it a whirl!