Manage Kubernetes authentication and authorization using Heptio Authenticator and Helm

It is not a question of if, but of when, followed by how.

Let me explain. As I see it, most Kubernetes cluster owners can be roughly divided into two groups — Operators and Administrators. They have a lot in common — they spin up the cluster, manage the cluster infrastructure and maybe even some of the middleware, and so on, but they grow apart when it comes to the application layer. Operators tend to manage that layer as well, while Administrators tend to push the responsibility back to the development teams. Take an applicative Helm chart for example — an Operator is the one writing the chart, while an Administrator, well, installs Tiller on the cluster. Get the idea? And the more services come into play, the more work (toil) it is for an Operator. So at some point you have to let go. It is not a question of if, it is a question of when.

Now let’s get to the how. “Letting go” in this context means to let others have access to the cluster, and once you pass responsibility — you have to provide visibility as well. The Kubernetes dashboard has only taken you so far, and people are rioting in the streets demanding kubectl, As they should. Can you imagine life without kubectl? So what do you do? Send over a copy of the kubeconfig file and hope for the best?

Before we continue, a word about authentication and authorization — it is like being under the legal drinking age. You can get in (authenticated), but you can’t have a drink (authorized).

Where were we? We want to let others roam within the cluster, but under two conditions — only authenticated users can execute commands using kubectl, and only authorized commands should get executed, and that depends on the user. So let’s get to it!

For the authentication part we will use the Heptio Authenticator, a great tool by Heptio, and for the authorization part we will use RBAC. Using RBAC, we can define a Role and a RoleBinding to enforce rules for specific users or groups of users.

Still reading?

That means you are interested in the Heptio Authenticator, which in turn means that you are using AWS to host your cluster. This (probably) means that you have setup the cluster using kops. This guide will take you step by step if this is true.

All the resources used in this post can be found under my cluster-auth repository in GitHub.

Step 1

The first thing to do is to update the cluster. Download the heptio-authenticator-aws binary and follow the Kops usage guide, but only perform steps 1–3, which are: generate certificates and kubeconfig and upload them to S3, update the cluster spec, apply the changes to the cluster. The remainder of the work is basically to create IAM users (consider groups for better understanding), and you are home free. A Terraform module is included in the repository.

Instructions:

1. Download the heptio autheticator and place it within $PATH:

wget https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/download/v0.3.0/heptio-authenticator-aws_0.3.0_linux_amd64 -O heptio-authenticator-aws
chmod +x heptio-authenticator-aws
sudo mv heptio-authenticator-aws /usr/local/bin/

2. Pre-generate the certificate, key, and kubeconfig and upload them to the kops state store:

heptio-authenticator-aws init -i $CLUSTER_NAME
aws s3 cp cert.pem ${KOPS_STATE_STORE}/${CLUSTER_NAME}/addons/authenticator/cert.pem;
aws s3 cp key.pem ${KOPS_STATE_STORE}/${CLUSTER_NAME}/addons/authenticator/key.pem;
aws s3 cp heptio-authenticator-aws.kubeconfig ${KOPS_STATE_STORE}/${CLUSTER_NAME}/addons/authenticator/kubeconfig.yaml;

3. Add the following sections to the cluster spec, either using kops edit cluster ${CLUSTER_NAME} or editing the manifest yaml file. Be sure to replace KOPS_STATE_STORE and CLUSTER_NAME with their appropriate values since those environment variables are not available at runtime. This downloads the files from the state store on masters to a directory that is volume mounted by kube-apiserver. Kops does not support adding additional volumes to kube-apiserver so we must reuse the existing /srv/kubernetes hostPath volume:

apiVersion: kops/v1alpha2
kind: Cluster
spec:
  kubeAPIServer:
    authenticationTokenWebhookConfigFile: /srv/kubernetes/heptio-authenticator-aws/kubeconfig.yaml
  hooks:
  - name: kops-hook-authenticator-config.service
    before:
      - kubelet.service
    roles: [Master]
    manifest: |
      [Unit]
      Description=Download AWS Authenticator configs from S3
      [Service]
      Type=oneshot
      ExecStart=/bin/mkdir -p /srv/kubernetes/heptio-authenticator-aws
      ExecStart=/usr/local/bin/aws s3 cp --recursive s3://KOPS_STATE_STORE/CLUSTER_NAME/addons/authenticator /srv/kubernetes/heptio-authenticator-aws/

If using a non-default AMI that does not have the AWS CLI, replace the second ExecStart statement with:

ExecStart=/usr/bin/docker run --net=host --rm -v /srv/kubernetes/heptio-authenticator-aws:/srv/kubernetes/heptio-authenticator-aws quay.io/coreos/awscli@sha256:7b893bfb22ac582587798b011024f40871cd7424b9026595fd99c2b69492791d aws s3 cp --recursive s3://KOPS_STATE_STORE/CLUSTER_NAME/addons/authenticator /srv/kubernetes/heptio-authenticator-aws/

4. Apply the changes with kops update cluster ${CLUSTER_NAME}. If the cluster already exists, roll the cluster with kops rolling-update cluster ${CLUSTER_NAME} in order to recreate the master nodes.

Step 2

The second thing to do is — deploy the cluster-auth Helm chart! To make things easier, this chart includes the most common groups needed — developers and administrators. All you have to do is change the user names and the account number and you are ready to deploy!

Some of the logic behind this chart is based on this blog post — ‘5 Things I Wish I’d Known Before Setting Up Heptio-Authenticator’.

Instructions:

git clone https://github.com/maorfr/cluster-auth.git
cd cluster-auth/helm/cluster-auth
# Edit users in values.yaml
helm install --name cluster-auth --namespace kube-system .

Step 3

I know, what about kubeconfig files for the users? It is fairly simple and you can probably create automation to do it for you. Follow this section of the Heptio Authenticator documentation to learn more.

Instructions:

1. Create an IAM user for each each user to be granted access to the cluster (these should correspond to the users in values.yaml).
2. Have the users configure their IAM credentials as the default profile.
3. Send over this kubeconfig file, it is the exact same one for all users:

apiVersion: v1
clusters:
- cluster:
    server: <api_server_url>
    certificate-authority-data: 
f1lly0urt0k3nH3R3Itw1llpr0b4blyb3l0ng3rth3nth1s0n3==
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: aws
  name: aws
current-context: aws
kind: Config
preferences: {}
users:
- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: heptio-authenticator-aws
      args:
        - token
        - -i
        - <your_cluster_name>

Summary

That’s it! You are done. You are now managing your Kubernetes authentication and authorization in one place. I bet you didn’t think it would be over so soon, did you?

Thanks for reading!