Auth’ing w/ Kubernetes Engine service accounts

Caveat Developer

You can authenticate against Kubernetes Engine resources using Kubernetes (!) service accounts. I was unable to find examples of this approach in Kubernetes (Engine) documentation so — please — treat with discretion and, if you’ve any security concerns about this approach, don’t employ it.

Our ambition is to use a Kubernetes Engine service account to authenticate and authorize some software process, e.g. a CI|CD pipeline.

NB When I initially looked into this problem, I assumed the service account would need to be a Google Cloud Platform service account not a Kubernetes Engine service account.

Cluster

I assume you’ve a (sacrifical) cluster (called cluster) that you are willing to use to test this and a context called root.

kubectl config get-clusters
NAME
cluster
kubectl config get-contexts
CURRENT   NAME   CLUSTER   AUTHINFO    NAMESPACE
          root   cluster   [[YOU]]

Context

To ensure we’re explicit about which context we’re using, let’s unset the default context; this will require us to explicitly reference our desired context for each kubectl command:

kubectl config unset current-context
kubectl get nodes
The connection to the server localhost:8080 was refused - did you specify the right host or port?
kubectl get nodes --context=root
NAME                                        STATUS   ROLES    AGE   VERSION
gke-cluster-01-default-pool-b0fa792d-lt2v   Ready    <none>   80m
gke-cluster-01-default-pool-b2ff416e-vtbt   Ready    <none>   80m
gke-cluster-01-default-pool-b457df6d-6n2s   Ready    <none>   80m

Namespace

To scope our work, let’s create and use a namespace for this:

NAMESPACE=test
kubectl create namespace ${NAMESPACE} --context=root
namespace/test created

Service Account

Let’s create a Kubernetes Engine service account. For simplicity, we’ll create the service account in ${NAMESPACE} but it need not be created in nor is limited to this namespace:

NAME=test
echo "
apiVersion: v1
kind: ServiceAccount
metadata:
 name: ${NAME}
 namespace: ${NAMESPACE}
" | kubectl apply --filename=- --context=root
serviceaccount/${NAME} created

NB My apologies if you dislike my overloading the name test. Because each resource is distinct, it isn’t a problem. Use your preferred values.

Authentication

Kubernetes Engine leverages Google [Cloud Platform] [OAuth2] authentication. If you inspect your Kubernetes configuration file, you’ll see that your credentials are obtained using gcloud config config-helper:

users:
- name: [[YOUR-GOOGLE-ID]]
  user:
    auth-provider:
      config:
        access-token: ya29.[[REDACTED]]
        cmd-args: config config-helper --format=json
        cmd-path: /usr/lib/google-cloud-sdk/bin/gcloud
        expiry: "2019-03-07T23:59:59Z"
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp

If you run this command, you’ll see the access_token obtained by the gcloud command and used by kubectl to authenticate you to the cluster.

One side-effect of the approach outlined in this post is that it doesn’t require gcloud to be available for the software to repeatedly auth against the cluster.

We’re going to create a new user (called ${NAME}) to represent the service account:

SECRET=$(kubectl get serviceaccount/${NAME} \
--namespace=${NAMESPACE} \
--context=root \
--output=jsonpath="{.secrets[0].name}")
TOKEN=$(kubectl get secret/${SECRET} \
--namespace=${NAMESPACE} \
--context=root \
--output=jsonpath="{.data.token}" | base64 --decode)
kubectl config set-credentials ${NAME} \
--token=${TOKEN}
kubectl config set-context ${NAME} \
--user=${NAME} \
--cluster=cluster

NB The set-credentials command is adding a bearer token to your Kubernetes configuration file. It is imperative that you control access to this token and the file. If you lose control of either, revoke the Secret (!) using kubectl delete secret/${SECRET} --namespace=${NAMESPACE} --context=root.

NB In the last command, we’re going to create a context that’s named after ${NAME}. You may prefer to name the context distinctly from the name of the user.

NB If you kubectl describe secret/${SECRET} ... you get given the base64 decoded value. But kubectl get secret/${SECRET} ... returns the base64 encoded value. We want the decoded value.

Now, if you review the ~/.kube/config file, you should see the addition of a user entry for ${NAME} and a context entry:

contexts:
- context:
    cluster: cluster
    user: ${NAME}
  name: ${NAME}
- context:
    cluster: cluster
    user: [[YOU]]
  name: root
current-context: ""
kind: Config
preferences: {}
users:
- name: [[YOU]]
  user:
    auth-provider:
      config:
        access-token: ya29.[[REDACTED]]
        cmd-args: config config-helper --format=json
        cmd-path: /usr/lib/google-cloud-sdk/bin/gcloud
        expiry: "2019-03-07T23:59:59Z"
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp
- name: ${NAME}
  user:
    token: ${TOKEN}

But:

kubectl get nodes --context=${NAME}
Error from server (Forbidden): nodes is forbidden: User "system:serviceaccount:${NAMESPACE}:${NAME}" cannot list resource "nodes" in API group "" at the cluster scope

Authenticated but not authorized.

Authorization

This account has no permissions on the cluster.

We’re going to grant ${NAME} the ability to use deployments. To do so we’ll need to a create a Role (I’m using Role rather than ClusterRole for specificity) and then a RoleBinding between ${NAME} and the Role.

echo "
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ${NAME}
  namespace: ${NAMESPACE}
rules:
- apiGroups:
  - apps
  - extensions
  resources:
  - deployments
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
...
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ${NAME}
  namespace: ${NAMESPACE}
subjects:
- kind: ServiceAccount
  name: ${NAME}
  namespace: ${NAMESPACE}
roleRef:
  kind: Role
  name: ${NAME}
  apiGroup: rbac.authorization.k8s.io
---
" | kubectl apply --filename=- --context=root
role.rbac.authorization.k8s.io/${NAME} created
rolebinding.rbac.authorization.k8s.io/${NAME} created

Test

Let’s create test deployments in namespace default and ${NAMESPACE}:

kubectl run nginx \
--image=nginx \
--replicas=1 \
--namespace=default \
--context=root
kubectl run nginx \
--image=nginx \
--replicas=1 \
--namespace=${NAMESPACE} \
--context=root

Our root context has full authorization but our ${NAME} context (representing our ${NAME} user) should only be able to enumerate deployments in ${NAMESPACE}:

kubectl get deployments \
--context=${NAME} \
--namespace=default
Error from server (Forbidden): deployments.extensions is forbidden: User "system:serviceaccount:${NAMESPACE}:${NAME}" cannot list resource "deployments" in API group "extensions" in the namespace "default"
kubectl get deployments \
--context=${NAME} \
--namespace=${NAMESPACE}
NAME    DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx   1         1         1            1           25s

NB kubectl run creates Deployments using extensions rather than the new apps API Group.

Conclusion

Kubernetes Engine auth powered by Google OAuth is excellent and generally the way to go.

As has been shown here, if you have software (rather than a human) that’s external to a cluster and needs to interact with the cluster, you may use Kubernetes Service Accounts to authenticate and grant the account precise permissions (authorization).

Going directly cuts out the need to include gcloud in the external software’s configuration. Although — please remember — that the token that’s added to the ~/.kube/config file is a bearer token. Whatever has the token can access the cluster with the service account’s permissions. If you lose control of the token or a Kubernetes config file that references it, delete the secret:

kubectl delete secret/${SECRET} \
--namespace=${NAMESPACE} \
--context=root

This will then disable use of the token:

kubectl delete secret/${SECRET} \
--namespace=${NAMESPACE} \
--context=root
secret "fred-token-nbqw9" deleted
kubectl get deployment/nginx \
--namespace=${NAMESPACE} \
--context=${NAME}
NAME    DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx   1         1         1            1           59m
kubectl get deployment/nginx \
--namespace=${NAMESPACE} \
--context=${NAME}
error: You must be logged in to the server (Unauthorized)

The result is not instantaneous but it should be prompt.

Tidy up!

A sledge-hammer approach to tidying-up is:

kubectl delete namespace/${NAMESPACE} --context=root

Because the Service Account, Role , RoleBinding and one Deployment were created in this namespace, they will be deleted with it.

To delete the other Deployment:

kubectl delete deployment/nginx --namespace=default --context=root

To remove the configuration file changes:

kubectl config unset users.${NAME}
kubectl config unset contexts.${NAME}

That’s all!