Kubernetes ServiceAccounts for use in automated systems

I get so many questions from our users at IBM Cloud Kubernetes Service regarding “non-human” users that I figured it was time to drop a little knowledge from experience: please use Kubernetes ServiceAccounts! Most of the docs you will find refer to their use for access from Pods. However, I’ve yet to hear any compelling reason why they are not a perfect fit for this use case. If you DO know of a reason, please let me know. Service Accounts meet a number of key requirements:

Some OIDC providers may issue non-expiring tokens so that your tools don’t need to keep going back for new tokens, but I don’t know of any; ours certainly does not. So you could use an IAM ServiceID, however, these are not a great solution for automated tooling since it would require the tooling to have the IBM Cloud CLI tools installed in order to get refreshed KUBECONFIGs to use.

So that leaves us with Kubernetes ServiceAccounts: they are easy to create, you can do some basic grouping of them, and they play by all the normal RBAC rules. It’s even easy to revoke their tokens.

• Provides no access outside the cluster the account is created in
• The JWT token never expires
• The JWT tokens can be instantly revoked
• Provides RBAC support for scoping access to minimal access rights
• Not associated with any human individual
• Audited like all other Kubernetes API access

Let’s look at an example where we are going to have Travis and Jenkins hooked up to our kube cluster to build and deploy to the cluster. I like to keep these automated system ServiceAccounts together in a serviceid namespace. Let’s start by creating a namespace and the ServiceAccounts…

jakekit at kitchbook in ~
$ kubectl create ns serviceids
namespace “serviceids” created
jakekit at kitchbook in ~
$ kubectl -n serviceids create serviceaccount travis
serviceaccount "travis" created
jakekit at kitchbook in ~
$ kubectl -n serviceids create serviceaccount jenkins
serviceaccount "jenkins" created

Fantastic. We’ve got a home for our service ids and a couple of accounts created. Now lets get some credentials and try them out…

jakekit at kitchbook in ~
$ JENKINS_KUBE_TOKEN=`kubectl -n serviceids get secret $(kubectl -n serviceids get secret | grep jenkins | awk '{print $1}') -o json | jq -r '.data.token'  | base64 -D`
jakekit at kitchbook in ~
$ TRAVIS_KUBE_TOKEN=`kubectl -n serviceids get secret $(kubectl -n serviceids get secret | grep travis | awk '{print $1}') -o json | jq -r '.data.token'  | base64 -D`
jakekit at kitchbook in ~
$ kubectl --server=https://169.61.83.62:31151 --token=$TRAVIS_KUBE_TOKEN get deployments -n squad-a-apps
Error from server (Forbidden): deployments.extensions is forbidden: User "system:serviceaccount:serviceids:travis" cannot list deployments.extensions in the namespace "squad-a-apps"
jakekit at kitchbook in ~
$ kubectl --server=https://169.61.83.62:31151 --token=$JENKINS_KUBE_TOKEN get deployments -n squad-b-apps
Error from server (Forbidden): deployments.extensions is forbidden: User "system:serviceaccount:serviceids:jenkins" cannot list deployments.extensions in the namespace "squad-b-apps"

First, take note of my little trick to get the jwt token out of the secret for the service account. Second, we can authenticate, but we don’t have any RBAC policy to grant us access to resources yet. Much has been written on RBAC, here are a couple of links from the community.

• https://kubernetes.io/docs/admin/authorization/rbac/
• https://kubernetes.io/blog/2017/10/using-rbac-generally-available-18/

Since we are here, let’s write some of our own that might be useful for Jenkins and Travis. Travis is only being used by squad-a. Travis only needs to be able to manage deployments and replicasets in the squad-a-apps namespace. Jenkins performs similar responsibilities for squad-b. Let’s see what that RBAC looks like

cicd-clusterroles.yaml
----------------------
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: cicd-apps
rules:
- apiGroups:
  - apps
  - extensions
  resources:
  - deployments
  - replicasets
  verbs:
  - create
  - delete
  - deletecollection
  - get
  - list
  - patch
  - update
  - watch
jakekit at kitchbook in ~
$ kubectl apply -f cicd-clusterroles.yaml
clusterrole.rbac.authorization.k8s.io "cicd-apps" created

We’ll also need bindings in place:

cicd-rolebindings.yaml
----------------------
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: travis-apps
  namespace: squad-a-apps
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cicd-apps
subjects:
- kind: User
  name: system:serviceaccount:serviceids:travis
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jenkins-apps
  namespace: squad-b-apps
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cicd-apps
subjects:
- kind: User
  name: system:serviceaccount:serviceids:jenkins
  apiGroup: rbac.authorization.k8s.io
jakekit at kitchbook in ~
$ kubectl apply -f cicd-rolebindings.yaml
rolebinding.rbac.authorization.k8s.io "travis-apps" created
rolebinding.rbac.authorization.k8s.io "jenkins-apps" created

We now have our ServiceAccounts, ClusterRoles, and RoleBindings all ready to go! Let’s put them to work:

jakekit at kitchbook in ~
$ kubectl --server=https://169.61.83.62:31151 --token=$TRAVIS_KUBE_TOKEN get deployments -n squad-a-apps
NAME           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
data-tier      3         3         3            3           25m
mobile-apis    3         3         3            3           26m
web-frontend   3         3         3            3           25m
jakekit at kitchbook in ~
$ kubectl --server=https://169.61.83.62:31151 --token=$JENKINS_KUBE_TOKEN get deployments -n squad-b-apps
NAME             DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
app-metrics      3         3         3            3           24s
user-analytics   3         3         3            3           17s

That should do it. We have two service accounts for our CICD tools along with appropriately scoped RBAC policy to go with them. It’s very simple to revoke access for these tokens as well in a situation where we feel it may have been compromised.

jakekit at kitchbook in ~
$ kubectl -n serviceids delete secret jenkins-token-tzxmv travis-token-h9vpq
secret "jenkins-token-tzxmv" deleted
secret "travis-token-h9vpq" deleted
jakekit at kitchbook in ~
$ kubectl --server=https://169.61.83.62:31151 --token=$JENKINS_KUBE_TOKEN get deployments -n squad-b-apps
error: You must be logged in to the server (Unauthorized)
jakekit at kitchbook in ~
$ kubectl --server=https://169.61.83.62:31151 --token=$TRAVIS_KUBE_TOKEN get deployments -n squad-a-apps
error: You must be logged in to the server (Unauthorized)

Happy automating and continuous integration/continuous delivery!