Service Accounts and Auditing in Kubernetes

Published in

IBM Cloud

9 min readJun 21, 2017

Service accounts and namespaces allow you to limit pod and user permission in Kubernetes. Audit logs provide insight into what accounts are accessing what resources. Learn how to use these Kubernetes features!

Prerequisites:

Overview

In this walkthrough, you create a namespace and add 2 service accounts, each with its own role, to it. One account can read secrets, Kubernetes objects that store sensitive information, and the other cannot. Each of these service accounts is attached to its own pod. Both of these pods run the same container image. At run time, secrets are read from within the container by using a curl command. The authorization to access the secret is determined by an API token that is mounted within the container. This token is generated by using the pod’s service account name. The API access is logged to a file, which contains the secret access by these service accounts.

Secrets give the ability to securely store data outside of a pod definition. Service accounts give the ability to grant and revoke access to resources from pods. Since both secrets and service accounts are scoped to a namespace, you can access them by using only accounts with appropriate permissions to the namespace. Additionally, audit logs allow traceability of accounts that access resources over the API.

Clone the GitHub Repo

Clone the Github project: https://github.com/IBM/k8s-service-accounts
The root directory of this project contains all the sample YAML files, the docker directory contains the Dockerfile used by the pod.

Each YAML file is named based on the resource it creates. For example the api-reader-cluster-roles.yaml file defines the cluster roles that you use in this project.

The api-reader-all-in-one.yaml file contains all the definitions in a single file. You can review the resources from a single location, however, in this guide you create each resource individually.

Start Minikube

For Kubernetes to honor the service accounts’ roles, you must enable Role-Based Access Control (RBAC) support in Minikube.

Because the audit log configuration options are different in Kubernetes 1.6 and 1.7, confirm which Kubernetes version you use. Run the following command:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.3", GitCommit:"029c3a408176b55c30846f0faedf56aae5992e9b", GitTreeState:"clean", BuildDate:"2017-02-15T06:40:50Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-05-09T23:22:45Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"}

In the output, you can see that the server version is 1.6.0.

If you use version 1.6.0, start Minikube by running this command:

$ minikube start --extra-config=apiserver.Authorization.Mode=RBAC --extra-config=apiserver.Audit.Path=/var/log/apiserver/audit.log
Starting local Kubernetes v1.6.0 cluster...
Starting VM...
SSH-ing files into VM...
Setting up certs...
Starting cluster components...
Connecting to cluster...
Setting up kubeconfig...
Kubectl is now configured to use the cluster.

If you use Kubernetes version 1.7, start Minikube by running this command (Not released at on Minikube the time of publication):

$ minikube start --extra-config=apiserver.Authorization.Mode=RBAC --extra-config=apiserver.Audit.LogOptions.Path=/var/log/apiserver/audit.log

Create the Namespace

When you run Kubernetes as a multi-tenant or multi-project environment, you create namespaces to scope resources. You create items like pods, secrets, and service accounts within a namespace and can set resource quotas at namespace level. You can learn more about namespaces in the Kubernetes docs.

Create a new namespace. Use the following resource definition:

apiVersion: v1
kind: Namespace
metadata:
 name: dev

You can find the namespace definition in the api-reader-dev-namespace.yaml file.

To create the “dev” namespace, run this command:

$ kubectl create -f api-reader-dev-namespace.yaml 
namespace “dev” created

To allow kubectl to run commands in the dev namespace, change the kubectl context. The following command obtains the current cluster name and updates the context that you use to run commands on it:

$ kubectl config set-context $(kubectl config current-context) --namespace=dev
Context “minikube” set.

The current cluster name is “minikube.”

Create the Secret

A secret is an object that is used to store sensitive information, like passwords and authentication keys. In this example, a user name and password are stored for demonstration purposes. Secret data must be encoded in base64. You can learn more about secrets in the Kubernetes docs.

The secret is defined in the api-reader-secret.yaml file. Its contents follow:

---
# Secret with base64 encoded values
apiVersion: v1
kind: Secret
metadata:
 name: api-access-secret
type: Opaque
data:
 username: YWRtaW4=
 password: cGFzc3dvcmQ=

To create the secret, run this command:

$ kubectl create -f api-reader-secret.yaml 
secret "api-access-secret" created

To verify that the secret was created, run this command:

$ kubectl get secrets api-access-secret
NAME                TYPE      DATA      AGE
api-access-secret   Opaque    2         8m

Create the Service Accounts

You can attach service accounts to pods and use it to access the Kubernetes API. If a service account is not set in the pod definition, the pod uses the default service account for the namespace. Files that are named token, ca.crt, and namespace are automatically mounted in the /var/run/secrets/kubernetes.io/serviceaccount/ directory of each container. Their contents are based on the service account name that you provide. Kubernetes docs

Note: The secrets that are shown in the /var/run/secrets/kubernetes.io/serviceaccount/ directory are service account specific secrets that are mounted by the Kubernetes system, not the secret that you created. The access to this secret does not indicate that the pod can access other secrets with this token.

The service accounts are defined in api-reader-service-accounts.yaml file. Its contents follow:

---
# Service account for preventing API access
apiVersion: v1
kind: ServiceAccount
metadata:
 name: no-access-sa
---
# Service account for accessing secrets API
apiVersion: v1
kind: ServiceAccount
metadata:
 name: secret-access-sa

To create these service accounts, run this command:

$ kubectl create -f api-reader-service-accounts.yaml 
serviceaccount "no-access-sa" created
serviceaccount "secret-access-sa" created

To confirm that the accounts exist, run this command:

$ kubectl get serviceaccounts
NAME               SECRETS   AGE
default            1         1d
no-access-sa       1         10m
secret-access-sa   1         10m

Create the Cluster Roles

A cluster role defines a set of permissions that is used for accessing resources, such as pods and secrets. Cluster roles are scoped to the cluster. The cluster roles that are defined here are attached to the service accounts via a role binding in subsequent steps. Using a role binding instead of a cluster role binding scopes the permissions to a namespace. For more information about roles, see the Kubernetes docs.

The roles are defined in the api-reader-cluster-roles.yaml file. Its contents are shown:

---
# A role with no access
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
 name: no-access-cr
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: [""] 
  verbs: [""]
---
# A role for reading/listing secrets
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
 name: secret-access-cr
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["secrets"] 
  verbs: ["get", "list"]

To create the cluster roles, run this command:

$ kubectl create -f api-reader-cluster-roles.yaml
clusterrole "no-access-cr" created
clusterrole "secret-access-cr" created

To verify that the roles were created, run this command:

$ kubectl  get clusterrolesNAME              KIND
...
no-access-cr      ClusterRole.v1beta1.rbac.authorization.k8s.io
secret-access-cr  ClusterRole.v1beta1.rbac.authorization.k8s.io
system:basic-user ClusterRole.v1beta1.rbac.authorization.k8s.io
...

Create the Role Bindings

To apply roles to service accounts, create role bindings that connect them. When you bind a role to a service account, the permissions that you defined in a role are granted to the account. Kubernetes docs

The role bindings are defined in the api-reader-role-bindings.yaml file, as shown in the following text:

---
# The role binding to combine the no-access service account and role
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
 name: no-access-rb
subjects:
- kind: ServiceAccount
 name: no-access-sa
roleRef:
 kind: ClusterRole
 name: no-access-cr
 apiGroup: rbac.authorization.k8s.io

---
# The role binding to combine the secret-access service account and role
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
 name: secret-access-rb
subjects:
- kind: ServiceAccount
 name: secret-access-sa
roleRef:
 kind: ClusterRole
 name: secret-access-cr
 apiGroup: rbac.authorization.k8s.io

To create these role bindings, run this command:

$ kubectl create -f api-reader-role-bindings.yaml 
rolebinding "no-access-rb" created
rolebinding "secret-access-rb" created

To view the role bindings, run this command:

$ kubectl get rolebindings
NAME               KIND
no-access-rb       RoleBinding.v1beta1.rbac.authorization.k8s.io
secret-access-rb   RoleBinding.v1beta1.rbac.authorization.k8s.io

Review the Dockerfile

The Dockerfile for this example contains a single pod. When you run the Dockerfile, you install curl in the container and copy a runtime.sh script, which contains the commands that the container runs at startup, to it.

To build this Docker image locally, switch to the Docker directory and run this command:

docker build -t <docker_image_tag> .

The Dockerfile contains the following code:

#ibmcloudprivate/k8s-service-accounts
FROM ubuntu

# Copy files
COPY runtime.sh /
# Modify file permissions
RUN chmod +x runtime.sh


# Install curl
RUN apt-get update
RUN apt-get install curl -y

# Run script on startup
CMD [ "/runtime.sh" ]

The runtime.sh script contains the following code:

#!/bin/bash

# Read mounted files
KUBE_TOKEN=$(</var/run/secrets/kubernetes.io/serviceaccount/token)
NAMESPACE=$(</var/run/secrets/kubernetes.io/serviceaccount/namespace)

# Resource to get in API [pods/secrets]RESOURCE="secrets"

# If an argument was set
if [ "$#" -ge 1 ]; then
 NAMESPACE="$1"
fi

#Curl against the resource
echo curl -sSk -H "Authorization: Bearer $KUBE_TOKEN" https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_PORT_443_TCP_PORT/api/v1/namespaces/$NAMESPACE/$RESOURCE
curl -sSk -H "Authorization: Bearer $KUBE_TOKEN" https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_PORT_443_TCP_PORT/api/v1/namespaces/$NAMESPACE/$RESOURCE
echo ""

# Don't wait on user exec
if [ "$#" -lt 1 ]; then
 # Sleep forever so the pod doesn't crash
 while [ true ]; do
 sleep 10
 done
fi

Create the Pods

You use service accounts to access the API from within a pod. In this model, your container accesses the secrets API at runtime instead of referencing secrets in the pod definition YAML. Secrets that are referenced in the YAML spec do not abide by the service account permissions.

Service account information is automatically mounted to the /var/run/secrets/kubernetes.io/serviceaccount/ directory and consists of the ca.crt, namespace, and token files.

The pods are defined in api-reader-pods.yaml. If you built the docker image locally in the Review the Dockerfile step, replace the ibmcloudprivate/k8s-service-accounts image parameter values in this YAML with the name of the image that you built.

---
# Create a pod with the no-access service account
kind: Pod
apiVersion: v1
metadata:
 name: no-access-pod
spec:
 serviceAccountName: no-access-sa
 containers:
 - name: no-access-container
   image: ibmcloudprivate/k8s-service-accounts

---
# Create a pod with the secret-access service account
kind: Pod
apiVersion: v1
metadata:
 name: secret-access-pod
spec:
 serviceAccountName: secret-access-sa
 containers:
 - name: secret-access-container
   image: ibmcloudprivate/k8s-service-accounts

To create these pods, run this command:

$ kubectl create -f api-reader-pods.yaml
rolebinding "no-access-rb" created
rolebinding "secret-access-rb" created

To verify that the role bindings were created, run the following command:

$ kubectl get pods
NAME                READY     STATUS    RESTARTS   AGE
no-access-pod       1/1       Running   0          16m
secret-access-pod   1/1       Running   0          16m

View the Pod Output

This step verifies that one of the pods can access the secrets API and the other cannot.

Verify that both of the pods that you created are in the running state by running this command:

$ kubectl get pods
NAME                READY     STATUS    RESTARTS   AGE
no-access-pod       1/1       Running   0          16m
secret-access-pod   1/1       Running   0          16m

Run this command to obtain the logs from the no-access pod:

$ kubectl logs no-access-pod
curl -sSk -H Authorization: Bearer <token> https://10.0.0.1:443/api/v1/namespaces/dev/secrets
User "system:serviceaccount:dev:no-access-sa" cannot list secrets in the namespace "dev".

The command output shows that access is denied.

Run this command to view the logs from the secret-access pod. The command output includes the secret that you created.

$ kubectl logs secret-access-pod
curl -sSk -H Authorization: Bearer <token> https://10.0.0.1:443/api/v1/namespaces/dev/secrets
{
 "kind": "SecretList",
 "apiVersion": "v1",
 "metadata": {
 "selfLink": "/api/v1/namespaces/dev/secrets",
 "resourceVersion": "2029"
 },
 "items": [
 {
 "metadata": {
 "name": "api-access-secret",
 "namespace": "dev",
 "selfLink": "/api/v1/namespaces/dev/secrets/api-access-secret",
 "uid": "4e965c40-550f-11e7-9f14-080027ab725e",
 "resourceVersion": "1917",
 "creationTimestamp": "2017-06-19T16:49:48Z"
 },
 "data": {
 "password": "cGFzc3dvcmQ=",
 "username": "YWRtaW4="
 },
 "type": "Opaque"
 },
 ...
}

Try to Access Another Namespace

You can use Kubernetes to run other commands on containers in your pod. When you run the runtime.sh script in the container, you can pass a namespace parameter to it. If you don’t pass a namespace parameter to it, it uses the namespace of the service account.

Run the runtime.sh script and specify the default namespace, as shown in this command:

$ kubectl exec -it secret-access-pod /runtime.sh default
curl -sSk -H Authorization: Bearer <token> https://10.0.0.1:443/api/v1/namespaces/default/secrets
User "system:serviceaccount:dev:secret-access-sa" cannot list secrets in the namespace "default".

Because the service account is scoped to the dev namespace, it is not authorized to use the default namespace. Its access to the default namespace is denied.

View the Audit Logs

Audit logs allow administrators to view the particular resources that an account has accessed. The audit logs are stored on the system that hosts Minikube. You set the log storage location when you started Minikube in by using the --extra-configparameter.

Run this command to view the logs that contain the dev service account:

$ minikube ssh sudo cat /var/log/apiserver/audit.log | grep "system:serviceaccounts:dev"
2017-06-20T11:29:23.510487235Z AUDIT: id="7a09d503-cf8c-48f8-8b4a-d92f775d6dc5" ip="172.17.0.2" method="GET" user="system:serviceaccount:dev:no-access-sa" groups="\"system:serviceaccounts\",\"system:serviceaccounts:dev\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="dev" uri="/api/v1/namespaces/dev/secrets"
2017-06-20T11:29:24.550061517Z AUDIT: id="f1ac0d89-a646-4fd4-aeb6-d90be0c2c900" ip="172.17.0.3" method="GET" user="system:serviceaccount:dev:secret-access-sa" groups="\"system:serviceaccounts\",\"system:serviceaccounts:dev\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="dev" uri="/api/v1/namespaces/dev/secrets"
2017-06-20T11:39:22.314592214Z AUDIT: id="3d5253ba-b1a8-4da4-baf2-7f81b85c5905" ip="172.17.0.2" method="GET" user="system:serviceaccount:dev:secret-access-sa" groups="\"system:serviceaccounts\",\"system:serviceaccounts:dev\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="default" uri="/api/v1/namespaces/default/secrets"

The log shows the URI of the accessed resource and the user who accessed it. For example, the last item in the log shows that the service account in the dev namespace that is named secret-access-sa (user=”system:serviceaccount:dev:secret-access-sa“) made a request for the secrets in the default namespace (uri=”/api/v1/namespaces/default/secrets“)

Conclusion

Service accounts are a powerful tool for cluster administration because you can use them to control and view access of resources in Kubernetes. You can use them to limit access to a particular namespace. You should limit pods to access only to what they need. The ability to know who accessed which resources and when they accessed them provides insight into cluster activity.

Originally published at developer.ibm.com.