Service Accounts and Auditing in Kubernetes

Jesse Antoszyk
Jun 21, 2017 · 9 min read

Service accounts and namespaces allow you to limit pod and user permission in Kubernetes. Audit logs provide insight into what accounts are accessing what resources. Learn how to use these Kubernetes features!

Prerequisites:

Overview

In this walkthrough, you create a namespace and add 2 service accounts, each with its own role, to it. One account can read secrets, Kubernetes objects that store sensitive information, and the other cannot. Each of these service accounts is attached to its own pod. Both of these pods run the same container image. At run time, secrets are read from within the container by using a curl command. The authorization to access the secret is determined by an API token that is mounted within the container. This token is generated by using the pod’s service account name. The API access is logged to a file, which contains the secret access by these service accounts.

Secrets give the ability to securely store data outside of a pod definition. Service accounts give the ability to grant and revoke access to resources from pods. Since both secrets and service accounts are scoped to a namespace, you can access them by using only accounts with appropriate permissions to the namespace. Additionally, audit logs allow traceability of accounts that access resources over the API.

Clone the GitHub Repo

Clone the Github project: https://github.com/IBM/k8s-service-accounts
The root directory of this project contains all the sample YAML files, the docker directory contains the Dockerfile used by the pod.

Each YAML file is named based on the resource it creates. For example the api-reader-cluster-roles.yaml file defines the cluster roles that you use in this project.

The api-reader-all-in-one.yaml file contains all the definitions in a single file. You can review the resources from a single location, however, in this guide you create each resource individually.

Start Minikube

For Kubernetes to honor the service accounts’ roles, you must enable Role-Based Access Control (RBAC) support in Minikube.

Because the audit log configuration options are different in Kubernetes 1.6 and 1.7, confirm which Kubernetes version you use. Run the following command:

In the output, you can see that the server version is 1.6.0.

If you use version 1.6.0, start Minikube by running this command:

If you use Kubernetes version 1.7, start Minikube by running this command (Not released at on Minikube the time of publication):

Create the Namespace

When you run Kubernetes as a multi-tenant or multi-project environment, you create namespaces to scope resources. You create items like pods, secrets, and service accounts within a namespace and can set resource quotas at namespace level. You can learn more about namespaces in the Kubernetes docs.

Create a new namespace. Use the following resource definition:

You can find the namespace definition in the api-reader-dev-namespace.yaml file.

To create the “dev” namespace, run this command:

To allow kubectl to run commands in the dev namespace, change the kubectl context. The following command obtains the current cluster name and updates the context that you use to run commands on it:

The current cluster name is “minikube.”

Create the Secret

A secret is an object that is used to store sensitive information, like passwords and authentication keys. In this example, a user name and password are stored for demonstration purposes. Secret data must be encoded in base64. You can learn more about secrets in the Kubernetes docs.

The secret is defined in the api-reader-secret.yaml file. Its contents follow:

To create the secret, run this command:

To verify that the secret was created, run this command:

Create the Service Accounts

You can attach service accounts to pods and use it to access the Kubernetes API. If a service account is not set in the pod definition, the pod uses the default service account for the namespace. Files that are named token, ca.crt, and namespace are automatically mounted in the /var/run/secrets/kubernetes.io/serviceaccount/ directory of each container. Their contents are based on the service account name that you provide. Kubernetes docs

Note: The secrets that are shown in the /var/run/secrets/kubernetes.io/serviceaccount/ directory are service account specific secrets that are mounted by the Kubernetes system, not the secret that you created. The access to this secret does not indicate that the pod can access other secrets with this token.

The service accounts are defined in api-reader-service-accounts.yaml file. Its contents follow:

To create these service accounts, run this command:

To confirm that the accounts exist, run this command:

Create the Cluster Roles

A cluster role defines a set of permissions that is used for accessing resources, such as pods and secrets. Cluster roles are scoped to the cluster. The cluster roles that are defined here are attached to the service accounts via a role binding in subsequent steps. Using a role binding instead of a cluster role binding scopes the permissions to a namespace. For more information about roles, see the Kubernetes docs.

The roles are defined in the api-reader-cluster-roles.yaml file. Its contents are shown:

To create the cluster roles, run this command:

To verify that the roles were created, run this command:

NAME              KIND
...
no-access-cr ClusterRole.v1beta1.rbac.authorization.k8s.io
secret-access-cr ClusterRole.v1beta1.rbac.authorization.k8s.io
system:basic-user ClusterRole.v1beta1.rbac.authorization.k8s.io
...

Create the Role Bindings

To apply roles to service accounts, create role bindings that connect them. When you bind a role to a service account, the permissions that you defined in a role are granted to the account. Kubernetes docs

The role bindings are defined in the api-reader-role-bindings.yaml file, as shown in the following text:

To create these role bindings, run this command:

To view the role bindings, run this command:

Review the Dockerfile

The Dockerfile for this example contains a single pod. When you run the Dockerfile, you install curl in the container and copy a runtime.sh script, which contains the commands that the container runs at startup, to it.

To build this Docker image locally, switch to the Docker directory and run this command:

The Dockerfile contains the following code:

The runtime.sh script contains the following code:

Create the Pods

You use service accounts to access the API from within a pod. In this model, your container accesses the secrets API at runtime instead of referencing secrets in the pod definition YAML. Secrets that are referenced in the YAML spec do not abide by the service account permissions.

Service account information is automatically mounted to the /var/run/secrets/kubernetes.io/serviceaccount/ directory and consists of the ca.crt, namespace, and token files.

The pods are defined in api-reader-pods.yaml. If you built the docker image locally in the Review the Dockerfile step, replace the ibmcloudprivate/k8s-service-accounts image parameter values in this YAML with the name of the image that you built.

To create these pods, run this command:

To verify that the role bindings were created, run the following command:

View the Pod Output

This step verifies that one of the pods can access the secrets API and the other cannot.

Verify that both of the pods that you created are in the running state by running this command:

Run this command to obtain the logs from the no-access pod:

The command output shows that access is denied.

Run this command to view the logs from the secret-access pod. The command output includes the secret that you created.

Try to Access Another Namespace

You can use Kubernetes to run other commands on containers in your pod. When you run the runtime.sh script in the container, you can pass a namespace parameter to it. If you don’t pass a namespace parameter to it, it uses the namespace of the service account.

Run the runtime.sh script and specify the default namespace, as shown in this command:

Because the service account is scoped to the dev namespace, it is not authorized to use the default namespace. Its access to the default namespace is denied.

View the Audit Logs

Audit logs allow administrators to view the particular resources that an account has accessed. The audit logs are stored on the system that hosts Minikube. You set the log storage location when you started Minikube in by using the --extra-configparameter.

Run this command to view the logs that contain the dev service account:

The log shows the URI of the accessed resource and the user who accessed it. For example, the last item in the log shows that the service account in the dev namespace that is named secret-access-sa (user=”system:serviceaccount:dev:secret-access-sa“) made a request for the secrets in the default namespace (uri=”/api/v1/namespaces/default/secrets“)

Conclusion

Service accounts are a powerful tool for cluster administration because you can use them to control and view access of resources in Kubernetes. You can use them to limit access to a particular namespace. You should limit pods to access only to what they need. The ability to know who accessed which resources and when they accessed them provides insight into cluster activity.

Originally published at developer.ibm.com.

IBM Cloud

Understand how to bring elastic runtimes to the Enterprise…

Jesse Antoszyk

Written by

DevOps Systems Engineer at BoxBoat Technologies. The opinions expressed here are my own.

IBM Cloud

IBM Cloud

Understand how to bring elastic runtimes to the Enterprise with effective security and data protection at scale.

Jesse Antoszyk

Written by

DevOps Systems Engineer at BoxBoat Technologies. The opinions expressed here are my own.

IBM Cloud

IBM Cloud

Understand how to bring elastic runtimes to the Enterprise with effective security and data protection at scale.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store