Kubernetes Authentication, Authorization & Audit. Part 4

Dmytro Nasyrov
Pharos Production
Published in
4 min read3 days ago

Event audit

The Kubernetes architecture can be schematically represented as follows

The key component of Kubernetes that is responsible for processing requests is the api-server. All operations on the cluster go through it. System audit is an interesting feature in Kubernetes, which is disabled by default. It allows all requests to be logged into the Kubernetes API. As you can easily guess, all actions related to monitoring and changing the state of the cluster are performed through this API.

So, to enable auditing, we need to pass three mandatory parameters to the container in api-server, which are described in more detail below.

--audit-policy-file=/etc/kubernetes/policies/audit-policy.yaml
--audit-log-path=/var/log/kube-audit/audit.log
--audit-log-format=json

In addition to these three required parameters, there are many additional settings related to auditing: from log rotation to webhook descriptions. Example of log rotation parameters

- audit-log-maxbackup=10 
- audit-log-maxsize=100
- audit-log-maxage=7

As already mentioned, all parameters are set in the manifest with the api-server configuration (by default /etc/kubernetes/manifests/kube-apiserver.yaml), in the command section. Let’s go back to the 3 required parameters and analyze them

audit-policy-file — path to the YAML file with the audit policy description. We will return to its contents later, but for now I will note that the file must be accessible for reading by the api-server process. Therefore, it is necessary to mount it inside the container, for which you can add the following code to the corresponding config sections

volumeMounts:
- mountPath: /etc/kubernetes/policies
name: policies
readOnly: true
volumes:
- hostPath:
path: /etc/kubernetes/policies
type: DirectoryOrCreate
name: policies

audit-log-path — path to the log file. The path must also be accessible to the api-server process, so we describe its mounting in the same way

 volumeMounts:
- mountPath: /var/log/kube-audit
name: logs
readOnly: false
volumes:
- hostPath:
path: /var/log/kube-audit
type: DirectoryOrCreate
name: logs

audit-log-format — audit log format. By default it is JSON, but legacy text format is also available

Audit policy

Now about the mentioned file with the description of the logging policy. The first concept of the audit policy is level, the logging level. They are as follows:

  • None — do not log;
  • Metadata — log request metadata: user, request time, target resource (pod, namespace, etc.), action type (verb), etc.;
  • Request — log metadata and request body;
  • RequestResponse — log metadata, request body, and response body.

The last two levels (Request and RequestResponse) do not log requests that do not access resources (requests to so-called non-resources URLs).

Also, all requests go through several stages:

  • RequestReceived — the stage when the request is received by the handler and has not yet been passed further along the handler chain;
  • ResponseStarted — the response headers have been sent, but before the response body is sent. Generated for long-running requests (e.g., watch);
  • ResponseComplete — the response body has been sent, no more information will be sent;
  • Panic — events are generated when an abnormal situation is detected.

To skip some stages, you can use omitStages.

In the policy file, we can describe several sections with different logging levels. The first matching rule found in the policy description will be applied. The kubelet daemon monitors the manifest with the api-server configuration and, if any are detected, restarts the container with the api-server. But there is an important detail: changes in the policy file will be ignored. After making changes to the policy file, you will need to restart the api-server manually. Since the api-server is running as a static pod, the kubectl delete command will not restart it.

When enabling auditing, it is important to remember that the load on the api-server increases. In particular, the memory consumption for storing the request context increases. Logging starts only after the response header is sent. The load also depends on the audit policy configuration.

Examples of policies

Let’s look at the structure of policy files using examples. Here is a simple policy file to log everything at the Metadata level

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata

In policy, you can specify a list of users (Users and ServiceAccounts) and user groups. For example, here’s how we’ll ignore system users, but log everything else at the Request level

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: None
userGroups:
- "system:serviceaccounts"
- "system:nodes"
users:
- "system:anonymous"
- "system:apiserver"
- "system:kube-controller-manager"
- "system:kube-scheduler"
- level: Request

It is also possible to describe the target:

  • namespaces;
  • verbs (get, update, delete, etc.);
  • resources (namely: pod, configmaps, etc.) and resource groups (apiGroups).

Resources and resource groups (API groups, i.e. apiGroups), as well as their versions installed in the cluster, can be obtained using the commands

kubectl api-resources
kubectl api-versions

You can say Hi to us at Pharos Production — a software development company

https://pharosproduction.com

Follow our product Ludo — the reputational system of the Web3 world

https://ludo.com

--

--

Dmytro Nasyrov
Pharos Production

We build high-load software. Pharos Production founder and CTO.