Featured

Kubectl-r[ex]ec: A kubectl plugin for auditing kubectl exec commands

By Marton Natko, Staff Engineer — Infrastructure, Adyen

As we’ve moved more towards containerized workloads at Adyen, we’ve increased our velocity of getting things done. We have also embraced one of our formula points — Launch fast and iterate — as containers provide an optimal way of doing fast release cycles . However, we know that velocity must not come at the cost of being compliant with regulations.

We choose Kubernetes as a container orchestrator for many of our greenfield projects and, to provide proper engineering autonomy, we give our engineers access to interact directly with it. Autonomy is especially handy for debugging, as engineering teams own their own applications.

Enter kubectl exec

If you are familiar with the Kubernetes ecosystem, then the kubectl exec command comes to mind when debugging a running application in Kubernetes.

If you are unfamiliar with it, the quick explanation is that it allows a user to execute commands inside a container running on Kubernetes, or even further, it is possible to get a TTY(teletype) in said container.

In our industry, it is imperative to have audit logs of such events. The problem is that not many solutions provide this feature.

Let’s go deeper

To solve this problem, let’s examine the data flow more. We’ll only cover the part between the user and the apiserver, which is the most convenient part of devising a solution.

When a user executes a kubectl exec command, we can easily catch the command parameter from the request. But suppose the user also requests TTY and passes their STDIN to the container, they will end up with a TTY inside the container, and any command executed will go through an upgraded protocol. For Kubernetes versions below v1.30, this was SPDY while in v1.30, Websocket became the default. At this point, it gets more complicated because auditing what is executed inside the container through these sessions is less convenient to observe.

Our Solution — kubectl-rexec

We decided to keep all the features that kubectl exec provides while considering possible solutions. So, we came up with the following problem statement: what if we have the kubectl execcommand as a kubectl plugin, but call a different API endpoint that ends up in a component we control?
Kubernetes has the APIService resource that allows us to extend the Kubernetes API server with the new endpoints.

We introduced an APIService shown in the code block below. It will make requests with the path /apis/audit.adyen.internal/v1beta1/… to be proxied to the service called rexec in the namespace kube-system.

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.audit.adyen.internal
spec:
  group: audit.adyen.internal
  groupPriorityMinimum: 100
  caBundle: caCertAsBase64… 
  service:
    name: rexec
    namespace: kube-system
    port: 8443
  version: v1beta1
  versionPriority: 100

We made a kubectl plugin implementing exactly what exec does except for one thing: instead of calling api/v1/namespaces/{{ namespace }}/pods/{{ pod }}/exec, it calls apis/audit.adyen.internal/v1beta1/namespaces/{{ namespace }}/pods/{{ pod }}/exec. This request ends up — through the APIService — in a component we control; let’s call it “rexec” proxy.

In the rexec proxy, we do two things: first of all, we rewrite the path to the native exec path and then we proxy it back to the Kubernetes APIserver. We have to mention here that this proxying happens through impersonation because, in this layer, we do not have the original user authentication details anymore; we only have the user and groups, which we can pass along through impersonation.

The proxying can happen in two manners:

1. When the user does not request TTY, we log the command parameters from the request as an audit event and proxy the request directly to the Kubernetes API server.
2. When TTY is requested, things become more complicated. In this case, the proxy will not forward the request directly to the Kubernetes API server. Instead, it will proxy it onto itself via a TCP listener on a Unix socket that is being started for each exec request. Each of these TCP listeners act as TCP proxies; proxying requests to the Kubernetes API server. By having a TCP proxy, we can investigate the raw TCP traffic, parse the Websocket frames and capture each keystroke the user’s terminal sends into the container’s TTY. Additionally, having a TCP session per user is an easy way to keep track of the user’s identity and session while remaining on the TCP level.

Now that we have solved half of the problem– we have audit logging–we still need to enforce that users do not use the native command. To do this, we added a ValidatingWebhookConfiguration.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: deny-pod-exec
webhooks:
- name: deny-pod-exec.k8s.io
  clientConfig:
    service:
      name: rexec
      namespace: kube-system
      port: 8443
      path: /validate-exec
    caBundle: caCertAsBase64...
  rules:
  - apiGroups: [""]
    apiVersions: ["v1"]
    operations: ["CONNECT"]
    resources: ["pods/exec"]
  admissionReviewVersions: ["v1"]
  sideEffects: None
  failurePolicy: Fail

This configuration will do one thing: every time the Kubernetes API server receives a request targeting api/v1/namespaces/{{ namespace }}/pods/{{ pod }}/exec, the api will call the rexec proxy with the path /validate-exec.
Now, this means we also have a way to gatekeep the native exec requests; we just have to find a way to deny everything not coming through rexec proxy itself. To tackle this, on exec requests coming through our proxy, we add a header with a value acting as a shared key between the two endpoints we provide. If the header is present and the value is the same, we can allow the request to go forward.

Conclusion

With this minimalistic application, we can easily audit exec commands, and we only have to install a few manifests on the Kubernetes side while distributing our plugin to our engineers.

If you are interested in trying it out, check it out on Github: https://github.com/adyen/kubectl-rexec.