Profiling Applications Deployed on Kubernetes using ‘Perf’ and Sidecar Injector

In our Kubernetes experiments, sometimes our team has the need to profile the deployed applications using Linux perf. We were looking for a mechanism where the perf data capture was specific to the deployed application and fully automatic, without any manual intervention.

This is when we thought of leveraging the automatic sidecar injector pattern, which is heavily used in Istio (service mesh). A few of our team members, namely Hemant and Christy, are part of the Istio development community and really helped to dig deeper into the nuances.

If you have a similar use case, this article might be helpful.

Automatic sidecar injection is actually a mutating admission webhook which is invoked during the mutation phase and allows modification of the resource object.

The perf sidecar injector is a mutating admission webhook which either uses an annotation or namespace label to add a sidecar to the pod.

The following picture shows a high level overview of how the pod gets modified (mutated) as part of the admission process.

Additionally, the sidecar injector leverages PID sharing to automatically capture the perf data for the pod processes.

Prerequisites

Ensure you are using Kubernetes 1.10+ and the following settings are enabled:

  • Feature-gate PodShareProcessNamespace=true is turned on for both apiserver and kubelet
  • Ensure kube-apiserver has the admission-control flag set with MutatingAdmissionWebhook and ValidatingAdmissionWebhook admission controllers added
$ kubectl api-versions | grep admissionregistration
admissionregistration.k8s.io/v1beta1

For IBM Cloud Private, you can use the kube_apiserver_extra_args and kubelet_extra_args configuration parameters during install.

For more details on the IBM Cloud Private install time config options, see the following link: https://www.ibm.com/support/knowledgecenter/SSBS6K_2.1.0.3/installing/config_yaml.html

Before getting into the setup details, let’s take a quick look at the perf container which gets added as a sidecar via the injector.

The source of the perf container is available from the following GitHub link: https://github.com/bpradipt/perf-container

The perf container can be run with the following options:

entrypoint.sh <stat|record > <trigger[1|0]> <max-run-time[X]> <repeat-count[N]> <extra-perf-args>

where:

  • stat: Runs the perf stat command.
  • record: Runs the perf record command.
  • probe: Runs the perf probe command.
  • trigger: The perf command can be started in response to a trigger — create/open/modify events for the file /tmp/startperf.
  • max-run-time: The max time that the perf command will be run. The default value is 60 seconds.

A sample invocation looks like this:

perf stat -a -p 1,3,8 -x ',' -I 1000 -A -o /out/perf-stat-output-abcxyz --append -g

This captures perf stat for PIDs 1, 3, 8, including call-graph and dumps to /out/perf-stat-output-abcxyz.

Deploy the ‘Perf’ Sidecar Injector

  • Clone the source:
$ git clone https://github.com/bpradipt/perf-sidecar-injector.git
$ cd perf-sidecar-injector
  • Create a signed certificate/key pair and store it in a Kubernetes secret that will be consumed by the sidecar deployment:
$ ./deployment/webhook-create-signed-cert.sh \
--service perf-sidecar-injector-webhook-svc \
--secret perf-sidecar-injector-webhook-certs \
--namespace default
  • Patch the MutatingWebhookConfiguration by setting caBundle with the correct value from the Kubernetes cluster:
$ cat deployment/mutatingwebhook.yaml | \
deployment/webhook-patch-ca-bundle.sh > \
deployment/mutatingwebhook-ca-bundle.yaml
  • Deploy sidecar injector resources:

Edit the configmap as per your requirements. By default the perf data is kept in the hostPath — /out

$ kubectl create -f deployment/configmap-record.yaml
$ kubectl create -f deployment/deployment.yaml
$ kubectl create -f deployment/service.yaml
$ kubectl create -f deployment/mutatingwebhook-ca-bundle.yaml
  • Verify:

Check to see whether the sidecar injector webhook is running using the following commands:

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
sidecar-injector-webhook-deployment-bbb689d69-882dd 1/1 Running 0 5m
$ kubectl get deployments
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
sidecar-injector-webhook-deployment 1 1 1 1 5m

Use

There are two ways by which the webhook can be instructed to add the perf sidecar:

  1. annotation: Adding the perf-sidecar-injector-webhook/inject: “yes” annotation to the POD spec will ensure automatic injection of the perf sidecar.
  2. namespace label: Adding the sidecar-injector=enabled label to a specific namespace will ensure automatic injection of the perf sidecar to every pod deployed in this namespace.

Let’s use the namespace label option such that any pod deployed to the default namespace will have the perf sidecar added automatically.

  • Add label:
$ kubectl label namespace default sidecar-injector=enabled
$ kubectl get namespace -L sidecar-injector
NAME STATUS AGE SIDECAR-INJECTOR
default Active 18h enabled
kube-public Active 18h
kube-system Active 18h
  • Deploy a sample application and collect perf data:

You can use perf report or other visualization tools to view the captured perf data.

Take a look at the following video for a complete demonstration on IBM Cloud Private:

https://asciinema.org/a/ROb7NNyibXXV1sRPg1ZG8jXjg