Profiling Applications Deployed on Kubernetes using ‘Perf’ and Sidecar Injector

Pradipta Banerjee
Jun 27, 2018 · 4 min read

In our Kubernetes experiments, sometimes our team has the need to profile the deployed applications using Linux perf. We were looking for a mechanism where the perf data capture was specific to the deployed application and fully automatic, without any manual intervention.

This is when we thought of leveraging the automatic sidecar injector pattern, which is heavily used in Istio (service mesh). A few of our team members, namely Hemant and Christy, are part of the Istio development community and really helped to dig deeper into the nuances.

If you have a similar use case, this article might be helpful.

Automatic sidecar injection is actually a mutating admission webhook which is invoked during the mutation phase and allows modification of the resource object.

The perf sidecar injector is a mutating admission webhook which either uses an annotation or namespace label to add a sidecar to the pod.

The following picture shows a high level overview of how the pod gets modified (mutated) as part of the admission process.

Additionally, the sidecar injector leverages PID sharing to automatically capture the perf data for the pod processes.


Ensure you are using Kubernetes 1.10+ and the following settings are enabled:

  • Feature-gate PodShareProcessNamespace=true is turned on for both apiserver and kubelet
  • Ensure kube-apiserver has the admission-control flag set with MutatingAdmissionWebhook and ValidatingAdmissionWebhook admission controllers added
$ kubectl api-versions | grep admissionregistration

For IBM Cloud Private, you can use the kube_apiserver_extra_args and kubelet_extra_args configuration parameters during install.

For more details on the IBM Cloud Private install time config options, see the following link:

Before getting into the setup details, let’s take a quick look at the perf container which gets added as a sidecar via the injector.

The source of the perf container is available from the following GitHub link:

The perf container can be run with the following options: <stat|record > <trigger[1|0]> <max-run-time[X]> <repeat-count[N]> <extra-perf-args>


  • stat: Runs the perf stat command.
  • record: Runs the perf record command.
  • probe: Runs the perf probe command.
  • trigger: The perf command can be started in response to a trigger — create/open/modify events for the file /tmp/startperf.
  • max-run-time: The max time that the perf command will be run. The default value is 60 seconds.

A sample invocation looks like this:

perf stat -a -p 1,3,8 -x ',' -I 1000 -A -o /out/perf-stat-output-abcxyz --append -g

This captures perf stat for PIDs 1, 3, 8, including call-graph and dumps to /out/perf-stat-output-abcxyz.

  • Clone the source:
$ git clone
$ cd perf-sidecar-injector
  • Create a signed certificate/key pair and store it in a Kubernetes secret that will be consumed by the sidecar deployment:
$ ./deployment/ \
--service perf-sidecar-injector-webhook-svc \
--secret perf-sidecar-injector-webhook-certs \
--namespace default
  • Patch the MutatingWebhookConfiguration by setting caBundle with the correct value from the Kubernetes cluster:
$ cat deployment/mutatingwebhook.yaml | \
deployment/ > \
  • Deploy sidecar injector resources:

Edit the configmap as per your requirements. By default the perf data is kept in the hostPath/out

$ kubectl create -f deployment/configmap-record.yaml
$ kubectl create -f deployment/deployment.yaml
$ kubectl create -f deployment/service.yaml
$ kubectl create -f deployment/mutatingwebhook-ca-bundle.yaml
  • Verify:

Check to see whether the sidecar injector webhook is running using the following commands:

$ kubectl get pods
sidecar-injector-webhook-deployment-bbb689d69-882dd 1/1 Running 0 5m
$ kubectl get deployments
sidecar-injector-webhook-deployment 1 1 1 1 5m


There are two ways by which the webhook can be instructed to add the perf sidecar:

  1. annotation: Adding the perf-sidecar-injector-webhook/inject: “yes” annotation to the POD spec will ensure automatic injection of the perf sidecar.
  2. namespace label: Adding the sidecar-injector=enabled label to a specific namespace will ensure automatic injection of the perf sidecar to every pod deployed in this namespace.

Let’s use the namespace label option such that any pod deployed to the default namespace will have the perf sidecar added automatically.

  • Add label:
$ kubectl label namespace default sidecar-injector=enabled$ kubectl get namespace -L sidecar-injector
default Active 18h enabled
kube-public Active 18h
kube-system Active 18h
  • Deploy a sample application and collect perf data:

You can use perf report or other visualization tools to view the captured perf data.

Take a look at the following video for a complete demonstration on IBM Cloud Private:

IBM Cloud

Understand how to bring elastic runtimes to the Enterprise…

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store