Profiling Applications Deployed on Kubernetes using ‘Perf’ and Sidecar Injector
In our Kubernetes experiments, sometimes our team has the need to profile the deployed applications using Linux
perf. We were looking for a mechanism where the
perf data capture was specific to the deployed application and fully automatic, without any manual intervention.
This is when we thought of leveraging the automatic sidecar injector pattern, which is heavily used in Istio (service mesh). A few of our team members, namely Hemant and Christy, are part of the Istio development community and really helped to dig deeper into the nuances.
If you have a similar use case, this article might be helpful.
Automatic sidecar injection is actually a mutating admission webhook which is invoked during the mutation phase and allows modification of the resource object.
perf sidecar injector is a mutating admission webhook which either uses an annotation or namespace label to add a sidecar to the pod.
The following picture shows a high level overview of how the pod gets modified (mutated) as part of the admission process.
Additionally, the sidecar injector leverages PID sharing to automatically capture the
perf data for the pod processes.
Ensure you are using Kubernetes 1.10+ and the following settings are enabled:
PodShareProcessNamespace=trueis turned on for both
admission-controlflag set with
ValidatingAdmissionWebhookadmission controllers added
$ kubectl api-versions | grep admissionregistration
For IBM Cloud Private, you can use the
kubelet_extra_args configuration parameters during install.
For more details on the IBM Cloud Private install time config options, see the following link: https://www.ibm.com/support/knowledgecenter/SSBS6K_188.8.131.52/installing/config_yaml.html
Before getting into the setup details, let’s take a quick look at the
perf container which gets added as a sidecar via the injector.
The source of the
perf container is available from the following GitHub link: https://github.com/bpradipt/perf-container
perf container can be run with the following options:
entrypoint.sh <stat|record > <trigger[1|0]> <max-run-time[X]> <repeat-count[N]> <extra-perf-args>
stat: Runs the
record: Runs the
probe: Runs the
perfcommand can be started in response to a trigger — create/open/modify events for the file
max-run-time: The max time that the
perfcommand will be run. The default value is 60 seconds.
A sample invocation looks like this:
perf stat -a -p 1,3,8 -x ',' -I 1000 -A -o /out/perf-stat-output-abcxyz --append -g
perf stat for PIDs 1, 3, 8, including call-graph and dumps to
Deploy the ‘Perf’ Sidecar Injector
- Clone the source:
$ git clone https://github.com/bpradipt/perf-sidecar-injector.git
$ cd perf-sidecar-injector
- Create a signed certificate/key pair and store it in a Kubernetes secret that will be consumed by the sidecar deployment:
$ ./deployment/webhook-create-signed-cert.sh \
--service perf-sidecar-injector-webhook-svc \
--secret perf-sidecar-injector-webhook-certs \
- Patch the
caBundlewith the correct value from the Kubernetes cluster:
$ cat deployment/mutatingwebhook.yaml | \
deployment/webhook-patch-ca-bundle.sh > \
- Deploy sidecar injector resources:
configmap as per your requirements. By default the
perf data is kept in the
$ kubectl create -f deployment/configmap-record.yaml
$ kubectl create -f deployment/deployment.yaml
$ kubectl create -f deployment/service.yaml
$ kubectl create -f deployment/mutatingwebhook-ca-bundle.yaml
Check to see whether the sidecar injector webhook is running using the following commands:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
sidecar-injector-webhook-deployment-bbb689d69-882dd 1/1 Running 0 5m$ kubectl get deployments
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
sidecar-injector-webhook-deployment 1 1 1 1 5m
There are two ways by which the webhook can be instructed to add the
- annotation: Adding the
perf-sidecar-injector-webhook/inject: “yes”annotation to the POD spec will ensure automatic injection of the
- namespace label: Adding the
sidecar-injector=enabledlabel to a specific namespace will ensure automatic injection of the
perfsidecar to every pod deployed in this namespace.
Let’s use the namespace label option such that any pod deployed to the
default namespace will have the
perf sidecar added automatically.
- Add label:
$ kubectl label namespace default sidecar-injector=enabled$ kubectl get namespace -L sidecar-injector
NAME STATUS AGE SIDECAR-INJECTOR
default Active 18h enabled
kube-public Active 18h
kube-system Active 18h
- Deploy a sample application and collect
You can use
perf report or other visualization tools to view the captured
Take a look at the following video for a complete demonstration on IBM Cloud Private: