Monitor Kubernetes pod status from a Jenkins pipeline

On one of the projects that I was working on, there was a requirement from our developers to get a logs output in a Jenkins pipeline from a certain pod upon completion. The idea behind the pod is as follows, run on the cluster where the pod will essentially perform certain tasks and complete. Initially, my approach was to proceed with Jenkins’s Kubernetes plugin however there were certain limitations, hence I’ve decided to use a bit of a hacker method, and that is using a Python script.

My Jenkins infrastructure is as follows; a Kubernetes cluster added as Cloud where the build agents are launched as pods. For this specific pipeline, I’ve added a Python pod in which we’ll inject our kubeconfig as a secret.

In this project, I’m using the official Kubernetes Python SDK. Lucky for us Kubernetes has watch query parameter built into their API. Briefly explained; it opens a connection to the API itself to watch for event changes on a certain resource (More on that). With that integrated half of our job has been already done, now we only need to open a stream and watch for event changes, below you can find the actual Python code that I’ve used in production.

The below app is assumed that we are monitoring the status for a single pod.

Before running let's not forget to install the Kubernetes module.

Initially, the idea for the application was discovered on, wait for it…. StackOverflow

Please keep in mind that the app will hold the Jenkins job as running until status change has been detected!

Further what I would proceed with improving is, create the app more dynamic, to be able to specify pod, namespace, config.. events that you want to monitor by using arguments when running it (Using: argparse).

Thanks for reading and enjoy playing!