Deploying Buildbot Nine on Kubernetes

As Buildbot Nine is gaining more horizontal scalability features, it has more than ever its place on IaaS (Infrastructure-as-a-service). You can have the proper balance between the power you want to allocate to your build infrastructure and the cost of such an infrastructure. Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.

Before following this tutorial you need:

• A Kubernetes cluster you can play with (for exemple minikube)

The goal of this tutorial is to deploy a sample test infrastructure for pyflakes (similar to the tutorial of Buildbot). These infrastructure will run:

• 1 crossbar router
• 1 postgreSQL database
• 2 masters
• 3 workers

You can download all configuration files here.

# minikube start  # See minikube documentation for the options
git clone https://github.com/Zempashi/buildbot_kubernetes_deployment
cd buildbot_kubernetes_deployement/kubernetes

Before deploying Buildbot into Kubernetes, you need to give Buildbot the URL were the webui will be available. It may depend on your Kubernetes setup (node-ip, load-balancer, ingress). For minikube you need to run:

minikube ip

Set the URL in buildbot-masters-config.yml

apiVersion: v1                            
kind: ConfigMap                           
metadata:                                 
  name: buildbot-config                   
data:                                     
  master_url: 'http://__the_minikube_ip__:30010/'
  num_workers: '3'

You also see the num_workers config with the value 3. You don’t need to modify this value, It already matches the number of workers in buildbot-workers.yml (under spec.replicas).

You’re ready to deploy !

kubectl apply -f .

If you go to the URL you provided Buildbot you will shortly see the Buildbot webui. You will see 3 workers, 2 masters. You can trigger build under Builds > Builders > runtests > force (at top right) > Start Build .

Buildbot test build of pyflakes (on Kubernetes)

What happens ?

Kubernetes has launched pods deployements from crossbar.yml , pg.yml , buildbot-masters.ymland buildbot-workers.yml . All these are pods (set of containers). They reach each other via services.

The crossbar and the postgreSQL services are optional if you have external services. You can adjust the URL provided in the buildbot-master.cfg.yml

[...]
c['mq'] = {                               
    'type': 'wamp',                       
    'router_url': 'ws://crossbar:8080/ws',
    'realm': 'realm1',                    
}
[...]
c['db'] = {
     'db_url' : "postgresql+psycopg2://buildbot:{PG_PASSWORD}@pg-buildbot/buildbot".format(
         **os.environ
     ),
 }

You see in the example that you can reach the crossbar on the crossbar host and the db on the pg-buildbot host. That is the nice feature of Kubernetes services: it creates resolvable names for services.

Workers also use service to connect to masters in buildbot-workers.yml :

env:
- name: BUILDMASTER
  value: 'buildbot-worker-endpoint'

This service match the two master containers and acts as a loadbalancer.

This deployement is only a sample that show the main capabilities of Kubernetes. You can look the yaml files in details, and refer to the documention of services, deployment, volumes, stateful set,and more.

Kubernetes dashboard for an overview of Buildbot pods

Let’s play a little…

You can experiment Kubernetes infrastructure healing. You can delete pods via the webui (minikube dashboard for minikube) or via the kubectl command line, and you will see Kubernetes fixing your deployment by spawning new pods. Deleting a master will kill a build running but won’t prevent the system to accept new one ! Due to a known limitation Buildbot has difficulties to recover the loss of the crossbar router: all masters will freeze. But that’s enough for Kubernetes to detect it via the livenessProbe and to trigger a restart of all the affected masters.

You now have an highly-available CI infrastructure !