OpenInfra Summit Berlin 2022
Zuul is an Open Source CI system originally developed in the Open Stack community, now called Open Infra. Here at Volvo Car Corporation (VCC) Zuul is our default CI/CD chain at Research and Development, and we use it because of this mantra:
Stop merging broken code!
Our system is built in gating, and mainly uses the check, gate and release pipelines when the gate merges to master. The good thing is that this is not just for a single change in one repository, but for any set of dependent changes. This is especially great for us having many nodes in the car that depend on each other — each developed by a different team.
Because of this, Zuul tests the upcoming future state of the product, but only let changes in that pass tests. To my knowledge, Zuul is the only CI system that can have dependencies between changes in Gerrit, Github and Gitlab, and carry these changes through all the way to git merges in the respective repositories.
Speculative Merge
To be able to test each change in isolation before a merge, Zuul uses speculative merge. Speculative merge saves a lot of time by testing in parallel. The only limit is the build capacity.
Input Scalability
Zuul is still the CI for one of the largest Open Source projects on the planet. Although our Zuul usage is not as big, it’s still quite a lot with around 4000 builds each 24 hours.
Community
And last but not least, Zuul is an active open source developed tool with a fantastic community, it gives developer the chance to participate, if they so like.
For the second year, I have the honor to speak at the Open Infra Summit, and here is some of my experiences from the conference.
The last summit was virtual. But this time around traveling to Berlin was fantastic. I have always appreciated east Berlin.
This year Zuul celebrates 10 years, so this was a special occasion. The keynotes of day 1 were quite interesting, especially Mark Collier´s presentation. Mark is the CEO of the OpenInfra Foundation and he really showed what is the potential of the cloud market and Open Source Software. Some of the greatest software stacks in the world have come into existence through different companies collaborating. Open stack is a good example with over 8700 developers contributing from 450 companies. Mark predicts that over 50 billion devices will be connected by 2030 and that there are interesting areas of development, like mobile tele medicine, where specialists can perform surgery from a distance and of course, and self driving cars. These problems can not be solved by isolated companies alone, so collaboration is necessary.
A talk that really stood out, was Building Environmental Dashboards for OpenStack at BBC R&D. They build Grafana dashboards that let users of their private OpenStack cloud track the carbon dioxide emissions resulting from each of their VMs. Combining real-time data from electricity producers with metrics from the compute hosts, they gave users a tool to monitor, and eventually to reduce, the environmental impact of their various workloads.
There were many interesting talks about Zuul, from its advanced features to a discussion among some of the developers in how to develop the security aspects to the system. For me, the talk from a large German car brand, No gain without pain, was very interesting. As the title indicates, they struggled too with optimizing all the components of the system to match scale. They actually run up to 6000 jobs in one hour!
They made a company direction decision early on to use Zuul, where we went from shadow IT to be default.
Then of course, James Blair and I gave a keynote about our growth and usage of Zuul here at Volvo Cars.
James Blair, the creator of Zuul and we collaborate with his company ACME Gating. James built an enterprise version of Zuul for us, where we sometimes are ahead of the upstream development, and sometimes after. Of course, all work is pushed up streams to the Open Source project.
In our keynote we spoke about how we went from shadow IT to become the default CI chain for the major part of the code that enter the core computer.
The cooperation with James has been a game changer, and we got many features developed for us: improved Azure drivers, semi-static nodes and enterprise wide semaphores. Our CI software stack contains of these technologies:
Zuul, Ansible, Gerrit, Artifactory, Grafana, Splunk, AWS, Azure & Kubernetes.
Since 2021 up until today we grew a lot, we went from around 200 Gerrit repositories up to over 600, and this proved to be a challenge.
Of course all the new teams needed support, but our back end of Zuul also needed some love.
Basically, teams that went from small scale Jenkins setups complained about the speed or to be more precise: time it takes from when a developer pushes a change to Gerrit until the first job starts in the pipeline. When it was at its worst, this could for short periods take up to 2o minutes.
Now, the average is around 1.5 minutes:
Basically, the solutions was: more detailed metrics, better use of Zuul tenants and optimization of the Zuul reconfiguration.
We also mentioned our current setup of the back end, we use 6 schedulers, 10 executors and 6 web pods all run in an EKS K8 cluster.
All in all, it was a fantastic experience to talk to all Open Source developers from around the globe, #weareopeninfra.
Watch the talk
You can watch the whole talk here, or just skip to the part where I talk about our setup at VCC:
The second and more detailed talk:
