MacPorts-Buildbot GSoC Report
Introduction
This summer I participated as a Google Summer of Code student under MacPorts. You can find my project abstract here: MacPorts Custom Views Plugin for Buildbot. It has been a wonderful experience so far, and I learned a lot about developing large scale software, thanks to my mentors: Mojca Miklavec, Pierre Tardy, Povilas Kanapickas, and all other members of the Buildbot and MacPorts communities; who helped me with my doubts and provided invaluable feedback.
The Project
I primarily worked on 3 tasks related to the Buildbot CI framework, which is used by MacPorts:
1. Custom Views Plugin
Buildbot frontend is written in AngularJS 1.x, which is an old framework for building single-page applications. Nowadays, developers prefer newer technologies such as Vue or React, which are comparatively easier to use. I wrote a plugin that would let developers write custom UI components using Vue. It is available here: https://github.com/macports/buildbot-macports-custom-views.
The basic plugin is ready and can be used directly by MacPorts (using PyPI package/port) or by hacking it (if other organizations/individuals want to use it for their Buildbot).
The challenging part was having multiple views (pages) inside one app. In the default Buildbot plugins, there is usually one page (a single item in navbar) inside one plugin. In this, I created a main AngularJS module and directives for each page (which is a single group in glMenuServiceProvider).
Another issue that I faced was manipulating the URL from Vue single-file components. One of the requirements was being able to bookmark some changes by id so that it can be shared/viewed later. At first, I tried using vue-router, however, I later realized that it wouldn’t work since the URL was controlled by AngularJS behind the scenes. So, we decided to use the $location API. This was tricky to implement. The new components allow us to see changes/builds before/after a particular date.
Here’s a demo of the changes list page:
What is left?
- Right now, there’s no support for live updates in the plugin UI (see issue #22).
- The Ports Build History page is using the macports-webapp API to fetch data. But this would be removed, and the view would be standalone (without any external API). The work is in progress in pull #22.
2. MacPorts Buildmaster
MacPorts was using Buildbot 0.8, hence the buildmaster needed a revamp to port to Buildbot 2.x. MacPorts has over 20k ports, and to tackle something of that scale, we decided to use virtual builders. In the new setup, we have a portwatcher, which is a builder that triggers build for a port (or list of ports). Now, for each of those ports, there is one virtual builder that starts running a build.
A mock setup that does fake builds is available here: https://github.com/macports-gsoc/macports-buildbot
A demo Buildbot instance has been deployed here.
What is left?
- We need to tweak the master.cfg to do real builds and deploy it to Braeburn (MacPorts server).
3. Features added to Buildbot
With over 20k ports, there would be more than 200k builders, which would make the default www plugins useless if used as-is. So I added some features like tagging, pagination, searching, etc to make it usable for large setups.
The builders page would now let us see all the ports, thanks to virtual builders.
Builds for Change page
I created a webpage that displays all builds for a change (see #3926). To do so, I added an endpoint to the data API to get all builds for a given change.
Committer field added to changes
Earlier, Buildbot had an author field for a change. In version control systems like git, the author and committer may be different. This is important for MacPorts since the author may lack commit permissions, and in case of build failure, both the author and the committer need to be notified.
- My commits to Buildbot are here
- Merged pull requests are here
- Pending pull requests are here
- My commits to macports-ports are here
What is left?
There are a few issues related to console_view which need some further discussion and design decisions (see #4951).
- All builds are not displayed for a change (#3945)
- Incorrect grouping of builds when using force/periodic scheduler (#3995)
These could be fixed if we used the APIs rather than having the complex frontend code which is currently implemented. However, this would lead to the chicken and egg problem: if we use the builds for change API, then we wouldn’t be able to get the forced/nightly builds (which were not triggered by any change), and if we use the changes for build API, then we wouldn’t get all the changes. We might eventually create 2 console views to fix these issues.
