Since April 2018, the Doctolib test suite runs on Heroku CI and this post shares the motivation behind the switch as well as the insights we collected from it. Before the change over, it ran on a self-hosted Jenkins in combination with Browserstack for our end-to-end integration tests which makes up a decent portion of our test suite.
We made the switch for several reasons:
- The Jenkins instance had a non-negligible maintenance cost; it needed regular capacity upgrades, tune-ups, and random restarts.
- Browserstack had some stability issues that could affect us for an entire day every other month.
- Most importantly, both Browserstack and our Jenkins imposed certain limits on the number of concurrent builds that we could run. We had a definite number of test nodes on Jenkins that could start a build, and our Browserstack plan also had its limits — a definite number of parallel tests we could run, as well as a wait limit for the pending tests. This caused some serious delays for the builds during periods of peak developer activity. Sometimes our duty guy, responsible for the daily production rollout, had to resort to extreme measures; he could be heard saying something like, “sorry folks, I’m gonna have to kill your builds, I need a green one on production branch for deployment” in order to skip the build queue. With the growth of our developer team, this was a recurrent issue. …