The Hack typeface build testing shootout: Travis CI vs. Semaphore CI

Over the last few months, we’ve compared build testing of the Hack typeface between the free, open source account plans that are offered by the Travis and Semaphore continuous integration testing services. This testing has been part of an examination of our CI testing approach across the Source Foundry software repositories that include open source Python, Go, C/C++, and UFO typeface source software.

Why Test?

The Semaphore team reports a 50% reduction in test times for the elixir-lang/elixir test suite relative to times that were observed on the Travis CI and Circle CI testing platforms. Here is a screen capture of the data from the semaphore.com landing page:

Intriguing! As a long time user of Travis CI, I decided to put the test services to the test and find out if these benefits apply to our software testing needs. The following analysis compares the two testing services with the same git commits that were pushed to the Hack typeface repository and built simultaneously on both CI testing platforms via GitHub git commit hooks.

The Build

The build process involves the following series of steps:

• Installation of Python typeface compile and post-compile font modification software with pip
• Compile of the FreeType & Harfbuzz libraries and the ttfautohint executable from C/C++ source with a make-based compile workflow
• Compile of four full character set font variants (regular, italic, bold, and bold italic) in desktop (*.ttf) and web font ( *.woff, *.woff2) formats from UFO version 2 typeface source files
• Compile of four character subset font variants in web font (*.woff, *.woff2) formats from UFO version 2 typeface source files
• Post-compile modification writes to the font binaries with instruction sets (i.e., hints)
• Post-compile modification writes to the font binaries with other OpenType table fixes including DSIG table mocking, fstype permission modification

This build process has historically required approximately 4–5 minutes on the Travis platform. We perform parallel shell script linting tests that are not considered in this analysis as they are brief relative to our compile tests and do not impact the total time to test completion. The time-intensive tests for us are those shown in the list above.

The Testing Platforms

The Travis and Semaphore tests were both performed on Ubuntu 14.04 LTS systems. The git versions were 2.13.0 on Travis and 2.14.2 on Semaphore. The gcc versions were 4.8.4 on Travis and 4.9 on Semaphore. The Python versions were 2.7.13 on Travis and 3.6.x on Semaphore.

Test Results

Here are the data on the build execution times required for five recent sequential commits in our repository. These data exclude queue time and represent the software build testing execution time only. The builds were run on both testing services simultaneously via GitHub git commit hooks.

Raw build times by git commit

The following figure demonstrates the raw build time in seconds by git commit to the Hack typeface repository. The git commits are labeled with git commit SHA1 short hash strings on the y-axis.

Build times (seconds) by Hack typeface repository git commit for Semaphore (blue) and Travis (red) builds

Summary statistics

Mean (+/- SD) for build times across five commits to the Hack typeface repository for Semaphore (blue) and Travis (red). p<0.05

There was a statistically significant (p<0.05) decrease in build time across these five recent commits that amounted to a mean 110.8 seconds saved on the Semaphore CI service. The mean build times were 208.6 seconds (+/- 16.7s) on Semaphore and 319.4 seconds (+/- 33.6s ) on Travis. Nearly two minutes were shaved off of our builds on average with the default Semaphore test settings! That’s pretty impressive by any standard.

Why is Semaphore CI faster?

I reached out to the co-founder of Semaphore CI, Marko Anastasov, to see if he could explain these differences and here is what he had to say:

“At Semaphore we take performance very seriously and our CI/CD platform is based on dedicated hardware with desktop-class performance. As a result the hardware capacity that comes in one CI worker (we call it a Box) is 2x comparing to other services.”

Marko also had the following recommendation for additional potential gains in projects like ours:

“We also recommend using the built-in caching mechanism to avoid re-building external dependencies that don’t change between builds.”

The documentation for the built-in Semaphore CI dependency caching that he referred to can be found here.

Conclusion

Based upon these findings and those from similar analyses in our Python software repositories, we are transitioning to Semaphore CI for most of our continuous integration testing across our Go, Python, and UFO typeface source projects. This service is worth a look for other teams that have relatively lengthy test build processes. You might identify significant time savings with a simple configuration tweak on your repositories.

Data

All of our historical testing data are open and available for review by git commit on our Travis and Semaphore accounts for those who would like to verify these findings and confirm that they have been stable over time. The data for this analysis are available in this Google docs spreadsheet.

• Semaphore: https://semaphoreci.com/
• Travis: https://travis-ci.org/