How to Choose the Right Continuous Integration Tool for Embedded Systems
Intro
This is the second post in a series of posts reviewing continuous integration for embedded software in IoT (take a look at our last post). This post presents a comparison between five continuous integration tools while examining them from the embedded software development perspective.
When examining a continuous integration and delivery tool for embedded systems, we chose to focus on the modularity of the tool. Our thinking is that as every embedded software project (mainly in IoT) is different and the developer will need to be able to adapt the tool to support his unique setup. Another consideration is being able to use the tool in case the company providing it decided to shut down the project (happens with large companies and startups). Embedded software companies need to make sure the tools they are using will last the project life time, which can even span to 15–20 years in Industrial IoT for example.
Besides these two considerations we also checked if the tool can support Hardware In the Loop tests, for “on device” tests, node-locked license tool chains and 3rd party integrations support.
We compared the following options: Jenkins, TeamCity, Travis CI, Circle CI and Buildbot. A General comment: the web is filled with comparisons between these tools, so we only added relevant features for embedded software and not a complete overview of the tools.
Note — when talking about a hosted solution it means it’s a cloud based service, managed by the company that created the tool itself.
Check out our proposal for how to build a continuous integration process for embedded here.
Jenkins
An open source, free to use, automation server. Jenkins supports many version control systems. Being an open source project that is being used across the board in many industries, it’s highly likely you’ll have support for the entire project life span. Jenkins can run locally or remotely so it supports Hardware In the Loop tests and can run compilers with node locked licenses.
Jenkins installation is simple, but as it is self-hosted initial setup time and maintenance take some time. In Jenkins, you have to rely on Plugins and there are over 1000 of them that can address your integration needs.
To sum it up, Jenkins is suited for embedded software development but requires resources from the company. As it’s an open source framework it means your can customize everything from the core to plugins to get all the needed flexibility. You can learn more about Jenkins for embedded here.
TeamCity
An automation server by JetBrains that offers out of the box usability. TeamCity provides integration to Eclipse and has a bunch of plugins for 3rd party integrations as well. TeamCity supports many version control systems. TeamCity offers full features in its free version and is only limited in build configurations and agents. TeamCity can run locally or remotely so it supports Hardware In the Loop tests and can run compilers with node locked licenses.
TeamCity offers more usable interface than Jenkins, reduces the administrative overhead and has a shorter learning curve than Jenkins.
To sum it up, TeamCity is suited for embedded software development. On the one hand it’s not an open source so you can’t know for sure, you’ll have support for the next 15 years, but it’s a mature tool from JetBrains so the risk is minimal.
Travis CI
An open source hosted service, which is free for open source projects. Start working with Travis CI is very easy, and it is configured using .travis.yml files, so getting started is faster than with TeamCity or Jenkins. Travis CI uses VMs for the build process. Travis CI supports only projects that are hosted on GitHub. With Travis CI you won’t be able to compile with node-locked licenses, and won’t be able to run Hardware In the Loop tests.
As Travis CI is a hosted service and can only work with GitHub hosted projects, it’s not a good fit for embedded software projects. If you are using GitHub, compile with GCC and don’t test on physical device, for example, it can fit your needs.
Check out this Travis CI process sample for STM32.
CircleCI
CircleCI provides both hosted and on premise service. CircleCI by far reduces maintenance and setup time in comparison to the other tools we’ve evaluated. Configuration is done using a YAML file. CircleCI is using containers to run jobs and let you choose the level of parallelization that suits your needs (require less system resource than Travis CI). CircleCI supports only Git repositories and integrates into GitHub seamlessly.
With version 2.0, CircleCI allows to reproduce environments locally, so that might work for Hardware In the Loop tests and node-locked licenses (running the container with — -net host), but we haven’t checked it.
Although CircleCI is modern and easy to use, it lacks the needed modularity for embedded software development as with Travis CI, but with version 2.0, they introduce some new features that need be checked.
Check out this Circle CI process sample for STM32.
Buildbot
An open source, free to use, Python based tool so it can run everywhere Python runs. Buildbot’s configuration files are Python scripts which mean you get flexibility, ease of use and many capabilities. Buildbot supports many version control systems. Buildbot learning curve is steep but gives a developer a lot of power. Buildbot installation and configuration is not an easy task and documentation can be found here. Buildbot can run locally or remotely thus it supports Hardware In the Loop tests and can run node locked tool chains.
Buildbot is suited for embedded software development for the same reasons Jenkins is, but we tend to like Jenkins more due to the abundance of plugins and tools integrations.
Final Thoughts
Choosing a continuous integration tool for embedded is different in many ways from choosing it for web applications development. It depends on your development tool chain licenses, supported platforms, version control system, lifetime support needs and where you need to run your tests. When choosing a modular continuous integration tool, it comes with long configurations and setup time which is something to consider.
About Jumper Labs
Jumper makes it fast and easy to test and build embedded IoT software. Jumper eliminates the hassle of testing on hardware devices using a unique behavioral simulation. Unlike testing on hardware, Jumper’s simulation is consistent, predictable and easy to automate.
With Jumper you benefit from 40% faster time-to-market, 20% increase in delivering new features, while decreasing a production error response time in an order of magnitude.
