How to Build a Continuous Integration and Delivery Process for Embedded Software

Researches of SW R&D teams show significant improvements in all R&D and operation metrics when proper continuous integration and delivery processes were implemented (take a look at this case study). We thought it would be highly beneficial to demonstrate how to apply these practices to embedded SW teams. Applying continuous integration and delivery for embedded SW development is not an easy task as you have a physical device with many constraints when it comes to rapid testing. However, the gained benefits outweigh the effort. For more info about CI advantages and considerations when it comes to embedded SW check out this post.

In this post, I’ll outline a guideline for building such a process for embedded SW development. Note that this post is relevant even if your company privacy policy implies restrictions on using cloud services for source control and other CI/CD tools. This post will be focused on the IT infrastructure and tools you need in order to implement CI/CD process in your organization.

CI/CD Flow overview

The basic CI/CD flow for embedded SW development will be one of this kind

Let’s dig down further and discuss the details of our suggested flow:

We chose to work with the Git Flow proposed by Vincent Driessen. In a future post we’ll dig down further and discuss how to implement Git Flow for embedded SW.

Update: here’s the Git Flow post link.

To enable this, we propose the following IT infrastructure.

IT Infrastructure for CI/CD flow

When building a CI/CD IT infrastructure for embedded SW we decided to use more flexible tools that can work on-premise and as hosted cloud services (there are companies that have strict privacy and security policies which prevent them from using hosted cloud solutions). Additionally, toolchains such as IAR that have a node lock license or tools like Segger (check out how you can use Segger for test automation), and when doing on target tests, you must have an on-prem server to serve as your test harness. All of these components run on VMs (either windows or Linux ones). We decided to use different VMs for different types of test for modularity and support for future infrastructure updates.

CI/CD Master Server

We decided to use Jenkins as our CI/CD manager. Jenkins can run on-prem and on the cloud, on Windows/Linux machined, supports HiL tests and everything you need for embedded SW development. We are also very familiar with it, so it was a straightforward choice for us to make.

Source Control Server

We decided to use GitLab as it supports both on-prem and hosted cloud service. GitLab has a very friendly user interface and reliable plugin to Jenkins. In an on-prem scenario, we also connected the VM to a central backup service. If you don’t have an on-prem constraint, GitHub is probably the way to go.

CI Build Server

Our build server will also perform static analysis tests. For a build server, we used one of the two options:

1. IAR build server — an on-prem Windows VM with Node lock license. Note that static analysis tools require a more expensive IAR license.
2. GCC ARM build server — can be on-prem or cloud-based Linux VM (or even Docker-based).

Non-Host Testing Server

A Linux based VM, where you can run all non-target tests — unit tests and module testing with mocking. Our favorite tools for unit tests for embedded C are the tools provided by ThrowTheSwich.org — Unity, CMock, and Ceedling.

Simulated Testing Server

A Linux based VM, where you can run your binaries on a simulator of your devices and your entire system. There are few options for that like Synopsys’ VDK, WindRiver’s Simics, QEMU, Mentor’s Vista, etc. We recommend our own solution — Jumper’s Virtual Lab (we’ll have a different post covering this as well). A proper simulation/emulation environment should allow you to run-

1. Unit tests
2. Simulating sensors and peripherals I/Os
3. Smoke tests — device and cluster of devices
4. Black box (end to end) tests
5. System level tests — load/stress, performance, security, etc. tests
6. Cluster (scalability) testing with communication simulation
7. Acceptance tests
8. Performance tests

Target HW Testing Server

A Linux/Windows VM (the physical server should be located in the physical lab), where you can run tests on the target HW. This server will act as the testing harness or will be connected to an external HW testing harness. This server will control lab testing equipment such as scope, network analyzer, voltmeter, logic, packet generator, Segger tools, etc. We haven’t chosen specific testing framework here, as it depends on the specific device and system setup. On the target device and physical lab you should run -

1. Smoke tests
2. Black box tests
3. System level tests
4. HW validation and timing related tests
5. Cluster testing with communication testing (note that the scalability here is limited)
6. Certification tests
7. Acceptance tests

Streamline Your CI/CD Flow and Automated Testing

The fundamental requirement for CI/CD flow is to have a fast feedback loop from code commit through integration to test results. The way to get there is through automating your process and mainly having automated tests. At Jumper, we’ve built a Virtual Lab platform to streamline CI/CD and automated testing. Jumper’s Virtual Lab allows R&D and QA teams to have a virtual production clone that runs the same physical device’s software executables with complete environment and communication simulation.

If you work with embedded systems and want to have an easy way for testing your physical device’s embedded software, you’re invited to try Jumper for free.

Summary

Building an infrastructure for CI/CD in your organization will require some effort from your R&D and DevOps/IT teams. It will require new budget allocation to buy tools to support this process, and you’ll probably need to implement new development practices with your R&D team (check out these five practical steps for moving from manual to automated testing). Nevertheless, when it will be fully embedded into your R&D cycle you’ll get significant ROI very soon.

Share Your Thoughts

Feel free to share your thoughts and contact us in the comments section or on feedback@jumper.io.