CI/CD on the Edge — Part2
Why is Continuous Integration challenging on the Edge?
In 2014, a case study was done on a Finnish provider of wireless embedded systems on how Lean thinking and Agile methodology can be applied to embedded software development. The findings of this case study were published in an IEEE conference journal [9]. The researchers conducted the case study in two steps. The first step was discussions with a focused group of 9 people including process managers, developers, and testers while the second one was a survey with the rest of the department.
The survey included questions from the discussions and some extra questions as well. The cumulative result of both these approaches resulted in the formulation of what are the essentials and what are the challenges for the adoption of Lean principles and Agile methodology in embedded systems. One of the major shortfalls of this case study was its focus on only the software aspect of edge software development. Although well-founded, the findings are not tailor-made for the embedded domain.
Two years later in the same conference another journal article, Towards DevOps in the Embedded Systems Domain: Why is It so Hard? [10], addressed the shortcomings of the previous case study. Lean principles are the foundation of DevOps and have been tailored to the IT industry. The authors address the fact that rather than frequent addition of newer features, reliability, cost, and time to market drive the product development in edge software development. Due to the separation of hardware and software and their respective domain expertise, a collaboration between the teams is one of the biggest challenges. With data inputs from 4 different companies, working in different sectors of the embedded domain, the article enlists the following key challenges faced by embedded domain:
- Hardware dependency and version compatibility.
- Limited visibility of the customer test environment.
- Lack of technology to deploy reliably, automatically, and repeatedly new features at customer-specific environments.
In the same year, another journal article on problems and experiences while applying CI to software-intensive embedded systems was published. This article [11], was published after conducting 2 case studies on 7 different topics. The findings of this article extend the results of the previous journal article. The article suggests that product testing via simulations does not entail a guarantee of actual on-device use, and that long build and test times are often results of tightly coupled systems.
Analysis of CI Approaches
The findings, from the 3 journal articles above, clearly established the challenges faced by edge software development. But there were no suggested measures provided to the reader, neither complete nor partial ones. This shortcoming has been addressed by other research and as technologies have evolved so have the solutions.
For instance, in an article from 2007 titled Mocking the embedded world [12], experiences of a company Atomic Object are discussed. This article was leaps and bounds ahead of its successors and one of the reasons for that was their commitment to constantly improving their own process. The authors used a combination of Test Driven Development (TDD), CI and Model Conductor Hardware (MCH) design pattern. Their approach was simple but novel, for the edge development space. System tests were written first, then integration tests and finally the unit tests. They called it the Russian Dolls approach. This approach gave the project members better clarity and would weed out most issues in the beginning. They would then proceed with the unit tests and write the Conductor logic such that the tests pass. While this was being done, both Model and Hardware were mocked. This technique revealed the Hardware and Model interfaces required which could then be translated to physical hardware. For testing, they used “System testing in Ruby” (Systir) [13] along with miniLAB 1008. Through these they were able to send analog and digital I/O signals and record the received response by means of a PC connected to the miniLAB 1008. The system and integration tests served as regression tests to be used when the product evolved and to ensure the quality of delivery. Although equipped with novel techniques it did not address the issue about testing on multiple hardware or remote hardware for that matter.
In another report by Jakob Engblom in 2015 [14], he proposes the lack of hardware or complexity of hardware as limitations of CI and suggests the alternative as setting up a virtual platform with a simulator using a standard PC. The report proposes the use of both board simulators and environment simulators so as to mimic the customer test environment as close as possible. Alternatively, a hardware test rig with a master PC that loads software onto the board and another PC equipped with test data and special interfaces to test the embedded system with realistic data is proposed. Although the author notes that this would be an expensive alternative, this would also be limited to a particular board type and for projects where the devices are manually updated. This is a rarity in current times. Referring to the figure, it is visible that only the test part of CI phase is handled in this report, the build and merge parts are not addressed.
In the 30th IEEE Conference on Electrical and Computer Engineering, a journal article titled Continuous Integration Platform for Arduino Embedded Software was published. This article [15], acknowledges the hardware limitation and testing complexities as challenges for edge software development. The authors use Jenkins and some of its plugins as a build server and GitHub is used as DVCS. On a successful code commit to GitHub, Jenkins is triggered. The Arduinos are connected to the machine running Jenkins via USB. To execute test cases, Plank Unit [16] Tests have been used. They are a generic, lightweight library for executing unit tests on edge devices. One of the criteria to create the CI system was to use open-source software. A CMake toolchain was used to create and upload the executable(s) to the devices using AVRDude. The article also proposes the usage of code reviews as part of the CI process.
A 2019 Ph.D. thesis from the University of Groningen [17] concurs the challenges listed above. The thesis also suggests the use of the EMFIS model which assists the organization in prioritizing the areas of improvement for CI and CD. Finally, the thesis suggests the following pointers for designing delivery pipelines for large-scaled software-intensive embedded systems:
- Verifying the quality of changes which could be ensured using testing and code analysis.
- Secure stability and integrity in the system, which is very much in line with the TDD implementation done in the article [12].
- Optimize test activities and performance.
- Ensure traceability of features to the use-case.
After review of the existing literature, it was observed that most of them understood the challenges observed by the edge software domain and many of them even came up with some novel ideas but none had opted for a widely available holistic solution which could alleviate a majority, if not all, of the challenges above.
In an upcoming article, we try to picture a holistic system that considers the mentioned challenges.
References
[9] P. Rodríguez, J. Partanen, P. Kuvaja, and M. Oivo. Combining lean thinking and agile methods for software development: A case study of a finnish provider of wireless embedded systems detailed. In 2014 47th Hawaii International Conference on System Sciences, pages 4770–4779, 2014.[10] L. E. Lwakatare, T. Karvonen, T. Sauvola, P. Kuvaja, H. H. Olsson, J. Bosch, and M. Oivo. Towards devops in the embedded systems domain: Why is it so hard? In 2016 49th Hawaii International Conference on System Sciences (HICSS), pages 5437–5446, 2016.[11] Torvald Mårtensson, Daniel Ståhl, and Jan Bosch. Continuous integration applied to software-intensive embedded systems — problems and experiences. pages 448–457, 11 2016.[12] Michael Karlesky, Greg Williams, William Bereza, and Matt Fletcher. Mocking the embedded world: Test-driven development, continuous integration, and design patterns. ESC, 413, 05 2007.[13] systir, Mar 2020. [online] https://github.com/atomicobject/systir accessed 22-Mar-2020.[14] Jakob Engblom. Continuous integration for embedded systems using simulation. 02 2015.[15] W. Penson, E. Huang, D. Klamut, E. Wardle, G. Douglas, S. Fazackerley, and R. Lawrence. Continuous integration platform for arduino embedded software. In 2017 IEEE 30th Canadian Conference on Electrical and Computer Engineering (CCECE), pages 1–4, 2017.[16] plank-unit, Mar 2020. [online]https://github.com/iondbproject/planck-unit.[17] Torvald Martensson. Continuous integration and delivery applied to large-scale software-intensive embedded systems. PhD thesis, University of Groningen, 2019.
Note: For the purpose of this article, the word “embedded” has been interchangeably used with “edge”
DISCLAIMER: This post only reflects the author’s personal opinion, not any other organization’s. This is not official advice. The author is not responsible for any decisions that readers choose to make.
