Let’s take some baby steps, Continuous Integration (CI) using GitLab
In an article series called “Is it too cold and damp?” we implemented a solution that utilizes our open-source sensor to ensure that the storage environment for our musical instruments is optimal, i.e., not too wet, not too dry, or too hot. The visualization and alerting were done in Grafana.
You can find the source code for all parts on our GitHub page.
Part of the project has been to learn more about building pipelines and learning more about CI/CD in GitLab. The project has software deployed in an embedded system and a back-end running in Google Cloud Platform.
Custom Docker Image
The HiMinds Env sensor is ESP32-based; we use the Mongoose OS on it, which requires the MOS-tool to build the firmware image. So we decided that the easiest way forward was to create a custom Docker image; it looks like this:
The Docker Image contains the MOS tools (line 10), cppcheck (line 7), and flawfinder(line 8) to test our c-code and jshint (Line 22) to test our JavaScript code.
From cppcheck website
Cppcheck is a static analysis tool for C/C++ code. It provides unique code analysis to detect bugs and focuses on detecting undefined behaviour and dangerous coding constructs. The goal is to have very few false positives. Cppcheck is designed to be able to analyze your C/C++ code even if it has non-standard syntax (common in embedded projects).
From flawfinder website
This is the main web site for flawfinder, a simple program that examines C/C++ source code and reports possible security weaknesses (“flaws”) sorted by risk level. It’s very useful for quickly finding and removing at least some potential security problems before a program is widely released to the public.
From JSHint
This is JSHint, a tool that helps to detect errors and potential problems in your JavaScript code.
We build the image and push it to our DockerHub account. This makes it accessible to the GitLab runner.
If you have docker installed and want to try it just do the following:
$ docker pull surudissanaike/himinds-env-sensor-ciCI Stages
The stages that we want to handle in our CI are the following:
- Source
- Build
- Test
- Deploy
The general idea is that it will trigger a notification if we get a failure in any of the stages (an e-mail is sent to someone). If you have never used GitLab, I recommend you to read the following documentation about how the GitLab CI/CD works.
Source stage
Usually, you trigger your pipeline if there is a change in your source code. Each branch in GitLab can have its own .gitlab-ci.yml. The only exception that I can think of that we have encounter is a project where we only wanted to trigger the pipeline if some specific files were changed. However, I am confident that you will find your own way.
Build and test stage for HiMinds Env Firmware
Here is an overview of our different stages for this project:
As you can see above, for the HiMinds Env Firmware project, we have implemented the build and test stage (only static code analysis). The build stage will basically use the “mos tool” to build the firmware image and store the output as a GitLab artifact (for 2 weeks).
In the static code analysis phase, we run three different tools to check our source code; this is only for educational purposes. They don’t really do anything useful. The Embedded JavaScript engine that Mongoose OS uses and JSHint don’t play well together. Our c-file is basically a dummy file, so cppcheck and flawfinder will not do anything useful. The cool thing is that the CI is prepared for some serious coding. 😃
Our .gitlab-ci.yml is pretty straightforward; we use the “himinds-env-sensor-ci” image we pull from DockerHub. We define two stages, and the “script:” rows show you how we use the different tools.
Worth mentioning is that flawfinder is set to the highest risk level (5). The exit code from cppcheck is changed to 1, default is 0; if we don’t change this, the stage will always pass. At the moment, we allow failures at each stage.
You can download the resulting firmware image from the pipeline page.
We only store the artifact for 2 weeks.
Build, test and deploy for NodeJS application
Here is an overview of our different stages for this project:
As you can see above, for the NodeJS project called “backend-mqtt-database-service,” we have implemented the build (install) and test stage.
The test stage runs both a static code analysis and a dynamic code analysis with code coverage. We use JSHint, jscpd, a copy/paste detector for programming source code, and the unit test is written using the Jest test framework. We only have unit tests for one source file; that is why we fail in this stage.
All the stages are scripts in the package.json which means that they are easy to test from the CLI. The most interesting parts are listed below:
You can see how we configure jest to include code coverage. We fail the Dynamic Code Analysis stage because we only have 9.47% code coverage at the moment.
Our .gitlab-ci.yml is straightforward; we use the “node image” we pull from DockerHub. We define three stages, and the “script:” rows show you how we use the different tools. First, the dynamic-code-analysis stage stores the test result as a junit.xml file. We do this because we want to see the test report in GitLab; more info can be found here.
Currently, we have a manual deployment of the application; we build a docker image that we push to our DockerHub account. You can find the image here.
Summery
When we started the project, we decided that part of the journey should learn more about building pipelines and learning more about CI/CD in GitLab. In this article, we have showcased the current state of our CI/CD pipeline. There is still plenty to do on the CD, but the CI looks promising.
Thank you for reading! Take care, and I hope to see you soon. 🙏🏽
This article is a living document; please reach out to me if you want to contribute or see anything inaccurate here. This article is part of our “Boosting Innovation” project.
