Writing Embedded Software Tests Fit for Continuous Integration Platforms Using Docker, Qemu, and Github Actions

tldr; Full working demo here

Note: The code blocks in this article don’t necessarily represent different files, they group common functionality together with the file’s name in comments above the code it contains. If you wish to run this locally, I would highly suggest cloning the demo’s repository to start, and running the project from that code. Some files are omitted from this article to keep the article size small.

Writing automated tests for embedded systems is often skipped, but it shouldn’t be. The Hardware Abstraction Layer (HAL) provides a nice way to not only decouple our logic from hardware but also a way to mock our hardware for the convenience of testing. This makes the tests able to run not only on a laptop but also on cloud CI platforms.

Making our environment portable

We want our tests to be contained in one tool, to make sure our environment is portable. Docker allows us to do this.

Emulating the processor

We want to make sure that the code we write can run on the processor of our embedded system, which often (almost always) differs from that on our laptop (and a cloud environment). We can use Qemu to emulate the processor.

Note: we won’t emulate our microcontroller/board exactly, since we are abstracting the hardware away.

Environment Summary

|-Docker-------------|
| |-Qemu-------|     |
| | test code  |     |
| |------------|     |
|--------------------|

Docker configuration

# Dockerfile
FROM debian:stable
# install dependencies
RUN apt-get update
RUN apt-get install -y make wget qemu gcc-arm-none-eabi python3 ninja-build gcc build-essential zlib1g-dev pkg-config libglib2.0-dev binutils-dev libboost-all-dev autoconf libtool libssl-dev libpixman-1-dev virtualenv
# install exact version of qemu (this takes a long time)
# as of writing this, v6.2.0 is not available in apt-get, 
# so I want to clone and build
WORKDIR /tmp
RUN apt-get install -y git
RUN git clone --depth 1 --recurse-submodules --branch v6.2.0 https://github.com/qemu/qemu.git
WORKDIR /tmp/qemu/build
RUN ../configure
RUN make qemu-system-arm
ENV PATH=$PATH:/tmp/qemu/build
WORKDIR /app

Startup File

We need to set up our stack pointer and branch to our program on startup, so we create an assembly file…

// startup.s
.thumb
.syntax unified
// this will be our entrypoint, 
// what instructions our processor first executes
.global ResetHandler
ResetHandler:
 // init the stack pointer to start-of-ram + 0x1000
 LDR SP, =0x20001000
 // branch to our code's entry point
 BL main
 B .

Then we need to instruct our linker to put the code in the correct locations

ENTRY(ResetHandler)
SECTIONS {
 /* put our startup file where qemu first executes code */
 . = 0x08000001;
 .startup : { startup.o(.text) }
/* put our program in flash memory */
 .text : { *(.text) }
/* set up our RAM */
 . = 0x20000000;
 __bss_start__ = .;
 .bss : { *(.bss) }
 __bss_end__ = .;
 .data : { *(.data) }
}

Our Simple HAL

To keep it simple, let’s say our HAL contains two functions. These functions turn an LED light on and off. Usually, HAL code is more complicated and is often a third-party (often a manufacturer’s) library. To keep this example small, I chose to write my own small HAL.

Note: this code has been tested and works on this Aideepen Board, if you want a tutorial on how to get a full working version on the board itself, I would suggest looking at this repository.

// HAL_LED.h
#define ADDR_GPIOC_START 0x40011000
#define GPIOC_ODR_OFFSET 3
#define LED_ON 0x00002000
#define LED_OFF 0xFFFFDFFF
void turn_led_off(void);
void turn_led_on(void);
// HAL_LED.c
uint32_t * GPIOC_start = (uint32_t *) ADDR_GPIOC_START;
void turn_led_off(void) {
  GPIOC_start[GPIOC_ODR_OFFSET] = \
    GPIOC_start[GPIOC_ODR_OFFSET] | LED_ON;
}
void turn_led_on(void) {
  GPIOC_start[GPIOC_ODR_OFFSET] = \
    GPIOC_start[GPIOC_ODR_OFFSET] & LED_OFF;
}

Our Business Logic: What we want to test

// LED.c
#include <string.h>
#include <stdio.h>
#include "HAL_LED.h"
// if we define HAL_INTERCEPT in our compilation process,
// we mock our HAL
// this is up for debate on how to intercept/mock HAL calls,
// but this is a simple way that works for demonstration purposes
#ifdef HAL_INTERCEPT
#include "HAL_LED_mock.h"
#include "HAL_LED_mock_intercepts.h"
#endif
void turn_led(char * state) {
 if (strcmp(state, "on") == 0) {
  turn_led_on();
 } else if (strcmp(state, "off") == 0) {
  turn_led_off();
 }
}

Our HAL Mocking

// HAL_LED_mock.h
#define MAX_CALLS_EXPECTED 5
#define CALL_ID_mock_turn_led_off 1
#define CALL_ID_mock_turn_led_on 2
void mock_turn_led_off(void);
void mock_turn_led_on(void);
void reset_mocks(void);
int * get_calls(void);
// HAL_LED_mock_intercepts.h
#define turn_led_on() mock_turn_led_on()
#define turn_led_off() mock_turn_led_off()
// HAL_LED_mock.c
#include <string.h>
#include <stdio.h>
#include "HAL_LED_mock.h"
static int calls [MAX_CALLS_EXPECTED] = {0};
static int call = 0;
void mock_turn_led_off(void) {
 calls[call] = CALL_ID_mock_turn_led_off;
 call++;
}
void mock_turn_led_on(void) {
 calls[call] = CALL_ID_mock_turn_led_on;
 call++;
}
void reset_mocks(void) {
 call = 0;
 memset(calls, 0, MAX_CALLS_EXPECTED * sizeof(int));
}
int * get_calls(void) {
 return calls;
}

Our tests

We can then write our tests like so. Remember: we are testing that our business-logic interacts with our HAL correctly.

// LED_test.c
#include <stdio.h>
#include "Unity/src/unity.h"
#include "LED.h"
#include "HAL_LED_mock.h"
void setUp() {
 reset_mocks();
}
void tearDown() {
}
void test_turn_led_off(void) {
 turn_led("off");
 int * calls = get_calls();
 TEST_ASSERT_EQUAL(calls[0], CALL_ID_mock_turn_led_off);
}
void test_turn_led_on(void) {
 turn_led("on");
 int * calls = get_calls();
 TEST_ASSERT_EQUAL(calls[0], CALL_ID_mock_turn_led_on);
}

Our test entrypoint…

#include <stdlib.h>
#include "LED_test.h"
#include "Unity/src/unity.h"
#include "output.h"
int main() {
 UnityBegin("LED_test.c");
 RUN_TEST(test_turn_led_off);
 RUN_TEST(test_turn_led_on);
 return UnityEnd();
}
// in order to print to stdout, 
// qemu forwards the first serial port to stdout, so we
// need to send data to that port's data register
void print_something(char s) {
 unsigned int * first_serial = (unsigned int *)0x40013804;
  *first_serial = (unsigned int)(s);
}

Running our tests

We can then run our tests in Docker using make. We should call make docker-build-and-test . You should be able to run this command on any linux or mac machine (since it uses shell pwd, it requires the terminal to have pwd in its PATH) and it will run the tests.

This will call a small python script to run and read our tests’ output, and exit accordingly. This is done because the tests run inside of a virtual machine, and it is difficult to get the exit code from the process. We will base tests’ pass/failure status on their text output.

# evaluate_tests.py
import re
import subprocess
import sys
if __name__ == '__main__':
    try:
        # run the actual tests, 
        # this assumes the tests will run in less than 5 seconds.
        # as the number of tests grows, 
        # the timeout will need to be increased
        # this is done this way because the tests run in a VM, 
        # so they are hard to get the
        # exit code from, so we timeout and read the logs
        # to find the results
        subprocess.run(
            ['make', 'run-arm', '>', 'test_log.txt'], timeout=5)
    except subprocess.TimeoutExpired as e:
        # we are expecting the tests to timeout, 
        # ignore this error and get the status from the logs
        pass
    with open('test_log.txt') as test_log:
        logs = test_log.read();
        print(logs)
        tests_results = re.search(
           r'([0-9]*) Tests ([0-9]*) Failures ([0-9]*) Ignored'
        , logs)
        if tests_results is None:
            print('could not find test results line')
            sys.exit(1)
        failures = tests_results.group(2)
        print('found {} failures'.format(failures))
        if (failures == '0'):
            sys.exit(0)
        else:
            sys.exit(1)

Our Makefile is found below. This compiles for/runs the tests on a cortex-m3 processor.

assemble-arm:
 arm-none-eabi-as -mcpu=cortex-m3 startup.s -g -o  startup.o
compile-arm : assemble-arm
 arm-none-eabi-gcc \
  -D HAL_INTERCEPT \
  -Tcortex-m3-tests.ld \
  -mcpu=cortex-m3 \
  -mthumb \
  -include output.h \
  -I Unity \
  -I . \
  startup.o \
  Unity/src/*.c cortex-m3/*.c *.c \
  -g -o led_test.elf
run-arm: compile-arm
 rm -f test_log.txt
 qemu-system-arm \
 -device loader,addr=0x08000001,cpu-num=0 \
 -machine stm32vldiscovery -cpu cortex-m3 -nographic -kernel led_test.elf > test_log.txt
docker-build:
 docker build -t hal_ci_example .
docker-test:
 docker run -v $(shell pwd):/app hal_ci_example:latest python3 evaluate_tests.py
docker-build-and-test: docker-build docker-test

We can set these up to run in Github actions like so

name: HAL CI Example
on: [push]
jobs:
  Run-Tests:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository code
        uses: actions/checkout@v2
      - run: git submodule update --init
      - run: make docker-build-and-test

At the end of the tests you should see output like this

LED_test.c:8:test_turn_led_off:PASS
LED_test.c:9:test_turn_led_on:PASS

-----------------------
2 Tests 0 Failures 0 Ignored 
OK

found 0 failures

You can see a full working demo here.