Accelerate IoT and Embedded Systems Development by Using Simulate Physical Hardware Systems

Developing for hardware — the need for testing FW over HW units

Developing firmware — what does it mean? Well, firmware is a low-level control for a device’s specific hardware — most of you are probably aware of that strong bond between the firmware and the hardware, one is meaningless without the other.

Given that, when it comes to testing a firmware release, it’s obvious the testing must run over the targeted hardware — is that so?

Individual software components — within the entire firmware application — can be tested simply by maintaining tests & running unit tests. Even if the tested code cover is relatively high (for embedded applications), you still have to worry about the functionality test of the actual device — how will it behave running over the actual hardware?

Considering a development model with a short release cycle of new features, improvements, bug fixes, etc, the physical functionality test over hardware can drastically slow down the velocity of the developments.

At Augury, we are in the process of adopting a unique, non-traditional method of testing firmware over the actual hardware target. Not necessarily using the actual hardware, but — when possible — using an emulated hardware. The simulation can go beyond the CPU, we can simulate an entire board and even simulate the wireless interface between a set of devices — an entire IoT network can be simulated for the purpose of a single firmware release test.

Difficulties over scale — maintaining portfolio of HW devices while continuously developing FW

When a company runs an entire portfolio of hardware devices, they sometimes run the same firmware application — with minor adaptations for the target device.

When speaking of IoT, a standard portfolio is usually structured by end devices and gateways. Considering a portfolio of 4 end device types and 4 gateway types, testing a single firmware release (either an end device or a gateway) will require testing over 16 different hardware permutations! That is crazy in the world of continuous integration & delivery.

Performing all these tests over physical hardware requires physical hardware — and quite a lot of it!

In Augury, we have several sets of units from each type of hardware, they are stored in trays & shelves and allocated dynamically to each developer/test.

The Hold & Release method we use is very straightforward — but needs to be handled with care.

Before each test scenario, the developer holds the required sets of devices — they get registered under the developer’s username and become unavailable to others.

When the test is completed, the sets are released and go back to the pool.

That system — if handled with care — can be a relatively comfortable platform for a group of developers, yet it’s not built for scale and can easily go off track considering all the required maintenance & care.

Scale, scale & scale

Having a group of developers Hold & Release sets of devices will probably work, but will add a huge effort to the developments and eventually decrease the cadence & quality of the releases.

Hardware emulation is the answer — we can have all this hardware, emulated, as a piece of code that simulates the entire behavior of the real hardware device.

The processor & board that the firmware runs on, and even an entire wireless IoT network of devices, can be emulated and used with as many instances & variations as we can possibly think of.

Hardware emulation

Our developers in Augury use the Renode platform, and collaborate with the Renode project (by Antmicro) developers, in order to continuously expand the simulated parts up until reaching a full network of wireless end devices & gateways.

The Renode project has a vast amount of simulated CPUs & Boards (mostly eval boards) — but they offer the ability to develop — over the platform — an emulator for your own board with the exact peripherals and external interfaces wire or wireless.

We have allocated time & effort and stepped up to that. By successfully running a POC of an emulated hardware — structure of an nRF52x SOC and a digital accelerometer sensor, connected via SPI — just like it’s implemented over the physical hardware.

Going in this direction, we have opened the door to full emulated devices, and we can now imagine a world where we can test firmware releases without any hardware in the loop.

Wrapping up

Emulating hardware — not just a processor — or a device — but an entire network is not effortless, but thinking about scale & quality is the only viable solution. Augury is gradually progressing towards that by developing the platform — connecting a Jenkins server running test scenario with a hardware server, taking the place of the target units under test.

As hardware developers we understand that it might not close the loop 100%, meaning that we will have to cherry-pick the most critical tests that must be run over real metal hardware — tests like current consumption or particular analog behavior — but the very most can be avoided.

Perhaps we will not be able to eliminate the use of hardware altogether, but we will most likely improve & enable the scale growth and quality that we’re looking for — becoming the category leaders.