EdgeX + Balena Walkthrough Tutorial

In this little tutorial, we are going to setup ΕdgeX on Raspberry pi, using balena as the device management platform of choice.

You will read about:

  1. What is Balena
  2. What is Edgex
  3. Why I chose Balena
  4. How to install Balena on Raspberry pi
  5. How to configure EdgeX using Balena
  6. Known problems, future contribution.

📍What is balena

As mentioned in their site:

The core balena platform, or what we call balenaCloud, encompasses device, server, and client-side software, all designed to get your code securely deployed to a fleet of devices. The broad strokes are easy to grasp: once your device is set up with our host OS ( balenaOS), you can push code to the balena build servers, where it will be packaged into containers and delivered to your fleet.

In essence it has a whole range of tools in order to efficiently provision, ship and manage different IoT devices using their own open source OS as well their docker-compatible container engine ( balenaEngine) which is optimized for the resource constraint IoT.

📍What is EdgeX

EdgeX is an Open Source IoT Platform, hosted by the Linux Foundation and spearheaded by Dell, amongst other key stakeholders.

As mentioned in their site:

Vendor-neutral, open source, loosely-coupled microservices framework providing you the choice to plug and play from a growing ecosystem of available 3rd party offerings or augment with your own proprietary innovations. With a focus on the IoT Edge, EdgeX simplifies the process to Design, Develop and Deploy solutions across industrial, enterprise, and consumer applications.

The development started in 2017 when Dell released the first versions of the PoC micro-services and is now under active development by numerous developers. Moreover, on their slack channel there are developers from various companies that actively support users the dwell into it. Version 1.0 (Edinburgh) is scheduled to be released in June 20 2019 and the main implementation language is Golang.

I choose EdgeX as the main IoT component of my solution not only because it’s a stellar Open Source project but also because it is ridiculously customisable and under active and heavy development, thus at the moment any contribution can make a huge impact.

You can find the main github repository here.

📍Why I chose Balena

Although I have no affiliation with the Balena project or the team, the choice was pretty obvious for me. Firstly, I wanted to stress-test EdgeX on a relatively constraint Edge device (such as the Raspberry Pi) and I wanted to do so in a scalable manner. Moreover, the Balena Dashboard made it a breeze to build and ship containerized services, taking care everything below the docker compose layer. Thus, I could focus on finding the right combination of images & services and speed-up the bugfixing process.

It is worth mentioning, that when using balena, you don’t have to build the images on your local dev machine or on the device. You just specify the architecture and the images are built on the balena servers for that specific device, finally they are shipped and bootstrapped automatically. When building an application, you can add many different devices that use the same architecture. balenaCloud will build the application for that specific arch and will ship it to each different device, taking account the uniqueness of each device.

The dashboard can be shown below, we will go into details as we continue the walkthrough.

📍How to install Balena on the Raspberry Pi 3

You can use any hardware platform with the armv8 architecture, in order to support 64bit OS. This is currently needed for the edgex-mongo micro-service and it's RAM requirements. With the Edinburgh release that will natively also support Redis, EdgeX will be runnable on 32bit systems as well.

To install Balena on your device and learn about how it works, check this detailed walkthrough while ignoring any reference to screen and hw, focus on the process and the software. Note that when creating your application, you will choose BalenaOS64 (BETA) for Raspberry pi.

After completing the above walkthrough and feeling more comfortable with balena, you can proceed here and read the documentation regarding developing & deploying with balena.

It is important to understand how balena works before proceeding further as we will be combining 2 distinct platforms (edgex-balena) and we will need to understand how they co-exist in order to speed-up development and bugfixing. If you haven’t already, read about balenaOS for the same reason.

📍EdgeX Deployment

When shipping software, balena supports both building your application logic in one container and multiple containers. Here, we can either build the whole EdgeX-monorepo in one container using a dockerfile, build each service individually into a different container using dockerfile and docker-compose or simply use the docker-compose file which can be found in the developer-scripts github repository (since the PR is not yet approved, you will find it here instead).

At this point, basic understanding of docker compose and dockerfile is deemed necessary, go back at the balena docs and verify.

We will be using the simplest deployment method, the docker compose. In essence, repeat the balena tutorial above, but replace the docker-compose.yml with the EdgeX one. You don’t need any other files (like a dockerfile), since the compose doesn’t build any image, but uses the ones that are uploaded to hub.docker. Balena will upload the compose to the build servers, pull the images and then download/orchestrate them in the device(s).

The file mentioned earlier was based on the docker-compose-delhi-0.7.1.yml but with certain changes:

  1. The architecture we use is NOT the default (x86). We will be pulling the arm64 version of the EdgeX images that can be found in the EdgeX hub.docker.
  2. balenaEngine doesn’t comfort to docker-compose version 3 but to a subset of version 2.
  3. docker-support-rulesengine even in the arm64 version produces: standard_init_linux.go:195: exec user process caused "exec format error" which means that in fact is NOT built for arm64 arch. Removed
  4. docker-device-virtual is built on Java and produces excess stress on the system due to unknown reasons. Guilty as charged. Removed
  5. Container names are not supported by balenaEngine. Removed
  6. There is currently a bug in the networking stack of balenaOS. Neither the networks setup works, neither network-mode. Since we are developing, we will use the default docker bridge. Thus, we don't mention anything about networks in the docker compose.
  7. Due to (6), hostname doesn't work either. The services are accessible using only the service name as defined in the docker compose. Since EdgeX services are hard-coded to look for certain hostnames (that are defined in the original delhi compose file), we rename the service name to match the original hostname. The services are now discoverable, bypassing the bug.
  8. portainer is not needed as the orchestration & management will be handled by the balena platform. Removed
  9. device-random is incorporated as the testing device service in lieu of the device-virtual.

By now you should understand the basics of balena, how it works and what modification must be done to EdgeX in order to play ball with balena. You should have a system that is up and running.

Let’s manage it!

📍Device Management

balena offers many features to manage each application and their corresponding devices. I will leave the exploration to each reader as the resources offered by balena are more than enough. We will focus on more EdgeX specific subjects.

In the dashboard image above, you can see an overview of the system, let’s see some interesting details.

  1. Logs can be configured to filter for certain keywords or produce results from a single service. It is super convenient to see all the logs in order to see patterns of bugs thattranscendnsend a single service.
  2. Each service can be run, stop or reset individually. This can throw the system into disarray, but it should stabilise itself. After each release, it takes some time for the balena server to inform the device, download the new release and restart all the services. Be patient as some services might take more time than others.
  3. Time online is not the same as uptime, it is mere for how long it has been connected to balena servers.
  4. It is possible to ssh into each service individually as also ssh into the hostOS. Note that depending on the service (and it's corresponding dockerfile) the container can even lack a shell. hostOS is also barebones, meaning that common commands like curl are not installed, while others are (wget).
  5. You can set device variables and complex device configuration as illustrated in the balena docs.
  6. Diagnostics Experimental is superbly useful to have an overview over RAM usage, real uptime, etc. Since Raspberry pi 3 is somewhat RAM contrant (1GB), the device could restart due to lack of memory. Diagnostics will help to figure out what has room for optimisation.

👉Note You can access both the logs, ssh and other functionality using the balena CLI, depending on the use-case, choose your tools accordingly.

📍Known Problems, Future Contribution

That’s it! You now have the gist on how to start experimenting on the two platforms, this is only the beginning. Real understanding will come from tinkering with the platforms, using tools provided by either community. Like EdgeX blackbox testing and EdgeX developer scripts. Also check the forums: balena and EdgeX.

  • MongoDB: It is way too heavy for Raspberry pi 3. The whole platform with only device-random leaves about 20–30% RAM free with potential spikes that can force a reboot.
  • RedisDB: The current release can be monkey-patched to work with Redis (in-memory DB), not tested yet to see performance and capability.
  • Scheduler: Not sure if working as expected, written in java and mostly hard-coded meaning that is not easily configurable.
  • Networking: System is not isolated (due to the balena bug), also gateway-service is not yet tested and no firewall is installed, thus every service's API is exposed. Highly unlikely for a true production environment.
  • Overall performance: It is not certain how much room there is for optimisation so as an EdgeX powered Raspberry pi 3 can offer viable functionality for even a light Edge Computing device. Note that to offer true functionality, the device should also have a number of device-services and maybe supporting ones (like a kind of rules-engine).
  • Distributed systems: A solution would be to create a cluster of Rpi3 that work as a unit, distributing the application logic over multiple locally connected devices. The use of a central consul registry can simplify the deployment of said system.

🚀I will continue to explore this interconnection of the platforms in order to produce an Edge-Device to function as a LoRa gateway & server as also incorporate a number of features, such as automatic hand-over, dynamic confederation creation between Edge devices and even filecoin. This project is part of my Diploma Thesis( Electrical Engineering), but I love to share and collaborate.

I will post continuous updates on this front to serve as a guideline for others that want to try and test. This post will be posted in both the EdgeX forums and Balena Forums.

🎯Please, do post any updates, corrections, improvements or comments regarding this subject. The idea is to help each other and boost the productivity 🔥 of everyone in the process.

Untill the next update, cheers 🤟