meta-iota — IOTA Ecosystem Development Fund — Milestone 1 Report

I’m glad to announce that my OpenEmbedded layer for IOTA projects — meta-iota — was recently selected for the IOTA Foundation’s Ecosystem Development Fund.

You can read an introduction to the project on my first post: IOTA cIRI on a BeagleBone Black with Yocto Project and OpenEmbedded. There, I explain what the Yocto Project and OpenEmbedded are, and explain a bit the context behind the idea of creating the meta-iota OpenEmbedded layer. Here’s a video summarizing what the Yocto Project is:

In a nutshell, my goal is to allow the integration of IOTA projects into Yocto-based Embedded Linux Distributions. Since the Yocto Project and OpenEmbedded are focused on IoT devices (as the video above brilliantly illustrates), it makes total sense to bring IOTA into this universe and connect that HDTV-BluRay-MicroWave-WashingMachine to the Tangle 🤓

For the first milestone I decided I would port three projects from the enTangled repository:

• cIRI
• CClient
• MAM

This article is meant to serve as a report of Milestone 1 of the development of meta-iota. I will describe in detail what was done for the integration of each of the three projects listed above, as well as challenges and opportunities faced during this period.

cIRI

cIRI is the low level implementation of an IOTA node in C, designed to provide the same functionality as the IOTA Reference Implementation (IRI). It allows users to become part of the IOTA network by acting as a transaction relay and a network information provider via an HTTP interface.

Because cIRI has much lower hardware requirements when compared to its Java old brother, it is well suited for many of the Embedded Systems supported by the Yocto Project. That’s one of the reasons I chose it to be my first project on meta-iota. I reported that work in my first article IOTA cIRI on a BeagleBone Black with Yocto Project and OpenEmbedded. Although this article is based on a BeagleBone Black, it should work for any other boards with a Yocto/OpenEmbedded Board Support Package (BSP). If you need help applying the tutorial to other boards, feel free to write a comment below, DM me on Discord (Bernardo [IF]#8478) or send me an email (bernardo.araujo@iota.org).

Unfortunately, with the upcoming Coordicide, it is likely that cIRI won’t stick around for a long time. At some point of 2020 or 2021, the network will eventually migrate to post-Coordicide node implementation bee. We have to wait and see how this project evolves, but I must admit I look forward to throwing bee into the meta-iota mix 🤓

The enTangled team chose Bazel as the cIRI build system. I found out that Mr. Hongxu Jia from WindRiver has already done an amazing job of porting Bazel into YP/OE on his meta-tensorflow layer. Nevertheless, a few adaptations were introduced into the cIRI recipe so the Bazel workspace would integrate well with BitBake and also so that snapshot, sql and config files were populated into the final package.

So let’s talk about the BitBake recipe that made this possible. You can find the recipe at recipes-iota/ciri/ciri_0.1.0.bb on the master branch of the meta-iota repository.

The SRC_URI variable lists the enTangled repository, as well as a list of patches and miscellaneous files that are meant to allow Bazel to behave correctly inside BitBake. While the patches are handled automatically by BitBake’s do_patch() task, the miscellaneous files need to be manipulated explicitly inside the do_configure_append() task to prepare the build environment so Bazel and BitBake work together correctly.

The do_compile() task is responsible for invoking Bazel with the correct parameters. Next, the do_install() does the job of populating the target root filesystem with the cIRI executable binary, as well as snapshot files, sqlite3 schemas and the conf.yml configuration template file.

Since the Bazel adaptation inside BitBake requires many workarounds, I hope the enTangled team considers porting cIRI to CMake as well. If that day ever comes, I want to adapt the cIRI recipe to use CMake instead of Bazel.

CClient

IOTA Clients create operations (transactions) to send data and/or IOTA tokens to each other’s addresses.​ The IOTA client libraries abstract the complexity of the IOTA protocol into simple methods that allow developers to create, send, receive, and request transactions from client devices.​

As detailed in the IF Research & Development Roadmap:

Most existing client libraries are written in higher-level languages unsuitable for embedded devices. Since IOTA is designed for the IoT, it will ultimately be necessary to support small, low-powered, embedded devices, and C is the ‘lowest common denominator’ programming language for many of these potential applications.

The IOTA client library implementation in the C programming language is called CClient.

Although CClient is also hosted on the enTangled repository and originally built with Bazel, the enTangled team (thanks Sam Chen!) has written a CMakeLists.txt file that allowed me to avoid Bazel and go straight to the CMake approach when I wrote CClient’s BitBake recipe. You can find the recipe at recipes-iota/cclient/libcclient_1.0.0.bb . It is relatively short and straightforward, with only one patch to avoid the usage of CMake’s ExternalProject feature, which doesn’t work well inside BitBake. The recipe’s final products are mainly libcclient.a and libcommon.a, which are exported into the /usr/lib path of the target rootfs.

As an example and Proof of Concept, I also wrote one recipe to demonstrate the usage of the CClient library for actual client applications written in C. For that goal, no better candidate than the c-iota-workshop repository, which contains a few examples of how to interact with the Tangle as a client. You can find the c-iota-workshop recipe at the recipes-iota/cclient/c-iota-workshop_git.bb path of the meta-iota layer. The do_compile() task performs the compilation of each example by “manually” invoking the Cross-Compiler (${CC}). It is important to note that because I avoided CMake’s ExternalProject, the symbols from the CClient dependency libraries (keccak, uthash, mbedtls, logger, http-parser, and cjson) are not shipped into the libcclient.a and libcommon.a libs, so they have to be linked explicitly on the ${CC} command. Also worth noting how I named each example executable under the template iota_c_*, so they’re easier to identify from the target’s shell.

The article OpenEmbedded, IOTA CClient and a Raspberry Pi shows a Proof of Concept of how to use meta-iota to cross-compile C applications to interact with the Tangle via CClient.

MAM

Masked Authenticated Messaging (MAM) is the result of a research collaboration between the IOTA Foundation and the Belarusian State University (BSU).

In a nutshell, MAM acts as a second layer data communication protocol that adds functionality to send and receive encrypted data streams over the Tangle. MAM aims at the important need for integrity and privacy in the IoT industry because how the IOTA consensus protocol encapsulates those message streams. For more in-depth explanations of MAM, please refer to:

• Introducing Masked Authenticated Messaging, by Paul Handy
• IOTA: MAM Eloquently Explained, by ABmushi
• IOTA tutorial 19: Masked Authenticated Messaging, by Mobilefish
• MAM Update: C Libraries and Release, by Aaron Alsop
• How is authenthicity and confidentiality maintained for MAM channels on the IOTA Tangle? by Marius Ødegard

Just like CClient, I was able to avoid the Bazel highway and went for a CMake-based approach. For that I have to thank my hermanos Daniel De Michele (Carpincho Dem) and Alejandro Elustondo (from XDK2MAM and IOT2TANGLE) for their awesome work on writing a CMakeLists.txt for MAM.

The MAM recipe can be found at recipes-iota/mam/libmam_0.1.0.bb on the master branch of the meta-iota layer. A few tarballs are used to populate some source code that was originally written dynamically by Bazel at compile time. The do_configure_prepend() task is responsible for placing this code on the build directory (${S}) . Again, one patch is used to avoid the usage of CMake’s ExternalProject feature, and another patch is used to fix a definition of flex trit conversion in the source code. The recipe exports libmam.a and libmam-api.a to the target rootfs.

Just like the CClient library, I felt the need to write a recipe that illustrates a practical usage example of MAM. So I borrowed the examples originally located at the mam/examples path of the enTangled repository and created a new entangled-mam-examples repo. Then, I wrote the recipes-iota/mam/entangled-mam-examples_git.bb recipe. I named each example executable under the template iota_mam_*, so they’re easier to identify from the target’s shell.

The article IOTA Masked Authenticated Messaging on OpenSTLinux of STMicroelectronics Discovery Kit presented a Proof of Concept of how meta-iota can help cross-compile MAM applications into Yocto-based Linux Distributions. In that case, the Distribution was OpenSTLinux, which added another cornerstone into the already existing collaboration between the IOTA Foundation and STMicroelectronics.

Finally, it is important to notice one important caveat. As detailed in the IF Research & Development Roadmap:

While MAM already exists, it is missing a few key features and documentation. A formal specification and improved version of MAM leveraging public-key encryption is currently being written by the experts from the Research Institute for Applied Problems of Mathematics and Informatics at Belarusian State University.

So we should expect a new MAM+ implementation in the future, which I look forward to integrating into meta-iota as well!

Milestone 2

For Milestone 2, I plan to integrate the Go and Python IOTA API Libraries, as well as the go-iota-workshop and python-iota-workshop repositories as usage examples of those libs.

This will allow the integration of client applications of both languages into Yocto-based Embedded Linux Distributions.