Mainflux 0.11 — Digital Twin, MQTT Proxy And More
New release of Mainflux brings important features — like Digital Twin, Zero-Touch Provisioning and MQTT Proxy — alongside with several bugfixes and improved documentation and tests
We at Mainflux Labs are building one of the world’s most sophisticated IoT system — an Unified Cloud-Native IoT Platfom. Open-sourced under Apache-2.0 license and published on our GitHub.
Mainflux marks a new milestone: 0.11 version has been released. Six months after the previous release, a lot of work has been carefully packaged into a release that is already used in several commercial products.
This release brings several important improvements to the Mainflux system. Most prominent new features are:
- Digital Twin
- mProxy — an MQTT Proxy
- Zero-Touch Provisioning
- Edge (Agent and Export)
Apart from this, there is a whole list of fixed bugs and other improvements like:
- Enhanced documentation
- Added Kubernetes Helm charts in DevOps repo
- Published detailed system benchmark, based on distributed MzBench testing platform, in a dedicated Benchmark repo
Digital Twin
Digital Twin refers to a digital representation of a real-world data system consisting of possibly multiple data sources/producers and/or destinations/consumers (data agents).
For example, an industrial machine can use multiple protocols such as MQTT, OPC-UA, a regularly updated machine hosted CSV file, etc. to send measurement data — such as flowrate, material temperature, etc. — and state metadata — such as engine and chassis temperature, engine rotations per seconds, the identity of the current human operator, etc. — as well as to receive control, i.e. actuation messages — such as, turn on/off light, increment/decrement borer speed, change blades, etc.
A digital twin is an abstract — and usually less detailed — a digital replica of a real-world system such as the industrial machine we have just described. It is used to create and store information about system’s state at any given moment, to compare system state over a given period of time — so-called diffs or deltas — as well as to control agents composing the system.
You use an HTTP client to communicate with the twins service. Every request sent to the twins service is authenticated by Users service. Twins service handles CRUD requests and creates, retrieves, updates and deletes twins. The CRUD operations depend on the database to persist and fetch already saved twins.
Twins service listens to NATS server and intercepts messages passing via NATS broker. Every Mainflux message contains information about subchannel and topic used to send a message. Twins service compares this info with attribute definitions of twins persisted in the database, fetches the corresponding twins and updates their respective states.
More details on Mainflux Digital Twins can be found in the official documentation.
MQTT Proxy
In order to decouple MQTT AuthN and AuthZ operation form underlying MQTT broker, we implemented MQTT proxy called mProxy. This is very useful library which allows everybody to use off-the-shelf MQTT broker while keeping all custom logic localized in a separate module.
MProxy is deployed in front of MQTT broker and can be used for authorization, packet inspection and modification, logging and debugging and various other purposes.
MProxy starts TCP and WS servers, offering connections to devices. Upon the connection, it establishes a session with a remote MQTT broker. It then pipes packets from devices to MQTT broker, inspecting or modifying them as they flow through proxy.
With implementation of all custom logic in Go, Mainflux is now capable to use any of the known and powerful open-source MQTT brokers (like VerneMQ, Aedes or Mosquitto) or commercial ones as well.
More information about mProxy can be found on the dedicated GitHub repo pages.
Zero-Touch Provisioning
In order to facilitate and automate provisioning process — especially provisioning of IoT gateways and edge devices — we have created and published Mainflux Provisioning service. Provision service should provide an easy way of provisioning your gateways i.e. creating bootstrap configuration and as many things and channels that your setup requires.
Additionally, provision service might be used for creation of X.509 certificates for each Mainflux Thing. Each service running on gateway may require more than one thing and channel for communication. Let’s say that you are using services Agent and Export on a gateway you will need two channels for Agent (data and control) and one for Export and one thing. Additionally if you enabled mTLS each service will need its own thing and certificate for access to Mainflux. Your setup could require any number of things and channels this kind of setup we can call provision layout.
Provision service provides a way of specifying this provision layout and creating a setup according to that layout by serving requests on /mappingendpoint. Provision layout is configured in config.toml.
Additional info can be found on dedicated documentation pages.
Mainflux Edge
In the pursuit of an ultimate goal of Unified IoT Platform (Edge — Fog — Cloud continuum), Mainflux develops further Edge part of the system.
Two services are crucial for connecting remote IoT gateways, Edge computes and remote Mainflux clouds:
- Agent — which assures secure and encrypted Control Plane
- Export — which is used for Data Plane, data export to remote clouds while tolerating intermittent connections with temporary local buffers
This release solidifies both Agent and Export services, enhances and documents protocols and adds additional tests.
More information about Mainflux Edge can be found on dedicated documentation pages.
Kubernetes
Mainflux published all Kubernetes scripts in a dedicated DevOps GitHub repo. Additionally, Helm charts have been added to further automate Kubernetes cluster deployment.
The solution was tested and documented in the official documentation.
Benchmark
Mainflux team found MzBench as a perfect tool for distributed (thanks to Erlang/OTP) stress testing.
MZBench runs test scenarios on many machines simultaneously, maintaining millions of connections. This makes it suitable even for large scale products (we’re talking Facebook large scale here).
Following the instructions found on the VerneMQ blog, Mainflux team developed a set of test-scenarios for the stress-testing Mainflux system. Tests are executed in a distributed fashion: a lot of Erlang processes are deployed on dedicated VMs in Docker containers and they can simulate thousands of MQTT devices sending messages.
Tests are executed in a realistic environment and with the highest and most demanding QoS (QoS 2, that guarantees that each message is received exactly once by the intended recipients by using a four-part handshake), in order to obtain fair and reliable results. Special attention is taken to assure that no message loss is permitted.
All tests, alongside results, have been published in a dedicated GitHub repo. In-depth documentation can be found here.
Future Steps
Mainflux is already a mature IoT system — started in 2015, it celebrated the 5th year anniversary.
However, Mainflux is also very ambitious cloud and edge system, and before releasing the long-term support 1.0 version, Mainflux team would like to add at least 2 big features to the system:
- Policy-based Access Control, which is defined in #931
- E2E aka “Black-Box” Testing, which is defined in #643
While having e2e testing is important for automated testing of the API contracts and should ensure that regression does not happen and thus facilitate long-term support of v1.0, policy-based access control is a different beast. It is important to ensure security and add fine-grained AC in the form of user, device and channel groups with organizations and admins. This feature might influence API changes and because of its importance is left for 0.12 release, last one before 1.0.
The next release (v0.12) is planned for the end of August, while 1.0 is planned before the end of year.
As a conclusion, it is really worth saying that 0.11 is one of the most complete Mainflux releases, showing the maturity of the platform. It also paves the way for implementing the Unified IoT Platform system, where both Edge and Cloud can share the same architecture. It facilitates the Cloud-Native setup of the system and further secures MQTT communication while isolating mProxy as a general-use open-source project. Digital Twin is probably the biggest addition in this release and will be further developed and improved in the future.
With 0.11, Mainflux becomes a stable feature-rich industrial system. Only a few additional features are left before the famous 1.0, which is expected by the end of the year.
To learn more about Mainflux and how we’re building the world’s most sophisticated open-source Industrial IoT cloud, visit our website and follow us on Twitter!
