MQTT in a Nutshell

What is MQTT?

According to the official website …

MQTT (Message Queuing Telemetry Transport) is an OASIS standard messaging protocol for the Internet of Things (IoT).

OASIS standard means, that the specification of the protocol has been created and is evolving involving individuals, organizations and governments in an open, non-prioritary manner.

Why should I care about MQTT?

MQTT is the de-facto standard for inter-device communication in a lot of domains like

• automotive (e.g. BMW Carsharing Platform, SAIC Volkswagen IoV platform),
• logistics (e.g. real-time monitoring of Matternet autonomous drones),
• manufacturing (e.g. power plan monitoring, various SCADA related use cases),
• smart home (e.g. home energy monitoring),
• healthcare (e.g. cardiac patient care system),
• consumer products (e.g. smart kitchen appliance) and
• railway (e.g. for various use cases in the DB Railway System).

Of course MQTT is not the right technology for every use case. However the chances are high that you’ll stumble over this technology at some point in time during your career. It will be even more likely in the future that you will have to understand the technology. Hopefully you’ll understand why after having read this post to the end.

About the history of MQTT.

According to wikipedia

… MQTT was developed by Andy Stanford-Clark (IBM) and Arlen Nipper (then working for Eurotech, Inc.) in 1999 to monitor oil pipelines within the SCADA framework.

SCADA stands for Supervisory control and data acquisition and is a control system architecture for high-level process supervisory management in the industrial domain.

Today Andy Stanford-Clark is CTO of IBM UK. Arlen Nipper is the President and CTO of Cirrus Link, a company providing integration solutions for the Industrial IoT.

It’s important to understand the context of MQTT w.r.t. the factory automation pyramid.

MQTT was created for optimizing the communication of data in the supervisor level (SCADA). Key takaway: The origin of MQTT lies in the factory automation domain. To be honest, this was surprising to me. I expected MQTT be be rooted in home automation or some other low risk domain. Many times in the field and control level there is need for hard-real time capable communication. This means in such a system a single missed deadline in the communincation of data leads to a major failure in the overall system. There are many protocols in the field level, common protocols are e.g. EtherNet/IP. IO-Link and CC-Link are comparably new protocols with a lot more capabilities w.r.t. field device management (e.g. firmware update management). In some cases there is need for functional safety-compliant communication in addition e.g. provided by Safety over EtherCAT.

In the SCADA level however there is no need for hard real-time capable communication. The well known “real-time” capable communication is sufficient. I’ve designed and implemented protocol inverters for field level protocols in one of my past projects. Given my personal background in my opinion the buzz word “real-time” is horrible, in fact people mean either firm real-time or soft real-time capable data communication. They almost never mean hard real-time capable communication. This is very important! Key takeaway: MQTT is firm real-time or soft real-time capable but achieving and proving hard real-time capability with MQTT is hard to impossible. This is not neccessarily bad, but should restrict the use cases for which MQTT should be used.

In the field level as well as in the SCADA level there have historically been vendor specific systems like e.g. the well known Siemens SIMATIC. Key takeaway: Historically many PLC and SCADA level protocols have been vendor specific. MQTT introduced an open standard in the SCADA level.

Reasons for why MQTT is turbocharging in the IIoT domain today.

The requirements in the industrial domain are shifting.

The requirements for timeliness of sensor data in the field level seems to change from hard real-time to firm/soft real-time capability for many use cases. This implies that the field level protocols become obsolete and field devices (actuators/sensors) can be connected using MQTT. This manifests in the fact, that there are more and more industrial gateways available which convert field level protocols to MQTT.

Some field level subsystems still require hard real-time capable control systems. However many SCADA systems provide integrations which allow to publish their data on MQTT like e.g. the SIMATIC S7–1500. Field level requirements for hard real-time capable communication and control does not affect MQTT adoption.

The most important fact: “Semantically summarized” data adds value. The effectiveness of single factories and interconnected factories depend on data. All the technolgies like e.g. digital twins (“digital copies of field level actuators and sensors as well as factory machines”), digital factories (“digital copies of factories”), etc. depend on data. However the amount of field level data is huge. E.g. dependent on the field level technology in use every single sensor can produce roughly as much as between 10–1000 single “data values” per second. Passing all these values to the SCADA level would not make sense of course. It’s important to summarize the data to data which has semantic relevance. The SCADA level does not care about data relevant for field level control. It’s sufficient to communicate to the SCADA system e.g. that there is a sensor XYZ, in factory XYZ, production line XYZ which reports abnormal sensor measurement values. This information is relevant for e.g. factory maintenance and product flow (if the production line has to be shut down for some time the upstream material flow will b blocked). We will see that MQTT in combination with the Sparkplug specification is a powerful technology set to achieve “sematically enriched” data in the SCADA level.

MQTT plays well with other upcoming IIoT technologies. A widely used industrial standard, OPC UA is adopting communication based on MQTT. OPC UA enables the use of field level protocols with functional safety relevance as well as MQTT for “low risk” data communication.

Reasons for why MQTT is getting adopted in other domains.

Many of the advantages of MQTT + Sparkplug like e.g. enabling an “event based” instead of a continuous data transmission paradigm and device status management is pushing the adoption of MQTT in other domains. The Sparkplug specification is beyond the scope of this post. I’m planning to write another post about Sparkplug. For now feel free to have a look at the great resources MQTT Sparkplug Essentials Part 1 — Introduction, MQTT Sparkplug Essentials Part 2 — Architecture and New White Paper and Webinar about MQTT Sparkplug.

How does MQTT work?

In summary every system involving MQTT commununication depends on at least one MQTT broker. The MQTT broker acts as forwarder for messages between MQTT clients. Every MQTT client has to connect to the MQTT broker. I don’t go into detail how the connection procedure exactly works. But you’ll find references below to learn about this and other details about MQTT. Usually the communication is implemented in a “secure” manner via X509 certificates and MQTT over TLS.

MQTT clients publish data via messages on topics which act as channels. MQTT clients can subscribe to topics provided by the MQTT broker. Every message which is published by a MQTT client on a topic is received by the MQTT broker. The exact behaviour of the MQTT broker depends on the configuration of MQTT. Independent of the configuration the MQTT broker stores every received message into a buffer, checks which clients have subscribed to the topic, transmits the message one or several times (dependent on the configuration) to every subscribing client with or without waiting for an ackowledgement for message reception (again, dependent on the configuration), removes the transmitted message from the buffer and waits for a new message. MQTT brokers are capable of handling a lot of message forwarding procedures in parallel of course.

Over time there have been released different versions of the MQTT protocol. The most used version at the moment is probably MQTT v3.1.1. MQTT v5.0 beeing the latest release. MQTT v5.0 offers significant improvements over MQTT v3.1.1. Key takeaway: There are different versions of the MQTT protocol with support of less or more features.

As a spoiler it’s worth to note that Sparkplug depends on the full feature set of MQTT v3.1.1. Key takeaway: Sparkplug depends the full MQTT v3.1.1 feature set.

To understand how MQTT works in more detail I highly recommend the blog post series “MQTT Essentials” by HiveMQ. and the corresponding video series “MQTT Essentials” as well as the video series “MQTT 5 Essentials”.

MQTT brokers

There are many MQTT brokers available out there. As the brokers are usually run inside one container or in several containers in high availability configuration the programming language they are implemented in is rather unimportant. Of course it makes a difference but the characteristics of the broker (cluster) seen as “black box” is crucial.

The brokers I know of with adoption in big enterprise projects or products are HiveMQ and EMQ X. HiveMQ is a german company and offers a open source variant of their MQTT broker as well as an enterprise-ready variant. EMQ X is a chinese company offering the EMQ X Broker which is an open source broker as well as EMQ X Enterprise which is an enterprise ready MQTT broker. “Enterprise-ready” means usually that they support the setup of high throughput, high-availability brokers.

What makes EMQ X unique is that they are targeting the “edge server market” specifically with EMQ X Edge (open source lightweight IoT message broker for edge), EMQ X Kuiper (SQL-based ultra-lightweight edge data streaming), EMQ X Neuron (industrial IoT edge protocol gateway) and EMQ Edge Manager (Visualized EMQ Edge Software Management platform for Neuron, Edge and Kuiper).

Broker implementations may support a single or several MQTT versions. However an implementation does not neccessarily implement the full set of features of a particular MQTT version. This may be a show stopper for beeing able to use a specific broker, especially if one wants to use the Sparkplug specification and imlpies the need for an in-depth evaluation of available brokers. Key takeaway: Not every broker implementation supports all MQTT versions. Not all features of a MQTT version are neccessarily supported.

MQTT clients

Again there are many MQTT client implementations out there. There is at least one implementation available for almost all programming languages you can think of.

Client implementations may support a single or several MQTT versions. However remember that an implementation neccessarily implements the full set of features by a particular MQTT version. This may be a show stopper for beeing able to use a specific client and imlpies the need for an in-depth evaluation of available clients. Key takeaway: Not every client implementation supports all MQTT versions. Not all features of a MQTT version are neccessarily supported.

IoT cloud provider compatibility

As far as I know all cloud providers targeting IoT use cases as well as IoT platforms support MQTT for inter-device communication of data and RESTful API for configuration and control features. However some important aspect is the MQTT version and feature set supported by the platforms and providers.

Let’s have a look at the cloud providers first:

Amazon AWS IoT Core support for MQTT is based on v3.1.1 but does not comply with the specification completely.

Google Cloud IoT Core support for MQTT is based on v3.1.1 but does not support all features. Not supported are Quality of Service (QoS) 2, arbitrary MQTT topics and subscriptions, Last Will and Testament (LWT), retained messages and persistent sessions.

Microsoft Azure IoT Hub supports MQTT v3.1.1 and seems to be compliant. At the time of writing the support for MQTT v5.0 is available as preview only.

Key takeaway: All cloud provider IoT offerings provide support for MQTT. However the support is limited to MQTT v3.1.1. Some of the MQTT v3.1.1 features are potentially not supported. This might prevent from implementing Sparkplug on top of MQTT. An in-depth evaluation of the cloud provider support for MQTT is required during the system design phase!

A report of IoT ONE for 2019 lists the top 500 IoT companies. A lot of them providing IoT platforms like e.g.

• Bosch IoT Suite: Seems to be not cloud provider specific.
• Device Insight CENTERSIGHT: Is based on Azure IoT.
• Cumulocity IoT: Seems to be not cloud provider specific, runs on top of AWS, Azure, etc.
• Schneider Electric ExoStructure Plant & Machine: Seems to be based on Azure.
• Siemens MindSphere: Seems to be not cloud provider specific.
• …

Key takeaway: The majority of IoT platforms are created on top of cloud providers. Platforms using Amazon AWS IoT Core or Google Cloud IoT Core won’t allow the platform IoT providers to add support for Sparkplug. Platforms using Microsoft Azure IoT Hub could potentially add Sparkplug support. For other IoT platform providers an in-depth analysis for Sparkplug compatibility is required.

Conclusion

MQTT is one de-facto standard in the IoT domain with an already extensive adoption. When using MQTT in combination with Sparkplug one can add semantic meaning to the data communicated. The MQTT version compatiblity of cloud providers and IoT platforms is important w.r.t. current and potential future Sparkplug support. The potential of MQTT is huge. Especially when used with Sparkplug. The probability for an even more extensive adoption of MQTT in the future is high.