MQTT on Steroids: Running EMQX Enterprise on Kubernetes

Tamas Foldi
HCLTech-Starschema Blog
5 min readFeb 14, 2023

--

Image generated by Dall-E.

Working a lot in the IT-OT integration space with demanding clients, we at Starschema have been using EMQX as a standard MQTT message broker due to its robust features, scalability and high-performance capabilities.

EMQX has proven to be a pretty neat broker, so to support the broader community, we’ve open-sourced our deployment scripts for it in Antares iDL, our modern data stack deployment system. In this blog, I will show you how to deploy it in your cloud or an on-prem Kubernetes cluster with a single click (well, more like a single command line).

Introducing EMQX, Your Open-Source MQTT Broker

EMQX is an open-source IoT (internet of things) message broker that supports MQTT, MQTT-SN, CoAP, Sparkplug B and LwM2M protocols. Its core advantages are:

  1. Scalability: EMQX can handle millions of concurrent connections and messages. It’s written in Erlang, a language designed for scalability with low latency and high availability.
  2. High availability: EMQX supports horizontal clustering and message persistence to ensure high availability.
  3. Customizability: EMQX offers a plugin architecture that enables custom extensions and enhancements.

We prefer EMQX in our projects for the following reasons:

  1. Open source: EMQX is open-source and free to use, with a large community of developers contributing to its development and evolution. There’s also an Enterprise version for larger deployments and commercial support.
  2. High-performance: It’s designed for high-throughput, low-latency message processing. In the IIoT (industrial internet of things) space, it’s common to expect messages from millions of devices or process millions of messages.
  3. Built-in processing engine: EMQX supports routing, filtering and alerting out-of-the-box. It can also send messages to external systems like databases or other message queues.
  4. And it works on-premises, as well in the cloud or hybrid deployments.

Deploying in Kubernetes

In our Antares iDL platform, we use only cloud-native services or Kubernetes deployments. For EMQX, we took the latter method and deployed it in Kubernetes, which provides several advantages, including:

  1. Scalability: Kubernetes makes it easy to scale EMQX horizontally by adding or removing nodes as needed. EMQX has its own Kubernetes operator to take care of scaling.
  2. Resilience: Kubernetes provides automatic failover and self-healing capabilities, ensuring high availability for EMQX.
  3. Portable deployment: Kubernetes allows EMQX to be deployed on-premises, in the cloud or in hybrid environments, which provides greater flexibility and portability. The same deployment automation can work everywhere.

Deploying with Command Line Only

In order to install EMQX, first we need to install cert-manager, a Kubernetes-native service for managing SSL certificates with Let’s Encrypt and other providers:

helm repo add jetstack https://charts.jetstack.io helm repo update helm install \ cert-manager jetstack/cert-manager \ — namespace cert-manager \ — create-namespace \ — version v1.11.0 \ — set installCRDs=true
Install cert-manager with Helm. Make sure you install custom resources (CRDs) as well.

When it completes, you can proceed to install EMQX’s operator. This operator will manage and operate EMQX clusters on our Kubernetes cluster and automate tasks such as deployment, scaling, upgrades and backups. It also provides ensures high availability through automatic failover.

helm repo add emqx https://repos.emqx.io/charts helm repo update helm install emqx-operator emqx/emqx-operator \ — namespace emqx-operator-system — create-namespace
Install EMQX Operator through their official Helm chart.

While other MQTT brokers (like HiveMQ) instantly provision a new cluster after Helm install, in EMQX you have to define your own cluster as a CRD. You can go with a plain EMQX broker or with the Enterprise Edition. In the latter case, the following Kubernetes resource will do the trick:

The resource kind is EmqxEnterprise in case of the Enterprise Edition.

If all is well, you should be able to see your EMQX pods and services. The admin port is exposed by the generated Service port 18083. The default username is admin and the password is public.

Fresh new installation of EMQX Enterprise Edition. Without a trial license, you’re limited to 10 connections per second. The open-source version does not have this limitation.

Deploying with Pulumi and Antares iDL

Internally, we prefer using Pulumi for deploying infrastructure. It offers several advantages over Terraform or manual, Helm-based installations. The biggest on is that you can write the infra code in modern programming languages (TypeScript, Go and Python, if that qualifies as modern), with advanced templating and control structures. On top of this, Pulumi’s cross-platform architecture eliminates the need to write separate code for different cloud providers, making it easier to manage infrastructure across multiple clouds.

To install cert-manager, we use the following Pulumi code, which relies on Release resource:

from pulumi_kubernetes.helm.v3 import Release, ReleaseArgs, RepositoryOptsArgs def deploy(): cert_manager_release = Release( “cert-manager”, ReleaseArgs( chart=”cert-manager”, name=”cert-manager”, repository_opts=RepositoryOptsArgs( repo=”https://charts.jetstack.io/”, ), namespace=”cert-manager”, create_namespace=True, values={“installCRDs”: “true”}, ), )
The cert-manager will have its own namespace.

Then, we install the EMQX Kubernetes operator:

Install and configure EMQX with Pulumi.

To install EMQX in a single command line, make sure you enable cert-manager and emqx in your Pulumi stack config.

Now, simply execute pulumi up, and everything will be there.

Output from Pulumi Up.

That’s pretty much it. If you want to try installing it with Pulumi, the code is here, while you can find the entire Antares iDL project page here.

Using EMQX — Testing the Installation

The most convenient way to test your installation is to use kubefwd. After creating the necessary port forward, you can access the admin interface at emqx-ee host’s 18083 port by default.

EMQX’s admin interface gives you the option to understand your deployment as well as to interact with plugins, set up rules, etc.

You can also check it from command line using any mqtt consoles.

Subscribing to “#” will show all incoming messages in the system.

Summary

I’ve shown how to deploy EMQX in a Kubernetes cluster with a single command line using Pulumi and our Antares iDL project. It’s definitely worth a look, as it has extensive infrastructure as code templates for the modern data stack components (Airbyte, Airflow, Fivetran, dbt, Dagster, Kafka, etc.) complete with some IIoT-relevant components. And, as always, reach out using the contact information below if we can help.

--

--

Tamas Foldi
HCLTech-Starschema Blog

Helping enterprises to become more data driven @ HCLTech, co-founder & former CEO @ Starschema