MQTT based Lightweight messaging using VerneMQ

Many of you must be hearing VerneMQ and MQTT for the first time. In this article we’ll be understanding MQTT and VerneMQ briefly, also we will be setting up a verneMQ Broker locally and publish messages to multiple clients in realtime. Also, we will discuss the wide applications of MQTT and the huge potential it holds.

As per the official documentation. “VerneMQ is a high-performance, distributed MQTT broker. It scales horizontally and vertically on commodity hardware to support a high number of concurrent publishers and consumers while maintaining low latency and fault tolerance. VerneMQ is the reliable message hub for your IoT platform or smart products.”

Yes in this growing world of smart IoT devices and mobile communication we need a scalable and reliable method for communication, which consumes lesser network bandwidth as well as consumes lower power. VerneMQ is one such solution.

The official docs read VerneMQ as a distributed MQTT Broker, but what exactly is MQTT and why is it called a broker.

In simple words, MQTT is an industry-standard protocol that supports publish-subscribe messaging. And it works perfectly for lightweight IoT or Mobile Devices which are so common in our daily usage. It is called a Broker because it sits in between multiple Clients or Backend Services to facilitate publishing and subscribing of messages. Something similar to the image shown below.

A General MQTT Broker managing publishing/subscribing of messages on multiple devices

In this article, we’ll be looking at setting up a verneMQ Broker locally on our system and try publishing and subscribing to messages via two separate clients.

Setting Up VerneMQ Broker

For running VerneMQ, you require Erlang/OTP 21.2 or a newer version and libsnappy-dev installed in your system. You can also follow the Official Github Readme for setting up the broker.

Git clone the Official VerneMQ Repository

git clone https://github.com/vernemq/vernemq.git
cd vernemq
make rel

Once this is done add the path to the verneMQ binary file in your .bash or .zshrc file and reload the terminal.

export PATH="<Your VerneMQ Directory Path>/_build/default/rel/vernemq/bin:$PATH"

Now you’ll be able to directly run verneMQ broker from the terminal using the commands mentioned and receive something similar and the broker logs will appear.

vernemq start // to start the verneMQ broker
vernemq console // to start the broker with logs
vernemq attach  //to log a currently running verneMQ broker
Remote Shell: Use "Ctrl-G q" to quit.
q() or init:stop() will terminate the vernemq node.
Erlang/OTP 23 [erts-11.1.5] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [hipe] [dtrace]
Eshell V11.1.5  (abort with ^G)
(VerneMQ@127.0.0.1)1>

Now since the broker is up and running, we can proceed to publish and subscribe messages, but wait verneMQ offers loads of configurations comprising security, queue management, TCP options, proxy protocol, etc. In this article, we’ll be sticking to the basics and communicating without any authentication checks. You can read a detailed view of the verneMQ configuration from the official documentation.

Welcome

Coming back to the verneMQ configuration file which is located in: ./_build/default/rel/vernemq/etc/vernemq.conf

In the configuration you can find the default TCP port the broker is listening to ie 127.0.0.1:1883, the default username and password if we use a security check, but the property we are interested in is allow_anonymous which is by default set to false. For our use-case set the value to true, this will disable any kind of authentication of clients, though it is suggested to use this for testing and development purposes only.

Restart the verneMQ broker after saving the changes in vernemq.conf file and we are all set to send and receive messages.

You can also view the status of verneMQ broker on localhost:8888/status, its a status dashboard to monitor the vernemq cluster and client connection details and publishing /subscribing rates.

Setting up VerneMQ Client

For the client, we’ll be using an Open Source CLI tool mqtt-cli which is very easy to use for testing and interacting with the MQTT Broker.

For macOS Users:

$ brew tap hivemq/mqtt-cli 
$ brew install mqtt-cli
==> Installing mqtt-cli from hivemq/mqtt-cli
brew  /usr/local/Cellar/mqtt-cli/1.0.0: 5 files, ... built in 2 seconds
$ mqtt <command>

For Linux Users (Debian/Ubuntu Systems):

$ wget 
https://github.com/hivemq/mqtt-cli/releases/download/v1.0.0/mqtt-cli_1.0.0_all.deb
$ sudo apt install ./mqtt-cli_1.0.0_all.deb

once the mqtt-cli is installed, we’ll go through some basic commands to start the MQTT client

$ mqtt shell
--> starts the shell and informs you about the default broker configuration, port, and the location of the log file.
Usage:  shell [-hlV] [COMMAND]
Starts MqttCLI in shell mode, to enable interactive mode with further sub commands.
Options:
  -h, --help      display this help message
  -l              Log to $HOME/.mqtt-cli/logs (Configurable through $HOME/.mqtt-cli/config.properties)
  -V, --version   display version info
Commands:
  help             Displays help information about the specified command
  version          Prints version information
  con, connect     Connects an mqtt client
  dis, disconnect  Disconnects an mqtt client
  switch           Switch the current context
  ls, list         List all connected clients with their respective identifieres
  cls, clear       Clears the screen
  exit             Exit the shell
Press Ctl-C to exit.

Once you enter the mqtt shell, we have a list of commands to use. We’ll be first connecting to the verneMQ broker,

mqtt> con -h 127.0.0.1 -p 1883 // h is the host address, p is the open port listening to TCP connections

after successfully connecting, you receive a random string as your allocated clientId, something of the form <Random String>@hostname, Now you are free to publish and subscribe to messages to one or more other clients.

Subscribe to a random_topic and wait for any published message from another client

mqttAGiwXaLL@localhost> sub -t random_topic -s

Simultaneously split the terminal and follow the same steps to connect to the broker, we’ll be publishing messages with this new client and transmit them to the subscribed topic clients.

mqttAGiwXaLL@localhost> pub -t dummy_topic -m "Hello World"

Woohoo, you’ll see the magic. You have successfully communicated between two clients (though locally), but it's a great start. I am adding a demo GIF of what it looks like:

Subscribing/Publishing to a random_topic locally using mqtt-client

Here I am listing all the Options for a more detailed understanding of tags and arguments while publishing and subscribing messages.

mqttAGiwXaLL@localhost> help
Usage:  >  { pub | sub | unsub | dis | switch | ls | cls | exit }
In context mode all mqtt commands relate to the currently active client.
Commands:
  help                Displays help information about the specified command
  version             Prints version information
  pub, publish        Publish a message to a list of topics
  sub, subscribe      Subscribe this mqtt client to a list of topics
  unsub, unsubscribe  Unsubscribes this mqtt client from a list of topics
  con, connect        Connects an mqtt client
  dis, disconnect     Disconnects this mqtt client
  switch              Switch the current context
  ls, list            List all connected clients with their respective identifieres
  cls, clear          Clears the screen
  exit                Exit the current context
====================================================================
mqttAGiwXaLL@localhost> help sub
help sub
Usage:  >  sub [-hJsT] [-b64] [-oc] [-of <publishFile>] [-q <qos>]... -t <topics> [-t <topics>]... [-up <userProperties>]...
Subscribe this mqtt client to a list of topics
Options:
  -h, --help             display this help message
  -t, --topic <topics>   The topics to subscribe to
  -q, --qos <qos>        Quality of service for the corresponding topics (default for all: 2)
      -up, --userProperty <userProperties>
                         A user property of the subscribe message
      -of, --outputToFile <publishFile>
                         A file to which the received publish messages will be written
      -oc, --outputToConsole
                         The received messages will be written to the console (default: false)
  -s, --stay             The subscribe will block the console and wait for publish messages to print (default: false)
      -b64, --base64     Specify the encoding of the received messages as Base64 (default: false)
  -J, --jsonOutput       Print the received publishes in pretty JSON format
  -T, --showTopics       Prepend the specific topic name to the received publish
====================================================================
mqttAGiwXaLL@localhost> help pub
Usage:  >  pub [-hr] [-cd <correlationData>] [-ct <contentType>] [-e <messageExpiryInterval>] -m <message> [-pf <payloadFormatIndicator>] [-rt <responseTopic>]
               [-q <qos>]... -t <topics> [-t <topics>]... [-up <userProperties>]...
Publish a message to a list of topics
Options:
  -h, --help                display this help message
  -t, --topic <topics>      The topics to publish to
  -q, --qos <qos>           Quality of service for the corresponding topic (default for all: 0)
  -m, --message <message>   The message to publish
  -r, --[no-]retain         The message will be retained (default: false)
  -e, --messageExpiryInterval <messageExpiryInterval>
                            The lifetime of the publish message in seconds (default: no message expiry)
      -pf, --payloadFormatIndicator <payloadFormatIndicator>
                            The payload format indicator of the publish message
      -ct, --contentType <contentType>
                            A description of publish message's content
      -rt, --responseTopic <responseTopic>
                            The topic name for the publish message`s response message
      -cd, --correlationData <correlationData>
                            The correlation data of the publish message
      -up, --userProperty <userProperties>
                            A user property of the publish message

Wide Applications and Advantages of Distributed MQTT based Messaging

There are a lot of Open Source MQTT Brokers one such is VerneMQ, which allows us to openly modify as per use-case and active development of new features for messaging and sharing data among multiple nodes.

• A lot of Messaging Apps/Services use MQTT based brokers for real-time push notifications and messages.
• Due to low network bandwidth consumption, it can be easily used in IoT Devices located in remote areas for sensing and transmitting data.
• It can scale up to handle millions of parallel connections and manage its state effectively.
• MQTT finds diverse applications in areas like the Automotive industry, Logistics, Manufacturing, Smart Devices, and Transportation.

For a more detailed insight into MQTT and MQTT based brokers visit the official documentation of VerneMQ and MQTT:

MQTT - The Standard for IoT Messaging

Getting Started

MQTT CLI: Smart, Compact, Open Source