Developing software for Bolt scooters: how we built the communication layer involving MQTT protocol

At Bolt, we’re building the best way for people to move in cities — a network of services for every distance, price range or customer need. Every fourth ride on the Bolt app globally is less than 3 km, which is a perfect distance to cover with a light electric vehicle, so adding electric scooters to our platform was a very natural step for us. Solving the “last mile problem”, scooters are definitely the most fun and quick way to ride around the city. Especially on a sunny summer day.

Moreover, scooters contribute to our long-term goal of reducing the ecological footprint of Bolt as a company. The CO2 emissions (per passenger) of a scooter are 63 times lower than those of the average single-passenger car! In a yearly view, the fuel cost of an 8km daily trip is higher than an average person’s spend on groceries in a month. For a scooter, these numbers equate to the cost of a cup of coffee in Starbucks. People will always want to take a car for certain types of rides, but a lot of times it’s easier and cheaper to do a short trip via scooter.

Bolt scooters currently operate in different cities across Estonia, Latvia, Lithuania and Spain, but even more locations are on the way.

In this post, I will share how we built the backend for the product, which involves a lot of communication between devices and a server, using MQTT network protocol.

Motivation

In order to manage such a widely distributed product we produce a lot of smart and complex solutions. Let’s skip the part where scooters should be delivered to the target city and assume that we already have a fleet of ready-to-use scooters in our warehouse. So now we need to perform a daily routine to keep the product running, such as deploying/collecting scooters in the morning/evening and, no matter how reliable and robust they are, maintenance of damaged vehicles. To make it all happen, we need to interact heavily with every vehicle.

Our very first running backend was written in a month or so, by a team of two extremely talented engineers. While the backend is our regular area of expertise, we also need to handle communication with scooters in some way. As we started with regular scooters, similar to the ones available in retail, we definitely needed something in-between to interact with them, because they only have Bluetooth to communicate with the user. So the first solution involved a third-party provider of IoT devices, which allowed us to get data from the scooter and perform actions with it.

The Basic Solution

Those black boxes on each scooter are actually kind of its second brain. The IoT device enables communication (with some logic of course) between the server and the scooter. We have chosen a company to provide these devices. They sell boxes and provide an API to work with. So the first architecture was pretty simple:

When using an API we stick to the functions that the third-party provider tells us about. The provider will take care of other things from their side. Sounds fair. But, with scaling we started to feel uncomfortable with this solution, as the API is limited and fully controlled by an external provider, and it is limited to HTTP(s) protocol, which is not great in our case. Webhooks are sent over quite a long time period, which makes tracking painful and imprecise, and we still pay money for each IoT that works. Also, we were looking for a solution that would be scalable enough to support our fast growth.

The Advanced Solution

At this point we decided to take a chance on our own IoT module. Using our own hardware means that we also need to develop our own software for it, and an infrastructure to handle raw communication with devices. So we built a prototype of an IoT device, put it inside a box and connected it to a scooter. Actually, it was a bit more complicated than that, but that’s another story for another article. This is where MQTT comes into play.

Let’s define a few things about MQTT for those who’ve never used it, as it is a specific technology used in the IoT world. First of all, MQTT is a lightweight publish-subscribe network protocol, designed for minimal battery loss and bandwidth on non-powerful devices. IBM developed it in 1999 for their oil pipelines connected to satellites. After its royalty-free release in 2010 the protocol became popular in the IoT world. Later versions of MQTT shifted the focus to a new target group of IoT devices, but it still has the same ideas at its core.

Quite often MQTT is referred to as a message queue, but it is different. In a message queue:

• the message is stored until it is consumed
• the message is only consumed by one client
• queues are named and must be created explicitly

In MQTT, messages can be consumed by a lot of clients at the same time and topics are dynamic, so clients can even subscribe using wildcards. But the most significant feature here is the storage part, as MQTT doesn’t store messages (except for specific types of messages, such as last will), so it can’t be used as a reliable way to send messages.

And in turn, MQTT has a set of out-of-the-box features:

• advanced auth
• advanced filtering
• Quality of Service levels
• persistent sessions
• retained messages, last will and testament

You can check out a really nice and detailed description of all MQTT features at HiveMQ blog (https://www.hivemq.com/mqtt-essentials/), but let’s also define some basics so that we are on the same page.

We mentioned that MQTT is a publish/subscribe network protocol. The publish/subscribe pattern (also known as pub/sub) gives you another method of communication other than the traditional client-server architecture. The pub/sub model decouples the client that sends a message (the publisher) from the client or clients that receive the messages (the subscribers). In such an architecture, the publishers and subscribers never contact each other directly and are not even aware that each other exists. The connection between them is handled by an external component (the broker). The job of the broker is to handle the communication, to filter all incoming messages and distribute them correctly to subscribers. With such a simple pattern we obtain a lot of advantages, such as scalability potential, decoupling between clients, durability, and at the same time a light and simple communication channel.

Most pub/sub systems have their logic on the broker side, and MQTT is a true pub/sub communication protocol in that way, as clients are as light as possible.

An MQTT client could be any device (from microcontroller to even mainframe) that runs an MQTT library and connects to an MQTT broker over a network. As the protocol is based on TCP/IP, the only condition for both the client and the broker is to have a TCP/IP stack. The MQTT connection is always between the client and the broker, so clients are never aware of other clients and do not communicate with them directly.

Therefore, we have our own IoT devices and our own infrastructure to handle all the communication.

Performance

What about performance and client footprint?

Let’s take a look at some comparisons (source: http://stephendnicholas.com/posts/power-profiling-mqtt-vs-https ).

There are two client applications for Android and two servers to work with. One app uses a MQTT client (standard Java MQTT client from IBM, with SSL enabled), and the other app uses HTTPS (standard HttpsUrlConnection and Comet-style long polling). The backend for the MQTT approach uses MicroBroker, and for HTTPS it is a simple node.js server. You can check out the details by clicking on the link to source.

1024 messages, of 1 byte a piece, to/from the phone, as quickly as possible.

For both scenarios MQTT shows much higher performance and lower footprint on the client. HTTPS turns out to be unreliable even, because some parts of the messages were lost, while MQTT got all the messages.

* — % Battery / Message is a slightly nonsensical metric. There is a fixed cost in having the Wifi or 3G active and so the actual cost of just sending / receiving a single message would be higher. However, these figures do serve to indicate the difference between the two approaches and hopefully gives you an indication of the battery usage involved.

Broker

Let’s go deeper with the broker, as it is the heart of communication.

We started with our own hosted open-source MQTT broker, Eclipse Mosquitto, an effective ready-to-use broker with a wide range of features, to test things out and do some prototyping, but it is not flexible enough. For example, authentication was possible only via loading the file on startup (yes, with hot reloading, but still). Some advanced auth can be done only with third-party plugins and direct access to database. Such things should be configured and handled using different settings, and we would be limited to the out-of-the-box possibilities and to those third-party plugins with tons of configurations for all our cases. On that basis, we decided to proceed with a more lightweight broker that can be built right into our system and that we can extend as much as we want and need. With this solution we can implement everything we need in-house and adapt it to all the nuances of our system including deployment, monitoring, scaling, etc.

Here we decided to try Aedes broker (https://github.com/mcollina/aedes). Why Aedes? It is a barebone MQTT broker, without any extra features that we don’t need. It is developed by the same people who created Mosca (https://github.com/mcollina/mosca), but it keeps in mind performance and stability as key factors. Also, it is written in Node.js just like our backend services. That means we can easily embed it using our expertise in building and running node.js services.

So far we have the next architecture on the top level:

The broker is depicted as an external module just to show it separately, but actually it is yet another service among other Bolt services.

Broker Scaling

There is not only one broker instance, apparently. But with multiple brokers comes another set of problems to be solved. A few examples:

1. Distributing messages that were sent to the first broker to all clients that were connected to, let’s say, the second broker;
2. Handling responses for non-fire-and-forget messages from clients, when it was sent through the first broker, and received on another broker;
3. Sending multiple messages in specific order with confirmations, through different brokers;
4. Handling of messages while brokers are redeploying.

Fortunately, Aedes broker has multiple implementations of persistence layer, which will help to synchronise the state on multiple brokers and will allow brokers to communicate. We decided to do it via Redis to be performant. The persistence layer solves problems with synchronisation, so brokers use Redis to sync persistent sessions, retained messages, messages with QoS > 0 and all information about current clients and subscriptions. They also sync published messages through Redis, so you can connect to any of the brokers and subscribe to what you need.

We also made the broker a part of a regular service, so that our backend won’t work with raw messages via MQTT protocol. The broker is running in the same process with a regular service, which listens to the broker internally, and passes all the data to further backend services via our standard communication channels, for example via simple HTTP calls or queueing data in the message queue. This approach allowed us to embed the broker and our own handling of MQTT communication in a much safer and quicker manner, because we have already worked with a third-party API to communicate with scooters. This means that it is a kind of imitation of a fully isolated service, and all other services haven’t changed too much.

Let’s take a look at the final architecture.

The last unexplained thing on this picture is the load balancer in front of the brokers. It is AWS NLB in our case, which does simple TCP load-balancing.

To Sum Up

So, we have built the MQTT broker inside our infrastructure and made the transition from third-party solution to our own as smooth as possible. Our own broker is enhanced with features like advanced authentication of each client, logging, scaling and monitoring. It works just great today, fitting all our needs and with the possibility of being enriched more and more.

What are our future plans? As Aedes is a barebone broker, in the near future we plan to implement advanced statistics on top of it with alerting features. Also, we will work on optimisation and syncing to multiple forms of storage for better stability and overall performance of the broker. We believe that MQTT can be used not just for IoT devices on scooters. As a good example, it can also be used for regular smartphones in scenarios where a client works with our app for a long time, like a driver. This can reduce the data and battery consumption significantly. Another example is interservice communication, which can be handled by MQTT. We will try to adopt this in the near future and will write about it!

If you’re interested in working with us on scooters, or any other exciting new products at Bolt, check our Careers page.