Did I beat Ben? Race tracking with Kafka KSQL, MQTT & Kibana

Simon Aubury
May 28 · 4 min read
Kibana dashboard — showing live race analytics

Design Overview

Runners are tracked with GPS based apps installed on their phones. Location updates are sent (via MQTT) to a central server. Kafka acts as a streaming platform with KSQL performing the live race computations (speed, rank, distance to finish line). The web-based mapping dashboard is generated using Kibana.

OwnTracks open-source phone app

Each runner has OwnTracks on their phone. OwnTracks is an open-source app (iOS and Android) for communicating phone location to an MQTT server. Phone location is a great proxy for runner location!

So why MQTT and not directly into Kafka? Well, MQTT is lightweight TCP/IP messaging protocol perfect for power constrained devices. It allows for short efficient messages to be constantly sent from the runners phone to a central server. It’s closer to a notice board than a queue, and allows for lightweight messages like “I’ve just moved to location x,y”.

CloudMQTT and OwnTracks app setup

I chose CloudMQTT to act as a hosted MQTT server, and the free-tier was super easy to configure. Each device has a unique login (eg., simonuser) and device ID (eg., simonphone). This allowed each runner to be individually tracked with a shared server.

MQTT to Kafka with Kafka Connect

Kafka and MQTT are two complementary technologies. MQTT acts more like a key/value store, whereas Kafka is a complete streaming platform. I used the Kafka Connect MQTT Connector to poll MQTT updates and bring the location updates down to a Kafka cluster.

Kafka Connect MQTT Source configuration

To check this is actually collection location data into the “data_mqtt” topic I can query like this

Looks like a valid stream of location updates. Now let’s process it to see who is winning.

Kafka Streaming Transformation with KSQL

I wanted to know where each runner was — along with their average speed and distance to the finish line. KSQL allowed me to determine each runners speed (over a 5 minute window) and distance to finish line.

By concatenating the latitude and longitude fields together I could also form a “geo_point” data type suitable for mapping in Kibana.

KSQL used to derive runner location and remaining distance

Kafka To Elastic & Kibana

Now I have 2 derived topics

  • runner-location — a stream of runner’s and their location
  • runner-status — a table of runner’s speed and remaining distance

I can write these topics into Kibana using the Elastic Kafka Connect Sink Connector.

Kibana Visualisations

A picture tells 10³ words

A Kibana dashboard is a great way to display a real-time display of

  • Runner location platted on a map
  • Speed graph per runner
  • Altitude plot per runner
  • A league table — who is closest to the finish line!

A nice trick for Kibana mapping when you want to zoom further to street-level data is to use Open Street Map as a third party map and tile services. Allows for much richer maps when zoomed in.

Did I beat Ben?

No. Ben beat me by a sizeable margin. Perhaps I should spend more time training than coding?

Me (left) who lost to Ben (right)

Want To Try It?

Want to build your own race mapper for displaying participant progress and location? You can build an entire Kafka, KSQL, Kibana dashboard with demonstration run data by cloning

Credits & References

Simon Aubury

Written by

Day job: data steaming & system architecture. Night gig: IoT and random project hacking

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade