Did I beat Ben? Race tracking with Kafka KSQL, MQTT & Kibana

Simon Aubury
May 28 · 4 min read

Live race tracking with MQTT and Kafka streaming. Race statistics using KSQL and visualisations with Kibana. Can I run a half marathon faster than Ben?

Kibana dashboard — showing live race analytics

Design Overview

Runners are tracked with GPS based apps installed on their phones. Location updates are sent (via MQTT) to a central server. Kafka acts as a streaming platform with KSQL performing the live race computations (speed, rank, distance to finish line). The web-based mapping dashboard is generated using Kibana.

OwnTracks open-source phone app

So why MQTT and not directly into Kafka? Well, MQTT is lightweight TCP/IP messaging protocol perfect for power constrained devices. It allows for short efficient messages to be constantly sent from the runners phone to a central server. It’s closer to a notice board than a queue, and allows for lightweight messages like “I’ve just moved to location x,y”.

CloudMQTT and OwnTracks app setup

I chose CloudMQTT to act as a hosted MQTT server, and the free-tier was super easy to configure. Each device has a unique login (eg., simonuser) and device ID (eg., simonphone). This allowed each runner to be individually tracked with a shared server.

MQTT to Kafka with Kafka Connect

Kafka Connect MQTT Source configuration

To check this is actually collection location data into the “data_mqtt” topic I can query like this

Looks like a valid stream of location updates. Now let’s process it to see who is winning.

Kafka Streaming Transformation with KSQL

By concatenating the latitude and longitude fields together I could also form a “geo_point” data type suitable for mapping in Kibana.

KSQL used to derive runner location and remaining distance

Kafka To Elastic & Kibana

  • runner-location — a stream of runner’s and their location
  • runner-status — a table of runner’s speed and remaining distance

I can write these topics into Kibana using the Elastic Kafka Connect Sink Connector.

Kibana Visualisations

A picture tells 10³ words

A Kibana dashboard is a great way to display a real-time display of

  • Runner location platted on a map
  • Speed graph per runner
  • Altitude plot per runner
  • A league table — who is closest to the finish line!

A nice trick for Kibana mapping when you want to zoom further to street-level data is to use Open Street Map as a third party map and tile services. Allows for much richer maps when zoomed in.

Did I beat Ben?

Me (left) who lost to Ben (right)

Want To Try It?

Credits & References

Simon Aubury

Written by

Day job: data steaming & system architecture. Night gig: IoT and random project hacking