How To Build Your Sigfox Server (Version 1.0)

How do you build an IoT server? As CTO of UnaBiz (Sigfox IoT network operator for Singapore and Taiwan) I’ve had the privilege of helping people around the world answer that tough question. (I’m also teaching a related course at Temasek Polytechnic). The IoT industry is starting to discover that it takes more than hardware to build a great IoT product — it needs the right kind of software and servers too.

After spending a year building and iterating backend servers for our UnaMap and UnaBell products, I now appreciate that IoT servers need to be highly reliable — any server downtime will cause missing device messages and loss of business value. Yet IoT servers need to be robust to allow for room to experiment with various devices and message formats, without causing the whole system to fail because of a tiny bug. This may be unique to Sigfox, but we are very fortunate to receive overwhelming volumes of sensor data because of its low cost. This becomes a serious problem if the IoT server is not scalable and elastic to cope with fluctuating load patterns.

Receive →Decode →Visualise →Capture

In this article I’ll share the software frameworks that I have created to build highly reliable, robust and scalable systems for processing Sigfox messages. We’ll look at a simple four-stage message processing pipeline:

Receive: This is the Sigfox Callback that receives device messages from the Sigfox cloud.

Decode: The 12-byte messages received from Sigfox are decoded here and expanded into multiple fields for easier processing.

Visualise: While the message processing pipeline is running, it’s useful to visualise the kind of sensor data that’s flowing through the pipeline. So you can be sure that you’re not losing any messages or decoding the messages incorrectly.

Capture: For high volumes of messaging, it will not be cost-effective to analyse the sensor data on the fly. Instead, we will store the sensor data in a database or data warehouse and process the data in batches later.

What’s in a Sigfox message?

The purpose of a Sigfox server is to receive and process Sigfox messages transmitted by our Sigfox devices to the Sigfox network. And to do so efficiently, with little loss of data.

We usually configure the Sigfox cloud to deliver the messages to our server in JSON format (like above, minus my comments at the right). You can see the full 12-byte message in the data field above. It contains three sensor data values that have been compressed into 12 bytes.

Can’t the Sigfox Cloud decode the sensor data?my students often ask. The Sigfox cloud is not a general-purpose cloud like Amazon / Azure / Google for running our programs. The Sigfox cloud can decode messages with simple, fixed formats, like say “data contains 6 integers of 2 bytes each”. If your device uses a few message formats, or if it the message uses special compression methods, then the messages may only be decoded in your Sigfox server. Later we’ll see how we may decode a custom-defined Structured Message Format in our server.

Receive

This is the crucial step that must never die. The Sigfox cloud expects us to provide a HTTP POST Callback that will receive any messages transmitted by our Sigfox devices. If the callback service is down, we will lose messages from Sigfox. (Unless you prefer to call the Sigfox API to reload the missing messages, which will be messy.)

Easy - I’ll install a Virtual Server at Amazon / Azure / Google Cloud to handle callbacks” But Virtual Servers can’t run 24x7 — they need to be patched and restarted occasionally, and adding redundant servers is costly. The best solution — use Cloud Functions (also called Lambda Functions) provided by Amazon / Azure / Google (or whatever cloud you’re using) because Cloud Functions are meant to run 24x7 non-stop. We may upload our code for receiving messages into a Cloud Function, and expose it as a HTTP POST service. (If you’re using Amazon Cloud, the API Gateway service will expose your Lambda Function as a HTTP POST service.)

I have created a reference implementation called sigfoxCallback that runs on Google Cloud, you can read about it here: https://www.npmjs.com/package/sigfox-gcloud

Passing Messages Reliably

After the Receive Function has safely received the message, it needs to hand the message over to other programs that may not operate as reliably as the Receive Function. We’ll use a Message Queue for passing messages between programs, because Message Queues won’t pause (block) our sending program while waiting for the receiving program to receive the message (which is a problem if we used HTTP to pass messages).

In my reference implementation I have created a Google Cloud PubSub message queue called sigfox.devices.all that contains all messages received by the Receive Function. Any program that wishes to listen to all incoming messages can hook on to this queue.

Routing Messages

Our Sigfox message will be delivered to a sequence of functions (Receive →Decode →Visualise →Capture). It’s possible to rely on a central database to decide how to route the message after every function, but that gives us a new point of failure to consider.

Instead, upon receiving the message we decide how the message should be routed, and write the entire route into the message. That way the message is self-sufficient in routing itself from function to function. In my reference implementation this is done by the Cloud FunctionrouteMessage

Decode

The Decode Function listens for new messages (through the sigfox.devices.allmessage queue) and decodes the sensor data fields in the message. Sigfox was designed for sending messages affordably over incredible distances, but the message needs to fit within 12 bytes. Our Sigfox server needs a clear strategy for maximising the use of the 12 bytes and decode many types of messages intelligently.

That was a real problem for me and my students at Temasek Polytechnic. Each student project group created devices with different sensors, transmitting different types of Sigfox messages to a common AWS backend. My solution was to use a Decode Function that could to decode the 12 bytes into 3 sensor data values and their field names (when used together with an Arduino message encoding library).

For example, the 12-byte message b0513801a421f0019405a500 will be decoded as tmp = 31.2, hmd = 49.6, alt = 16.5 (for temperature, humidity, altitude). You can see my implementation in the decodeStructuredMessage function.

Here’s a Google Sheet that uses the same algorithm to decode a structured message (please make a copy before editing): https://docs.google.com/spreadsheets/d/1X3pEPz8zX7lajEPFnzJc0cLkCrFya8ila_eYjGBPb8Q/edit#gid=480889255

Visualise

For visualising the sensor data I experimented with Ubidots, an IoT platform that was recommended by my IoT students.

With the current server framework based on Cloud Functions loosely coupled with Message Queue, I could easily build new functions to tap into the Sigfox message stream and visualise the sensor data. To integrate Ubidots I built a Cloud Function that listens for incoming messages and transmits them to the Ubidots cloud through the Ubidots API. The data appears in Ubidots in real time, useful for checking the flow of sensor data before the final Capture step.

The code is available here: https://www.npmjs.com/package/sigfox-gcloud-ubidots

Capture

The final step in our simple processing pipeline is the Capture Function for writing the sensor data into the database. Based on the same architecture, I have written a Cloud Function called sendToDatabase that records the decoded Sigfox message into the SQL database of your choice, including MySQL, Postgres, MSSQL, MariaDB, SQLite3 and Oracle. More details here: https://www.npmjs.com/package/sigfox-gcloud-data

Decoded Sigfox messages stored in MySQL

The sensor data captured in the database will typically be fed to a post-processing pipeline (see below). If there is sufficient interest I’ll cover this in another post.

Receive →Decode →Notify →Pre-Visualise →Capture
 →Analyse →Learn →Predict →Post-Visualise

And Over To You

And there you have it — my reference design and implementation of a four-stage Sigfox message processing server that I’m using to create UnaBiz products and for teaching.

At UnaBiz we understand that IoT is a highly challenging problem to crack because it covers so many diverse skills, so I’m contributing my expertise in IoT architecture to the IoT community. Together let’s solve more problems with IoT!

Do you find these steps too complicated? Would you prefer to use a Sigfox server that’s already preconfigured for you? Lemme know!

Further Reading

This article explains how to implement a Sigfox Downlink server and how to configure the Sigfox backend for downlink. The article also includes a complete Arduino program that will wait for downlink while it continues to read the sensors…

Here is the STM32 Blue Pill version of that Sigfox client program. The STM32 version includes the STM32 code that simulates a Wisol Sigfox transceiver and a BME280 sensor in software…

Since the Sigfox breakout board is not available for Asia, I attempted to create my own using the Seeed Fusion PCB manufacturing and assembly service…