Dart in the IoT world — a practical example

Part 1 — Introduction

Note: things have changed for the better since this piece was written, we no longer need the python scripts, see here for more details.

My software career over many years has been grounded mainly in real-time processing ranging from very small embedded systems to very large distributed Supervisory Control and Data Acquisition systems(SCADA) systems across the defence, transportation and nuclear industries. I’m also a Dart fan, so what better to marry these two worlds together by using Dart to develop IOT applications.

I’ll show by example, using a real Dart project, how to get data from a sensor, transfer the readings over MQTT to devices created in Google’s Cloud IoT Core service and from there push the data into a Big Query table for storage and analysis.

This is not a step by step ‘do this next..’ guide, more an example by demonstration of how to actually do this using real code, as such I will refer to Google’s documentation when needed, please read this in conjunction with the project code to aid your understanding. Although comprehensive the documentation can be daunting to those not familiar with the area, as an example this shows how to set up the Iot Core MQTT bridge, reading this along with a real implementation I find is much less off-putting to newcomers to this area.

For those wanting an early look at where we are going with this please look here, this is our reference working project containing all the Dart code, the cloud function code and the shell/python scripts needed.

For those who don’t have access to maker boards such as Raspberry Pi’s, Arduino’s etc don’t despair, we can drive the information chain without access to real sensors purely from the Dart project itself. You will need however a functioning GCE project and be familiar with using Google cloud through its web interface.

Ok, next, a little on terminology, the IOT world has a plethora of terminology often interpreted differently depending on what your view is.

Part 2 — Terminology

I’ll keep this short and hopefully as clear as possible, please keep in mind this is my view of IOT terminology in relation to this project, yours may well differ.

From the bottom to the top we have 4 levels :-

1. Sensor
2. Gateway processor
3. Edge processor
4. Cloud

Taking these in turn a sensor does what it says on the can, it senses its environment and produces measurements in some form representative of a dimension of that environment, e.g light, temperature, pressure etc. For our purposes sensors have no control component other than basic start/stop and produce a sensed value on demand, e.g an integer. These devices can be very small MCU based devices with extremely limited memory and processing capabilities. They can also be simulated, which we use to our advantage later.

A gateway processor is a larger device with say an ARM processor and around 512Mb of RAM, network enabled, capable of running a Linux distribution, that hosts and collects data from sensors. It is typically used as a house controller for instance.

An edge processor is a much larger device again with powerful processors, greater than 2Gb RAM, basically a mini-pc type device. An edge processor would be used to control several gateway processors, filtering and transforming data before onward transmission to the cloud. Using our gateway example above an edge processor would control a housing estate rather than an individual house.

The cloud is well, the cloud, this where large data aggregation and analysis is carried out along with asset control and monitoring functions of various kinds. For our purposes the cloud is Google’s cloud but all the large cloud suppliers now have MQTT enabled IoT offerings like Iot-Core.

We are only interested in sensors, gateways and the cloud, edge processing is for another story. Speaking of which from the above what would you think of if you heard the phrase ‘edge device’? Answers on a postcard…

Ok, now we all know what we are talking about lets get down to doing some real work.

Part 3 — Getting sensor data into the cloud

If you haven’t already done so clone the iot-home project and set it up on a Dart platform of your choice. You don’t need to do this on a gateway device at the moment, your usual Linux/Mac/Windows environment will suffice. You will need to fill in your project details in the secrets.dart file

Lets start with the source of the data, our sensors, specifically look at the dummy_sensor.dart class in the sensors directory. As the comment says this class generates a stream of integer values generated randomly between the range of 0..40. A base of 64 is then added to make the generated value printable. A value is generated on a timed basis and made available through the getSensorData() method. So, this is a sensor as defined above in all but the ‘sensing its environment’ sense, it produces values on demand, in this respect it is no different from the temperature or light sensors, i.e it supplies the same interface.

Now we have our sensor device we need a counterpart in the cloud, log into your favourite GCE project, go to device registries in Iot Core and create yourself a registry, I called mine ‘iot-home’, in the registry create a device named ‘dummy-sensor’(please use this name, the code will refer to it as this, if you don’t then you need to change the code accordingly) as an MQTT device. Generate an RS256 public/private key pair, paste the public key into the device in the registry and the private key into the file secret/dummy-sensor-pk.key.

Now, back to the code, please study the mqtt_bridge.dart class, this class implements this page from the Google documentation, i.e it generates the JWT token and the other credentials needed to drive the IoT Core MQTT bridge, its worth taking a little time here, especially check the entries in your secrets file for registry/project name etc.

Ok, that done now have a look at the iot_home_dummy.dart file in the bin directory, this just initialises the components, starts the sensor and connects to your device over MQTT. If you run this you should see the output like the following if all is well :-

Welcome to iot-home for device dummy-sensor with a sample rate of 10 seconds
Authenticating with username '{unused}' and password '{eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MjU0Mjk2ODAsImV4cCI6MTUyNTUxNjA4MCwiYXVkIjoid2FybS1hY3Rvci0zNTYifQ==.XD5gVIUjGw6rteAlgoTYTGDLOLDSHfxYJTvITdc82OMSF8DybBIQ0kG-9B2QeHFi7YjMPc4rXS7w1ybgKh7FeXE8IMR_B4PGFn0rR0SO0njUDbRM1gdIV0_fhG88dYXnnFb8m0xv-BY4lr7Y7G_b0TrOW-YdO8ATe_spgdM1kUhG-yZlSkxyWaZApbohGSYS3Hqb3xkYLzPKqxOHq9UG2rrgvo8N8LUpDbdh_p3hygKHw_QD7RZUlr89JsXqDi35Om4bJpJcx1LRKis1UEehvGD3ZMFwHUBE37Zkv4KGWe_VLO6qsHDI4CYEqqaavxnFrzqnb44qN1GM8DZlZ8aVN_5flzLFHjMHn1x-Kq5kB4uh2gt3HCrC2dgr4xF2Hjs75AHOhJrmHZVtWY9nG3e-MVI6ngKubeV8O9ZdO-n81mFv15AQC7yLQMJ16Qvcl-91sfQf1dbq0BC9tNEYzdHcz4-s-JE8EvncsAorC5i3R3T4T475nd8v99hb10F8jvU8yrwlwKFBkeLjWRxwTNPaGr9HwcVDr-nu43bSr-DY25hCAnOM7EGPr34Hj6FcT2J1YtCz0R20MuumMnivoOS5OT9k4oXwHluclWScYNr-xjIgMh7sCFpITfqGusZOqSLtC2zH1yL7Frf357pi2J3TLmB2EVaTzOEQl55xoq73AnI=}'
Password length (802) exceeds the max recommended in the MQTT spec. 
Dummy sensor value is 95 at time 2018-05-04-11:28:09
Dummy sensor value is 97 at time 2018-05-04-11:28:19
Dummy sensor value is 99 at time 2018-05-04-11:28:29

Your password will be different of course. You can turn on more detailed logging with the -l option and change the sample rate with the -s option. If you now look at your device in your registry you should see the Last seen time has been updated to the current time. You should also see the devices Heartbeat (MQTT only) and Telemetry event received times update.

The MQTT bridge is quite unforgiving, if things aren’t exactly as it wants you’ll just get back either ‘failed to connect’ or ‘failed to authenticate’, again, please recheck your naming.

Ok, now we only have to push the data into Big Query so we can have a look at it, that’s next.

Part 4 — Processing and storing our sensor data

So where(and what) is our sensor data? When you created your device registry you also created a Pub/Sub topic for it where your device’s sensed data can be accessed, named projects/<project id>/topics/<registry name>. Going back to the MqttBridge class you can see how the MQTT topic we subscribe to is constructed by the getTelemetryTopic() method. This allows us to specify which device in our registry we are supplying data for. If you run the sample dummy sensor code and subscribe to your registry topic(you can do this a number of ways, one being Google’s gcloud tools) you will see your dummy sensor data being updated.

Notice that the data is a string, not just the sensor value, the data we send over the MQTT bridge is actually a stringified version of the SensorData class which gives us the type of the sensor, the value and the timestamp(the ‘at’ property) the value was sampled. This timestamp, called sample time or more formally acquisition time is quite important in the IoT world, no matter how much more processing you do with your data this is the real time at which the associated value was read. As a result of this in large scale SCADA systems the accuracy of time across the sensor estate is crucial, usually large sensing plant such as say a nuclear facility synchronize time via the use of highly accurate radio clocks and master clocks. Acquisition time should be stamped on the sensed value as soon as possible, preferably by the sensor but this is not always possible.

So, lets now look at what we need to represent this data in BigQuery. In BigQuery create a dataset, I named mine iot_home and a table named ‘sensor_data’(this is used in the cloud function so if you change it beware!). It should have just 3 columns that tie in with our incoming sensor data, see below.

Right, so all we need to do now is populate it. We do this using a cloud function, this attaches itself to our registry topic(called its trigger), transforms the incoming data and updates our table above. Cloud functions are implemented in GCE using nodejs, so they are written in java script. Create a cloud function(name isn’t important) and as its trigger attach it to our iot-home topic, this should be visible in the topic drop down. Now have a look at the index.js file in the cloudfunction directory of the project.

This is a small java script function that imports our BigQuery interface, gets the incoming topic data, logs it, transforms it into values we can use to update our BigQuery table and updates it. It really is that simple, you can copy this into your cloud function, also update the package.json file with the one from the project to pull in the BigQuery API.

Now, every time our sensor updates we should trigger our cloud function which will update the BigQuery table, from there we can query the incoming data, export it and do what ever analytics we want.

So lets do it, run the dummy sensor code, if your cloud function is set up correctly you should be able to go to your Logging pages and see lines like this :-

D  iot-home-sensor 93758829817053 Function execution started  iot-home-sensor 93758829817053
I  iot-home-sensor 93758829817053 dummy:67:1525430789125  iot-home-sensor 93758829817053

If your not seeing this then something is wrong, if all is well go to your BigQuery table and run a query on it like this :-

SELECT value, at FROM [iot_home.sensor_data] 
WHERE type = 'dummy' ORDER BY at DESC

This will show you your incoming dummy sensor data values as they come in in real time.

From here you can export the data to any analytics tool of your choice to graph the data, I simply exported the data as a CSV file and used Google Sheets to do this, there is also a plugin for Google’s DataStudio product to try if you wish. I’ll leave this as a reader exercise.

OK, we have now set up and proven our information chain from sensor to storage, lets have a look at interfacing to real sensors.

5. Using real sensors

To use real sensors we of course need to host them and interface to them. There are a number of whats known as ‘maker’ boards on the market, such as Arduino’s, Raspberry Pi’s etc that support sensors and operating systems that allow you to interface to them and read their data. The board I used for this example project is the BeagleBone Green, which is a BeagleBone black with a wireless interface. This board has 4Gb of Flash, 512Mb memory and runs an ARM version of Debian, what we would call a gateway processor. It matters not what your choice of board is as long as it supports the Dart VM and is network enabled. Pick your board and install Dart onto it, clone and install the iot-home project.

Ok, so how do we interface the VM to real sensors on real device buses, well in short its tricky at the moment to do this natively, sensor libraries such as mraa for Linux would need to be implemented in Dart, these in turn boil down to ioctl system calls on Linux and DeviceIoControl(or equivalent) calls on Windows, its hard to unify these calls into a generic device driver layer. You could use Dart native extensions to call these libraries but these too have their drawbacks. If you have simple sensing requirements(like us) there’s usually a simpler way to do this.

One way is to use an already device aware language to interface to our sensors and use the output of that as our sensor data stream. A good choice here is Python. The BeagleBone comes with Python bindings to mraa allowing us get sensor data and print it etc. so all we need do is get Dart to call a Python script and get its generated output, fortunately Dart supports this using the Process class from dart.io. This allows us to run external scripts and collect the output, have a look at the ExecuteSensorScript class in the project for an example.

Basically we call the external script and get its output on stdout as our sensor value, so all our external script has to do is read our sensor on demand and print its value. Have a look at the Python scripts in the scripts directory, these of course are specific to the BeagleBone but are easily changed for your board of choice. We also make it a bit simpler by not calling these directly from Dart, we actually call a shell script which calls the Python script, this allows us to set up IO redirects and to set appropriate privileges if needed, in our scripts for example we need to be root to read sensor data on the BeagleBone , so we use the sudo command to call the python scripts with. This can be seen in the shell script examples in the script directory.

If you now look at the iot_home_temperature/light dart programs in the bin directory you can see how these use the TemperatureSensor and LightSensor classes in exactly the same way as the DummySensor class to drive the MQTT bridge, i.e our interface to our devices is the same, we just generate the values differently.

Once set up and you have created your temperature and light devices in your registry you can run the associated dart programs and populate your BigQuery table from real sensors, seeing the data is just a simple matter of changing your query slightly, for the light sensor :-

SELECT value, at FROM [iot_home.sensor_data] 
WHERE type = 'light' ORDER BY at DESC

Before we go a quick roundup of further thoughts and a few questions.

6. Where next?

Right, so looking at what we have done above a few questions should immediately spring to mind like, Why do we need the registry devices? We could just publish sensor data onto a Pub/Sub topic and farm it out from there, most if not all gateway level devices will support this for even high data rates. Answer, we don’t need the device registry, pushing straight up to Pub/Sub from the Dart VM works, I’ve done it, Iot-Core hasn’t been around as long as Dart so at one point it was a viable choice.

Hey, if all we want to do is get data into BigQuery why even use Pub/Sub, just update your BigQuery table straight from the Dart VM and have done with it. Again, you could do this, it depends on what you are trying to achieve.

The above strategies work for relatively simple use cases but when you bring the real world in things get more complex. Not everything is new, many(millions) of sensors are out there already talking over MQTT, not Pub/Sub, then there’s even thornier issues of device management, security, error handling and reporting(alarm management) etc. etc.

In general the more standard your interfaces(MQTT) and the more decoupled and event driven your environment the better. Even our simple example affords many inspection points for the sensed data before it gets into BigQuery, we could massage the data on sensing in the Python scripts, in the Dart VM or in the cloud functions. Also its all separately testable which allows for better integration, remember what works for a few 1000 devices probably won’t scale to millions.

Our little example project is just that, please use it as an educational tool, not an example of how to implement a real-world IoT gateway device. Please however play with it as you wish. I’ll gladly answer technical questions or any questions really, you may wish to raise these as issues on the project itself if you think it may get a little long rather than comments on this article, choice is yours.

The IoT world has many facets, there are many write-ups on IoT device handling that go far beyond the scope of this simple example, if this has whetted your appetite please go find and enjoy!