Collecting, modeling, and analyzing your IoT Data stream in real time ? Simply use data pods
--
Disclaimer
You can find a general introduction on how to get started with data pods in a previous article. In this article we want to demonstrate you how simple it is to model continuous flow of sensors data with data pods and actually configure realtime analysis dashboards. Whenever you feel that important information is lacking in this article on how to create a table, to configure a data transformation pipe, or to set up a report, please read this article to find more detailed information to get started.
Introduction
In this article we focus on modeling IoT data coming from sensors into data pods. Handling IoT data is of high interest especially in the manufacturing industry sectors. The IoT data in these cases could be coming from sensors of a fleet of cars, smart facility applications, or machines from a number of factory sites.
Since IoT data of different types of sensors varies in load, frequency, and quality, combining all of this information is a lot of work but it leads to a more accurate description of the state of your assets and improves maintenance service. It will allow a more precise identification of the malfunction or even, using smart algorithms, the prediction of future malfunctions. This will ultimately result in lower costs, reduced machine downtime, and optimized processes.
Generally, manufacturing companies that want to get started with IoT are faced with questions like:
How do you ..
* transmit the data?
* model the sensor data?
* relate different data streams?
* add information of machines/sites coming from other sources?
and ultimately
* automate the entire process to analyze and visualize data in almost realtime?
All of these questions will be discussed in this article with sample data using data pods. I set up the following pod to illustrate the IoT data stream. Feel free to check it out.
So how do you get started
If you have not done so yet, you should sign up for a free data pod to get started (www.repods.io).
At this point, data pods provide you with a user interface to subscribe to WAMP messages. WAMP is a lightweight messaging protocol suited for IoT applications. The messaging infrastructure for WAMP consists of a central messaging router and clients connecting to the router. We will use in the following example crossbar (crossbar.io) as the messaging router. You can setup and configure your crossbar router running the official crossbar docker image.
As a WAMP client we choose a Raspberry Pi 3 running Raspbian OS and a DH11 temperature sensor. Setting up the hardware is an easy task. Look for an example at the following tutorial. Our goal will be to run an application that reads data from the temperature sensor and publishes the sensor data to the crossbar router on a specified topic. For that I present you a working example python code. It is a python-twisted application that connects to the crossbar router, reads temperature data from the sensor and publishes data as WAMP events.
Starting from this example you can modify the code and connect your favorite sensor tailored to your needs.
You can run the following separate python script to subscribe to the topic and check if data is actually published to your crossbar router.
If you see the data printing inside the terminal you know that you are successfully publishing data to the crossbar router.
Inside the data pod
We are now in the position to actually enter your pod and collect the data from your device. For this task you (1) open the “Import” tab and (2) click the “IoT Import” button, and (3) define the parameters of the connection to your crossbar router, including url, realm and authentication method. Then (4) click the “Save” button to create the connection.
Once the connection is defined go ahead and configure the subscription to your data stream. (1) Click the “Add subscription” button, enter the necesary information and (2) click the “Save” button to add the subscription.
You will receive the data sent on this topic in packages of size and frequency set in the configuration of the subscription to the new table S_DATA_TEMPERATURE. The table columns in the case of IoT import are standardized.
The next step is to (1) press the play button to establish the connection to your crossbar router and (2) actually subscribe to the IoT data stream by checking the box.
Next you need to verify that data is actually flowing into your source table. For that you need to (1) enter the “Import” tab to see the incoming packages. Please click on the timeline to zoom into the current data loads. Here you see the incoming packages and by clicking on a given package, you can view the data in the data grid. If you want to get a summary of the time scope of the data sent and package size, you just hover over the package.
You can see the content of the source table by (1) entering the “Pipes” tab and (2) clicking on the source table icon. The data is entered in a default format, as mentioned before, with the actual sensor data included in the payload_args JSON object. The column “received_at” should not be confused with the actual time of the measurement. It is the time the event was received by the data pod.
The next step will be to create a table in your data model. Here you (1) enter the “Model” tab and (2) click on “Add Table” button. You can choose between an event table and state table, define the columns of your IoT event / resp IoT state. In our case we will model the data as events. Next (3) give the table a name, (4) choose type of table as event table, (5) define data columns, and (6) click the “Save” button. The table will appear in the graphical representation of the model.
Next step is to add a pipe to transform the incoming data and load it into the Evo table we just created. Please click on the add pipe button on the “Pipes” tab. (1) Choose a table from the list of existing tables, (2) select the table we just created, and (3) select the source table from the list and (4) press the “Create” button.
A new pipe will appear in the pipes dashboard. You now need to click on the pipe symbol connecting your source table to your evo table. A dialog window opens and allows you to (1) edit the SQL query that transforms data to the desired format that enters the Evo table. Please have a look on how to actually extract the details of the IoT event from the JSON object.
```sqlSELECT
(r1.payload_args[1]->>'tsp') as "event_date",
(r1.payload_args[1]->>'val')::text::numeric as "temperature"
FROM
S_DATA_TEMPERATURE r1
```
Next you need to (2) check the query return before confirming the pipe transformation is done by clicking the play button next the SQL editor. If it works, (3) click save.
Finally (4) click on ‘Force Pipe Flow’ to load the existing data into the Evo table. If this works you can (5) enable data free flow. This option executes the pipe process for every incoming data package automatically allowing for continuous data flow into your Evo table.
That is it. Nothing else to do. Your data model contains real time IoT data.
Check this by clicking on the Evo table in the “Pipes” tab filling up with data.
Now we are in the position to relate the IoT data to other information in the data model as for example production planning, warehouse stocks, information of the last maintenance session, and even delivery periods of, e.g. your replacement parts suppliers.
We go ahead and generate a report on the data. For that we (1) click on the “Reports” tab and add a report to the dashboard. We choose the temperature table and (2) select the action we apply to the different data columns. You need to (3) set the time interval and the grain size in time, (4) press execute to calculate the results, that you can view as a graph by (5) clicking the symbol next to the action. The output graph should display temperature per device as a function of time. If this works well, you can set the analysis on free flow mode by (6) clicking the static button to toggle to free flow.
Summary
We discussed all the necessary steps to establish a continuous IoT data stream into data pods. Once the data model is configured and in free flow mode we generate the newest results in almost real time.
There are important issues we had to leave out that would have lengthened the article and that will be covered in later articles such as authentification methods for the crossbar connection, combining multiple data streams, alternative IoT data messaging protocols, generating info graphics to embed in foreign websites, etc. There are lots of possibilities with data pods.
Please feel free to enter my public data pod IoTImportExamplePod, leave comments and suggestions and start working with your own data stream in your own data pod.
