How to Stream (Push) Data to a Real-Time Dashboard API in Python

If you are building projects in Python or using Python for data analysis, there is a good chance you will want to capture different types of data. You might even want to create meaningful data visualizations out of those data sets. For example, a dashboard of performance data (CPU/memory/network usage). Or, a dashboard full of sensor data (temperature/motion/contact/energy). Maybe a dashboard displaying the status of critical processes (running/exited). If it is data you can access in your Python code, you can easily build a real-time or IoT dashboard from that data.

This tutorial will show you three different ways to use the basics of Python to stream/push different types of data to a real-time dashboard API. While this tutorial will leverage Initial State’s API as an example, you can apply the principles in this tutorial to interface similar APIs.

Background Info: Initial State’s Events API

Initial State is a streaming service that allows you to visualize and analyze data in your web browser. It’s a powerful tool to prepare data for analysis by putting it in a well-designed dashboard. You can enjoy a 14-day free trial to experience all features of the platform.

Interfacing the Initial State Events API is quite simple, which makes it a good API to use as an example. The basic concept of streaming data in Initial State is as follows:

• When you create an Initial State account, you get a private account streaming access key (e.g. ist_TS0QPEcCQ8CLiahhFmB5EbEzPrCZACdr).
• Data is streamed into a data bucket (Initial State’s organizational terminology … more info), which has a bucket key you specify when you create the bucket (e.g. abc123).
• These two keys give you the ability to write data into that specific data bucket through Initial State’s Events API at any time. That means you can start/stop/append data from any number of applications as long as you have these two keys.
• These two keys do NOT give you the right to read data for security reasons.
• A data bucket can contain any number of data streams — a collection of time-series data points with a common name (e.g. temperature).
• A dashboard is automatically created for you in your Initial State account when you create a data bucket and send data into that bucket.

You simply need two keys, some data, and knowledge of how to use the Initial State Events API to create a live updating, customizable dashboard.

Method 1: A Python Module

Once you have settled on a data service, the next thing you should do is check for an existing Python module you can use to interface their API. There’s a good chance a well-written, well-maintained module will contain several goodies that will save you lots of time and lines of code.

The Initial State Python Data Streamer module is a good example of what you want to find (github repo and documentation). You don’t have to look at any API documentation to use the Initial State events API if you utilize this module. One line of code will create/append to a data bucket and one line of code will push your data to that destination.

To install the ISStreamer module on a Unix based system:

sudo pip install ISStreamer

Below is an example script that uses the ISStreamer module to create a data bucket and send the number 7 into the data stream myNumber. You simply need to place your account access key on line 3 before running.

You can modify the script to stream different numbers into the myNumber data stream or create additional data streams such as

streamer.log("myMessage", "hi")
streamer.log("myLocation", "44.5013406,-88.0644023")

Log into your Initial State account and click on the newly created data bucket to watch the dashboard update as you run your script.

Your dashboard should look something like this

The ISStreamer module has another built-in feature that is quite handy — a data buffer. Initial State’s Events API has a limit on how many API calls you can make in a given period of time (30 calls per 10 seconds). But, you can send up to 10 events per API call (more info). Instead of making an API call each time you call streamer.log(), data is timestamped and buffered until you either have 10 events in the buffer or you manually flush the buffer using streamer.flush(). This lets you more efficiently interface the API and stay within your account rate limits. Coding up our own data buffer would be a pain. This is a good example of the kind of extra functionality a great Python module should contain.

Method 2: Interfacing the API Directly with JSON

The first step to interfacing an API directly is to find well-written API documentation. The Initial State Events API is documented at https://initialstateeventsapi.docs.apiary.io/# . Apiary is a fantastic API documentation and testing service. You can even make example API calls in the built-in Apiary console. The Initial States Events API documentation contains descriptions and code examples for every publicly available API endpoint. These examples will provide the foundation for writing our own API implementation.

The first function we need to write is one that will create a data bucket. If you scroll down to the Event Buckets section (https://initialstateeventsapi.docs.apiary.io/#reference/event-buckets/buckets-json/create-a-bucket ), you can find all of the information you need to programmatically create a data bucket. Click on the Create a Bucket link and choose Python as the language to see example code.

The example code gives practically everything you need. Since the example code is using the urllib2 library, we will write our own createBucket function using the analogous requests library:

• The requests and json libraries are imported into this script on lines 1, 2. If either are missing, use pip to install them.
• You need to specify your account access key on line 4.
• The createBucket function takes in two parameters, bucketKey and bucketName, both of which are specified on lines 5 and 6.
• The required API endpoint documented in the API spec is on line 9, https://groker.init.st/api/buckets .
• The required headers are specified in a dictionary starting on line 10.
• The data payload is the bucketKey and bucketName and specified in a dictionary starting on line 15.
• The actual HTTPS POST is made on line 19. Notice we are converting the dictionary to a json object using json.dumps().
• The result is printed to the screen on line 23. A 201 response means a new data bucket was successfully created. A 204 response means the bucket already exists. A 400 response means you did something wrong and it failed (all responses and their meanings documented in the API docs).

Run the createBucket.py script and make sure a new data bucket shows up in your Initial State bucket shelf.

Next, let’s create a function that pushes data to our newly created data bucket. Scrolling down to the Events JSON section of our documentation (https://initialstateeventsapi.docs.apiary.io/#reference/event-data/events-json/create-a-bucket ), we can click the Send Events link and ensure Python is the selected language to see example code we can follow.

Just like before, the example code given is pretty extensive. Let’s use this code to create a streamData function using the requests library:

• You need to specify your account access key on line 4 and bucket key on line 5.
• The streamData function takes two parameters, name and value. name specifies the data stream and value specifies its value.
• Notice the API endpoint changed on line 8 to https://groker.init.st/api/events .
• The data payload we are using is simply a single key/value pair of name and value. The example code also specifies a timestamp in either epoch or iso8601 format, but this is optional.

Run the script to push data into our newly created data bucket. You can change line 22 to send different values to the myNumber data stream or create new data streams by simply using a new name.

Method 3: Interfacing the API Directly with URL Parameters (no-JSON)

Some data APIs implement a break in the HTTP/1.1 spec to simplify sending data. The address bar of your web browser performs a HTTPS GET whenever you enter an address and hit enter. This can be quite a convenient method for sending data even though you are doing a GET instead of a POST (this is the referenced “break in the HTTP/1.1 spec”). The Initial State Events API takes advantage of this by giving you the ability to send data into a data bucket via simple URL encoded parameters.

To create a data bucket, the URL format looks like the following:

https://groker.init.st/api/buckets?accessKey=YOUR_KEY&bucketKey=BUCKET_KEY&bucketName=NAME

This URL format is documented at https://initialstateeventsapi.docs.apiary.io/#reference/event-buckets/buckets-no-json/send-events . We can build this URL and make a HTTPS GET in Python as follows:

• You need to specify your access key, bucket key, and bucket name on lines 3–5.
• The complete url is constructed on line 8.
• The HTTPS GET call happens on line 9.

Run this script and make sure a new data bucket is created in your account.

Sending data to this bucket using URL parameters looks very similar to creating a bucket. The URL format looks like the following:

https://groker.init.st/api/events?accessKey=YOUR_KEY&bucketKey=BUCKET_KEY&signalName=NAME

This URL format is documented at https://initialstateeventsapi.docs.apiary.io/#reference/event-data/events-no-json . The Python code that builds this URL and sends data is as follows:

Place your access key and bucket key on lines 3 and 4. Run this script and make sure data is pushed into your data bucket.

Conclusion

The number of available data APIs will only grow and grow over time. Using Python for data analysis and data streaming is very useful. The examples in this tutorial should give you a quick start to interfacing APIs similar to Initial State’s Events API. Great APIs are a blast to work with. Leveraging Python modules such as requests, json, and ISStreamer can significantly reduce coding time and frustration.