IoT considerations — APIs and real time transports

The real trick in an Internet of Things product is moving data in an efficient and fast way — so at the heart of any IoT implementation is the API.

Done correctly, it can simplify your code, speed up development, and if you ultimately deliver data seamlessly to the user, it can actually enhance your user’s experience.

Done incorrectly, it can tie you up in knots, make it a hassle to support and give your interface a slow, laggy feel, hurting how your user perceives your product.

[If you missed the previous articles in the series about IoT considerations, the overview with links to all the other content is here.]

What is an API?

An API provides a predictable interface through which clients can talk with a server, in order to pass data or information.

It abstracts away the infrastructure behind it — to the client the API is all there is, and is its’ only way of interacting with the server.

What happens beyond that point is not really of interest to the client — in fact, the client may not even be talking with the actual server, it might only be talking with an intermediary server or cache in the middle.

APIs used on the web can take many forms. Let’s take a look at a few.

Basic APIs

A very basic API could be created by publishing a http url, which when requested , returns a some plain text CSV with some dynamically generated data in it. This is very simple, and not very useful — but it is an API of sorts.

In the late 1990s, Microsoft were working on a better way of connecting together clients and servers. Out of this, XML-RPC emerged as a way to execute Remote Procedure Calls using XML as the language of data transfer.

In effect, this means the client sends a piece of XML describing the function they want to run, and the parameters of the function, to the server. The server runs the corresponding function and returns the result.

It’s simple, lightweight, and contains some simple conventions which make it predictable. However, it’s designed for doing simple things and passing simple data — not complex objects and data like we often use today.

SOAP

SOAP (Simple Object Access Protocol) is an extension of XML-RPC and is much more powerful but also much more complex.

SOAP allows for Uniform Resource Identifiers (URIs) which can be used to divide up the API into services applicable to different types of objects.

Separately grouping the methods relevant to objects or different blocks of functionality is a more logical approach and leads to consistency and re-usability. For example, one URI could relate to a person, and another to a house.

Many enterprise size organisations still expose a SOAP API, but as the modern, javascript enabled web has become dominant over traditional client-server software, it has become overtaken by newer technology.

REST

REST (Representational State Transfer) is a concept, not a protocol. But it is the basis of the most widely used form of API today. RESTful APIs are widely used in the modern web, and data transfer usually takes place using JSON or XML over HTTP.

REST is a stateless concept — e.g. that the client will connect and consume the API on demand, passing any data which is relevant to the request at time of use. It does not maintain a constantly open connection so it is very scalable. The server does not care if the client goes on the missing list or only consumes the REST API occasionally.

REST also has the concept of a Uniform Resource Identifier, which lets you specify different resources which the client can consume under different URLs.

REST APIs typically use methods of the HTTP specification to perform different actions. For example — POST, GET, PUT, DELETE can be logically mapped to SQL CREATE, SELECT (READ), UPDATE and DELETE functions.

This is known as CRUD and it means that everything you might want to do to a piece of data stored on a remote server can be done predictably through a REST API.

As it uses HTTP, any device which can connect ot the web and request a web page can use a REST API.

Problem — APIs are push/pull

One of the issues of using a REST or SOAP API is that these are inherently push/pull type technologies.

That is, when your client device needs to send up data it must specifically connect to the server and upload it. When it want to download changes from the server, it must again specifically connect to the server to do it.

This is fine if your data changes slowly and you are not expecting frequent updates from the cloud, but what if something happens in the cloud (server side) that you need to know about immediately? Unfortunately, you’re out of luck. The server has no way of contacting the client to let it know that a change has occurred.

Instead, you must wait for the client to connect in and check for changes. Real time transport of data needs something else

Real time data transport

Moving data around instantly sounds tricky. You need to maintain an open connection from the server to the client at all times. That sounds wasteful — how will I scale my system if the server is tied up handling lots of permanently connected clients?

Well — that may have been true in the past, when hardware was more limited and some providers even limited the max number of TCP connections the OS could accept.

But it’s better to think of modern real time data transports as more like chat rooms. They are lightweight, idle for much of the time, and only passing short snippets of data [I hope you thought through your data model…]

Real time transport libraries

Thankfully, there are plenty of libraries for doing this. You could look at MQTT, XMPP, ØMQ, RabbitMQ, WebSockets. This is not an exhaustive list but is a good place to start. I’m not going to go in detail into any of them, but I will describe the concepts in general.

Note — different libraries will have different terms describing the concepts — and some will have extra layers — but the concepts are similar.)

General concept

These frameworks are all designed for passing short snippets of information (messages and data) across the web.

Your cloud server can publish topics (like chat rooms) which other clients can subscribe to. If a message comes in about that topic, it will relay it to other clients which are interested in that topic. An example for a topic might be sensor data.

So, your client could register for that topic and then send a message saying a temperature had updated, and attach the data relevant to the change (which sensor, what is the new temperature).

Your server could do something with the data, but it can also relay it to the other clients who are interested in the topic.

The clients who ultimately receive the message know what the topic is (in our example — sensor data), what the message is (temperature updated), and the relevant data (e.g. what is the new temperature and which sensor did it happen on).

As a result, it’s easy for it to do something based on the message — just register a handler function and write your code from there. it could even respond with a message of its’ own.

The server could also publish a topic about something else which clients may be interested about, e.g. if another device connects or disconnects. The server simply sends a message to interested clients when it happens. Presto, real time data transfer!

What next?

It’s worth reading up on the various technologies I mentioned — not all will work exactly as described, but it should give you a rough outline of what to expect.

Most are open source, and some (e.g. WebSockets) are IETF specifications, e.g. are defined in the same way http is, or specifications with other organisations.

Which you choose depends on what you want to do, and what your priorities are (speed, overhead, reliability, long term stability of spec).

Some are simple to use but have pitfalls which you will need to take steps to mitigate against (e.g. is the data guaranteed delivery or just best attempt?). I’ll discuss the problem of lack of guaranteed delivery another time.

Others will have all the bells and whistles but may be more complex to work with.

Conclusion — A mix of both

In reality, your API may be a mix of both. REST APIs are great for passing bulk data, but today users are getting accustomed to real time updates in everything they use.

So there’s a little more to think about with your API than it may seem at first glance — make sure you think it through before you get started!

Next time out, I’ll explore the bumpy and varied landscape of cloud infrastructure, Software as a Service, Platform as a Service and Infrastructure as a Service.

Update — Part 6 is now available: Cloud Services — Iaas, PaaS, SaaS, build your own

—

Des Flynn is CTO of Lattice Research, who help companies to design, build, deploy, operate and service innovative and cost-effective IoT control systems to meet their customer’s needs. More information at www.lattice.ie