Getting Started with Realtime

If you’re new to realtime APIs, this article will discuss the common mechanisms that APIs use to create realtime applications.

Unlike request/response APIs, which almost always use conventional HTTP exchanges, realtime APIs may use HTTP in unconventional ways or even use non-HTTP protocols. The four common realtime API mechanisms are listed below:

  1. HTTP Streaming
  2. HTTP Long-Polling
  3. WebSockets
  4. WebHooks

HTTP Streaming

Examples: Twitter, DataSift, Superfeedr, Gitter

HTTP streaming provides a long-lived connection for continuous data push. The client sends a request to the server and the server holds the response open for an indefinite length. This connection will stay open until a client closes it or a server side-side event occurs. If there is no new data to push, the application will send a series of keep-alive ticks so the connection doesn’t close:

{“tick”:1311694603,”status”:”connected”,”message”:”Waiting for data”}
{“tick”:1311694604,”status”:”connected”,”message”:”Waiting for data”}

There are many services that use HTTP streaming to push data continuously. For instance, Twitter’s realtime API uses HTTP streaming to push new tweets as they occur. It’s important to note that streaming connections are one directional and not bi-directional like a WebSocket. You may want a bi-directional connection if you to talk back to the server to look for other data sources.

HTTP Long-Polling

Examples: Dropbox, Amazon SQS, Livefyre, Consul

This technique provides a long-lived connection for instant data push. The server holds the request open until new data or a timeout occurs. Most send a timeout after 30 to 120 seconds, but it depends on how the API was set up. After the client receives a response (whether that be from new data or a timeout), the client will send another request and this is repeated continuously.

Long-polling is reliable because you tend not lose data with periodic requests. If any issues do occur, the next request will return the new data that the last one failed to retrieve. HTTP long-polling is not great for continuous pushes, though. Since there is a delayed response between requests, you can’t get second by second realtime data.

WebSockets

Examples: Blockchain, Coinbase, CBIX, Slack

A WebSocket provides a long-lived connection for exchanging messages between client and server. The connection is full-duplex, meaning data can be sent and received in realtime from both ends. This bi-directional connection is established through a WebSocket handshake. Just like in HTTP streaming and HTTP long-polling, the client sends a regular HTTP request to the server first. If the server agrees to the connection, the HTTP connection is replaced with a WebSocket connection.

To get a better understanding on how a WebSocket handshake is “accepted,” we must examine the HTTP header request:

GET ws://websocketexample.com/ HTTP/1.1
Host: http://example.com: 8000
Upgrade: WebSocket
Connection: Upgrade

The upgrade header distinguishes it from other HTTP requests. This header communicates to the server that this is a request for a WebSocket connection. If the server recognizes the upgrade, it returns an upgrade header in response:

HTTP/1.1 101 WebSocket Protocol Handshake
HTTP/1.1 101 WebSocket Protocol Handshake Upgrade: WebSocket
Connection: Upgrade

The WebSocket handshake is then complete! After that, the bi-directional stream is established and you can begin to exchange data. Services like Slack utilize WebSocket connections to allow users to send and receive messages in realtime.

WebHooks

Examples: GitHub, Facebook, Stripe, Zapier

A WebHook registers or “hooks” to a callback URL and will notify you anytime an event has occurred. You register this URL in advance and when an event happens, the server sends an HTTP POST request with an Event Object to the callback URL. This event object contains the new data that will be pushed to the callback URL. You might use a WebHook if you want to receive notifications about certain topics. It could also be used to notify you whenever a user changes or updates their profile.

Unlike the other realtime mechanisms we’ve discussed, WebHooks don’t use long-lived connections. It’s actually quite a simple setup because no additional tools are needed.

Which Do You Choose?

You can use any of these mechanisms to create your own realtime API. They all have their benefits and drawbacks to them as you can see. You don’t necessarily have to choose one either. Some services and APIs use a combination of these realtime mechanisms.

Pusher primarily uses WebSockets for bi-directional communication, but use HTTP-based fallbacks as well. Pushpin, a reverse proxy and open-sourced version of Fanout, allows you to implement HTTP streaming, HTTP long polling, or WebSockets services. It’s all about finding the one (or two) that work best for your application.

Conclusion

If you’re looking to learn more about realtime APIs in the meantime, feel free to check out Real Time API Hub. It goes over the basics, design guides, realtime API infrastructure, and more. Of course, I’ll be writing more about realtime in the upcoming weeks as well. So feel free to check out my profile as well. You can read my last DZone article about realtime APIs here.

A single golf clap? Or a long standing ovation?

By clapping more or less, you can signal to us which stories really stand out.