Event-driven APIs — Understanding the Principles
What are event-driven APIs? How do they differ from REST APIs? How to use the Webhooks, WebSockets, and Server-Sent Events to build them?
In this article, you’ll learn the foundations of event-driven APIs, how they interact with consumers, the technology choices to build them, and how to document them with AsyncAPI specification.
Polling is dead. We must move on
We, as information consumers, have a craving desire to know things as they happen.
Where is my package right now? What’s the score of the game? How is Dogecoin performing today? Likewise, the list goes on. A majority of Internet users today would like information to be pushed towards them rather than pulling.
As an application builder, how do you push information to users as they happen?
Today, a good amount of Internet applications are powered by HTTP APIs. Their interaction model is request-response driven and synchronous. The only way for a consumer to know what’s going on the server is to keep polling the server. Polling is dreadful, wastes valuable CPU cycles on both ends.
So, there must be some way to improve this interaction model.
The modern user experience
But, some exciting developments are happening in this space lately.
If you happen to use applications produced by major league Internet companies such as FaceBook, Instagram, or Uber, you must’ve experienced the seductive real-time interactions offered by them. For example, Facebook notifies you instantly when someone liked your content. Uber tells you where your ride is at the moment and how long it’ll take to get there.
These interactions glue users to their platform, improving the engagement never before.
Now, our question is, how do they do that? If they can, why can’t you? Let’s find out.
The foundations of event-driven APIs
The REST-style HTTP APIs often have a unidirectional and synchronous interaction model with their consumers. To get the latest information, the consumers always have to poll the API backend.
What if we turn this model upside down and allow the backend to notify the client when something interesting happens?
Well, that is the basis of “Event-enabled” APIs. They are often called “asynchronous,” “push,” or “streaming” APIs as they keep pushing information to the client, in contrast to the polling.
An event-driven API must offer two capabilities to its consumers.
- A mechanism to allow consumers to subscribe to events of their interest.
- Deliver events to subscribed consumers in an asynchronous manner.
Based on these, we can define an interaction model like the following for event-driven APIs.
1. The client subscribes with the API
At this point, the client of the event-driven API registers its intent to receive asynchronous updates with the API. That is called the subscription. Upon subscription, the client often specifies an endpoint to which the API should post update events.
2. Asynchronous event delivery
When something interesting happens at the backend, API delivers that to all subscribed clients in the form of an event, asynchronously.
An example to understand
Before we dive into the technical nitty-gritty, let’s take an example to understand how event-driven API works in real life.
Imagine you are lodging a complaint about a power outage through a web application. Upon receiving the complaint, the web application takes you to the ‘My Complaints’ page, where it displays all the complaints made by you along with their statuses.
Let’s assume that the status of a complaint transitions as follows.
As an impatient customer, what can you do with this web application at this point? Keep refreshing the “My Complaints” page until you see any change in the complaint’s status, right?
Well, that’s painful and not a good user experience for sure. Now, let’s see how to build the same application with event-driven APIs.
With event-driven APIs, the approach will be very similar until you submit the complaint. As usual, the application will take you to the ‘My Complaints’ page. But this time, the application shows you a notification whenever the status of the complaint transitions to the next one.
The above is one example that where the Internet is heading these days. There, you didn’t have to refresh the page to see s status change in your complaint. The backend kept sending that to the browser.
As a customer, you were calm and knew that things are moving at the other end.
Building event-driven APIs
I assume now you have a solid understanding of event-driven API capabilities and the principles in which they operate.
Now, the question is how to build them.
Several protocols and frameworks exist today to help you build event-driven APIs that push out events to your consumers. However, regardless of the implementation, the above discussed foundational interaction pattern remains the same.
That is, as an API provider, you should let your consumers subscribe to APIs. Secondly, you should deliver event notifications to them asynchronously.
There are several technology choices to consider when implementing an event-driven API. Each choice will depend on your use case, skillset, and infrastructure restrictions. Webhooks, WebSockets, and Server-Sent Events (SSE) are few prominent choices we can see today.
I’ll briefly introduce you to each technology in the coming sections and discuss how they fit into the interaction model we discussed. I suggest you read the excellent article written by Kristopher Sandoval for a deep dive.
Technology choice #1 — Webhooks
If you are building an event-driven API, perhaps the most straightforward way to get started is to allow your consumers to register a Webhook to receive event notifications from your API.
A Webhook is a publicly accessible HTTP POST endpoint managed by an event consumer. An event producer, such as an API server, can send event notifications to a webhook when something interesting happens.
You can think of Webhooks as reverse APIs. They completely detach the HTTP request and response from each other.
As an API provider, you have to consider two things when building event-driven APIs for Webhooks-enabled consumers.
1. Allow API consumers to subscribe to your APIs
Consumers can subscribe to your API by registering a Webhook URL as the callback.
The Webhook subscription itself can be managed as a REST resource, and it must include the list of event types and the subscription(callback) URL at a minimum level. It should also offer a way to unsubscribe from the API.
The following is the Webhook subscription request format offered by SurveyMonkey.
2. Deliver events to API consumers asynchronously
After sorting out the subscription, your next task is to deliver events to the consumers.
As the API provider, you can make an HTTP POST call to the consumer’s Webhook when something interesting happens at the backend. For example, a database record has been updated.
Webhooks forces the event consumer to establish a publicly accessible HTTP endpoint to receive events. It comes with other concerns like securing it with certificates, preventing DDoS attacks, etc. These could be a burden in the long run in terms of maintenance.
Moreover, Webhooks can’t be used to push event notifications to end-user consumers such as mobile and Single Page Applications (SPA) as they can not in possession of an HTTP endpoint.
Despite the above drawbacks, Webhooks can still be ideal for implementing a server-to-server event notification mechanism.
Because of its simplicity, Webhooks still serve as the first choice for building asynchronous APIs. It is backed by 100+ API providers who offer Webhooks-based APIs today. Kin Lane wrote about that sometime back.
Technology choice #2 — WebSockets
WebSockets is another protocol that you can use to build event-driven APIs. Unlike Webhooks, the WebSocket protocol allows for constant, bi-directional communication between the server and client, which means both parties can communicate and exchange data as and when needed.
Similar to Webhooks, you’ll have to consider two things when building an event-driven API with WebSockets.
1. Allow API consumers to subscribe to your APIs
To keep getting event notifications, the consumer has to establish a WebSocket connection with your API in the first place.
A WebSocket connection is established by making an HTTP call to the API and then asking for an upgrade on that connection. After that, the communication takes place over a single TCP connection using the WebSocket protocol.
2. Deliver events to API consumers asynchronously
When the API has something interesting, it can notify the consumer by writing a data frame to its WebSockets connection. From a programming point of view, it is very similar to writing to a socket.
The consumer who had already established the connection can parse the event data from the wire and update its UI accordingly.
WebSockets are far more efficient than WebHooks as they remove the HTTP header overhead by operating on the TCP layer. In the early days of WebSockets, they were constrained by the lack of support offered by browsers. However, a majority of browsers today support WebSockets natively.
If you like to get hands-on with WebSockets, you can refer to my article below.
Building a Real-time Sales Dashboard with WebSockets and Quarkus
A simple example to show how to use WebSockets with Quarkus
Technology choice #3 — Server-Sent Events(SSE)
Server-Sent Events, or SSE, is a communication protocol much like WebSockets, but with the implication of unidirectional data. SSE enables a browser-based consumer to receive a stream of event notifications sent from an API server.
Subscription and event delivery
The consumer subscribes to your API by creating a new EventSource object and passing the URL of an endpoint to the server over a regular HTTP request. After that, the consumer keeps listening for a response with a stream of event notifications.
If there are no more events to send, the server (your API) can terminate the connection. Or, the server can open the connection until the consumer closes it explicitly.
Being a unidirectional protocol, SSE is a good choice to build your event-driven APIs if you think about less bandwidth consumption and not maintaining long-lived HTTP connections to consumers. However, there are some challenges related to security, as there’s no way of challenging API consumers for tokens.
Documenting event-driven APIs with AsyncAPI
A good API definition is complemented by comprehensive documentation and a set of language-specific code generators. REST APIs are meeting that need with the OpenAPI specification. Fortunately, for event-driven APIs, we have the AsyncAPI specification.
The AsyncAPI specification is a machine-readable document that documents and describes your event-driven APIs. It is not only just a specification but a rich ecosystem full of code generators, validators, and test generators.
AsyncAPI is designed along with the same elements of OpenAPI and shares many common constructs to simplify the adoption, but it also comes with additional features to accommodate eventing. It supports a wide variety of messaging protocols and transports (such as AMQP, MQTT, WebSockets, Kafka, JMS, STOMP, HTTP, etc.) and event schema formats. Therefore, the API definition will contain the event payload definition, channel name, application/transport headers, protocols, and other eventing semantics to connect, publish, and subscribe to the API. — Dakshitha Ratnayake
If you are starting your event-driven API journey today, I strongly suggest you follow the AsyncAPI initiative.
Takeaways and what’s beyond this
Event-driven APIs add rich user experience to your end-user applications in usability, performance, and responsiveness. A typical event-driven API must offer two capabilities to its consumers. First, it should let them subscribe to your APIs. Secondly, you should deliver event notifications to subscribers asynchronously.
There are several technology choices to pick for building event-driven APIs. Webhooks, WebSockets, and Server-Sent Events (SSE) are some leading examples. Also, it is essential to document event-driven APIs for a wider community adoption, maintainability and automate the process of implementation code generation. AsyncAPI specification serves that objective to the best.
But the critical point to note here is that event-driven APIs do not limit to their implementation. They should always be accompanied by value additions like an API Management system, a centralized event broker, and an enterprise integration solution to manage the lifecycle of APIs, manage event subscriptions, and provide connectivity between systems.
In her article Event-driven APIs in Microservice Architectures, Dakshitha Ratnayake breaks down the recommended reference architecture for adopting, building, and operating event-driven APIs within your organization.
Dakshitha Rathnayake — Event-driven APIs in Microservice Architectures
Emmanuel Picard — Event-driven vs REST API interactions
Kristopher Sandoval — 5 Protocols For Event-Driven API Architectures
Lukasz Gornicki — WebSocket, Shrek, and AsyncAPI — An Opinionated Intro