Event-Driven APIs with Apigee, Pub/Sub and Cloud Run
Introduction
A Gartner report from 2017 predicted that 50% of managed APIs will support event-driven IT use-cases by 2020.
Event-driven APIs support many use cases where the RESTful request-response model may not be a good fit. They don’t aim to replace REST APIs, but they rather supplement each other in enabling enterprises to accelerate their API-first strategy.
One can’t talk about Event-driven APIs without talking about Event-driven Architectures (EDAs). The following diagram illustrates the core concepts of EDAs.
In EDAs, event publishers and event subscribers/consumers are decoupled from each other by using a message broker. The latter acts as the backbone of the architecture and allows asynchronous communication to happen between publishers and subscribers.
Note: if you aren’t familiar with the concepts above, I recommend checking this post.
In the next section, we will discuss an architecture that shows how Apigee can empower EDAs by managing both REST APIs and event-driven APIs. This architecture makes use of Apigee, Pub/Sub , and Cloud Run as described below:
- Pub/Sub will be used as a message broker.
- The Apigee Platform (Apigee X or Apigee hybrid) will expose REST APIs for publishers to publish messages to Pub/Sub topics. It will also expose Async APIs (Websockets) that can be “consumed” by “Subscribers”.
- Cloud Run will host services that perform “Pub/Sub to Websockets” transaltion.
Architecture
The following diagram shows how Apigee and GCP can be leveraged for managing both REST APIs and Async APIs (event-driven APIs).
From left to the right, the architecture above is composed of the following blocks:
Ingestion Layer
It allows event publishers to publish messages or events to a specific topic. This layer is composed of REST APIs that are managed and exposed through the Apigee API Platform.
Apigee allows the management of the complete API life cycle of these APIs (design, security, deployment, monitoring, and even monetization). For example, Apigee secures the APIs endpoints and makes sure that a given publisher is only allowed to publish messages to a specific topic. It also hides the complexity of the message broker from publishers allowing them to use a single interface (REST APIs) for publishing messages.
We recommend using the OpenAPI Specification standard to describe the APIs of the consumption layer.
Message Broker
It allows the decoupling of event publishers from subscribers/consumers. Pub/Sub will be used for this purpose.
There is a 1:1 mapping between Pub/Sub topics and the REST APIs. The mapping is done using the metadata of Apigee API products.
A service account will be used to authenticate Apigee requests to Pub/Sub. GCP IAM will manage the authorization part.
Adaptation Layer
It is composed of serverless services that run on Cloud Run. They pull Pub/Sub messages and send them to subscribers using Websockets. For further details on how this works, we recommend checking the solution described here.
It is important to note that there is a 1:1 mapping between a Pub/Sub topic and a Cloud Run service. In other words, each Cloud Run service subscribes to a specific Pub/Sub topic and then sends all the messages it receives to subscribers over Websockets.
Consumption Layer
It exposes Async APIs (over Websockets) that allows subscribers to subscribe to a topic and receive messages over Websockets. These Async APIs are “proxied” through Apigee. The latter takes care of authenticating the subscribers while enabling API lifecycle management. We recommend using the Async API Specs to describe these APIs.
The interactions between the different building blocks above are authenticated using GCP Services Accounts.
All the API endpoints (Ingestion and Consumption layers) can be exposed publicly through the GCP HTTP Global Load Balancer (GLB). Cloud Armor can be used to further secure the API endpoints, see here for further information.
Conclusion
The architecture above offers a great flexibility because it allows both publishers and subscribers to interact with each other without having to know the complexity of the message broker. All interactions are API-based, and this allows for supporting hybrid setups where subscribers and publishers:
- aren’t managed by the same entity or don’t belong to he same domain,
- aren’t deployed in the same environment (e.g GCP or On-prem).
Using Pub/Sub as a message broker allows a great scalability.
Leveraging CloudRun (a highly scalable serverless solution) will allow the Consumption Layer to scale seamlessly as the number of subscribers increase.
Last but not least, the architecture offers also the possibility for event publishers to directly publish to Pub/Sub without going through Apigee. Event subscribers can also subscribe to a Pub/Sub topics and pull messages from there without having to go through the Consumption Layer.
Note: the architecture proposed above is provided “as-is” for demonstration purpose only. It was not tested in a real production environment.
Demo
Here is a recorded demo.
