Google Cloud Pub/Sub + AsyncAPI
I’ve been covering different aspects of AsyncAPI in my recent blog posts. In this final post of my AsyncAPI blog post series, I want to talk about how to document Google Cloud’s Pub/Sub using AsyncAPI.
AsyncAPI has pretty good support for Google Pub/Sub, thanks to contributions from Jeremy Whitlock, an engineer from Google, and the flexibility baked in AsyncAPI spec. Jeremy also has a nice blog post on this topic that you can read for more details.
Recap: Google Cloud Pub/Sub and AsyncAPI
Let’s first do a quick recap Google Cloud’s Pub/Sub and AsyncAPI.
Pub/Sub is a managed service in Google Cloud for asynchronous and scalable messaging. It has everything you expect in a messaging service such as producers, consumers, topics in a fully managed and globally available service.
AsyncAPI is an open source specification to define asynchronous APIs, similar to what OpenAPI (aka Swagger) does for REST APIs.
If your event-driven service in Google Cloud sends or receives messages using Pub/Sub, you can use AsyncAPI to document that service and its messaging schema, thanks to flexibility of AsyncAPI with its bindings.
Bindings in AsyncAPI
Bindings in AsyncAPI is a mechanism to extend AsyncAPI beyond its spec. One can have server bindings, channel bindings, operation bindings and message bindings. That’s how AsyncAPI supports different messaging systems such as Google Cloud’s Pub/Sub, Apache Kafka, Solace, etc.
Google Cloud Pub/Sub Bindings documentation for AsyncAPI is the source of truth.
Let’s go through an example to see how you can define an application using Google Cloud Pub/Sub with AsyncAPI.
The full example is in google-pubsub-sample.yaml.
Pub/Sub Endpoints → AsyncAPI Servers
First, you need to provide some info about your Pub/Sub application:
asyncapi: 2.6.0
info:
title: Google Cloud Pub/Sub Topology
description: AsyncAPI definition for Google Cloud Pub/Sub
version: 0.0.1Pub/Sub provides a globally available endpoint but it also have regional endpoints. You need to define those endpoints as server in AsyncAPI. Here’s how you can have the global endpoint as default but also restrict to region-specific endpoints:
servers:
cloudPubSub:
url: '{cloudPubSubEndpoint}.googleapis.com'
description: The API for Cloud Pub/Sub.
protocol: googlepubsub
variables:
cloudPubSubEndpoint:
description: The Cloud Pub/Sub endpoint region.
# Default to the global endpoint
default: pubsub
# Or, restrict to only the following region-specific endpoints.
# enum:
# - us-central1
# - us-central2Pub/Sub Topics → AsyncAPI Channels
Next, you need to define some channels to publish messages to or receive messages from. Channels in AsyncAPI map to topics in Pub/Sub and topics in Pub/Sub can have Avro, Protobuf, or no-schemas defined in Pub/Sub.
For example, here’s a channel with an Avro schema backed topic:
topic-avro-schema:
bindings:
googlepubsub:
topic: projects/your-project/topics/topic-avro-schema
schemaSettings:
encoding: json
name: projects/your-project/schemas/message-avro
publish:
message:
$ref: '#/components/messages/messageAvro'
subscribe:
message:
$ref: '#/components/messages/messageAvro'Notice how bindings is being used to provide Pub/Sub specific configuration.
Similarly, this is a channel with a Protobuf schema backed topic:
topic-proto-schema:
bindings:
googlepubsub:
topic: projects/your-project/topics/topic-proto-schema
messageRetentionDuration: 86400s
messageStoragePolicy:
allowedPersistenceRegions:
- us-central1
- us-central2
- us-east1
- us-east4
- us-east5
- us-east7
- us-south1
- us-west1
- us-west2
- us-west3
- us-west4
schemaSettings:
encoding: binary
name: projects/your-project/schemas/message-proto
publish:
message:
$ref: '#/components/messages/messageProto'
subscribe:
message:
$ref: '#/components/messages/messageProto'Notice that there are additional Pub/Sub settings like messageRetentionDuration or messageStoragePolicy that you can set.
Finally, you can also have a channel with no Pub/Sub schema:
topic-no-schema:
bindings:
googlepubsub:
topic: projects/your-project/topics/topic-no-schema
publish:
message:
$ref: '#/components/messages/messageNoSchema'
subscribe:
message:
$ref: '#/components/messages/messageNoSchema'Pub/Sub Schema → AsyncAPI Message Payload
Finally, you need to define the message schemas in AsyncAPI.
For Avro messages, you refer to the schema definition in Pub/Sub. You also define the payload inline for tooling and documentation:
components:
messages:
messageAvro:
bindings:
googlepubsub:
schema:
name: projects/your-project/schemas/message-avro
type: avro
contentType: application/json
name: MessageAvro
payload:
fields:
- name: message
type: string
name: Message
type: record
schemaFormat: application/vnd.apache.avro+yaml;version=1.9.0For Protobuf messages, you still refer to the schema definition in Pub/Sub but you cannot define the payload, as Protobuf is not supported in AsyncAPI yet:
messageProto:
bindings:
googlepubsub:
schema:
name: projects/your-project/schemas/message-proto
type: protobuf
contentType: application/octet-stream
name: MessageProto
# Proto is currently not supported in AsyncAPI, we're simply
# indicating that there's a payload here.
payload: trueFinally, message without schema can be defined as follows:
messageNoSchema:
contentType: application/octet-stream
description: A message having no schema
payload: trueThat’s it! I’m pleased to see how flexible AsyncAPI is in helping to document all sorts of event-driven services, including Google Cloud’s Pub/Sub.
This wraps up my 5-part blog series in AsyncAPI. Here’s the whole series for your reference:
Originally published at https://atamel.dev.
