Using Microservices and Kubernetes Pods to hide Google Pub/Sub

Our event driven system runs on Google Cloud and the heart that makes it possible for us to pass all these events around is Pub/Sub, Google’s messaging service. In many ways this has been a great experience, but some things annoyed us. One of them was the fact that it’s a bit cumbersome to use, the SDK (in our case the Java one) is not all that intuitive, it’s in Beta (Alpha when we started using it), and things keep changing.

We tried to get around this by creating a library, where we created some abstractions to hide the nitty gritty details of the Pub/Sub SDK. This made it a little bit better, but we still had other problems. Transitive dependencies (a good reason to stay away from libraries) for instance. Our Beta version of the Pub/Sub SDK conflicted with the 1.0 version of our Datastore SDK.

On the bus to work one day I had an idea, why not put this (the Pub/Sub integration) in a microservice instead? We already do it for pretty much everything, so why not. We use the Kubernetes Engine and deploy all of our services as Docker containers. In Kubernetes there is a concept called Pod, which is a group of containers sharing network and storage. Two containers in a Pod can communicate with each other via localhost. Great, so then lets give all of our services that talks to Pub/Sub a buddy in form of a Pub/Sub bridge service running in the same Pod. No more direct interaction with Pub/Sub, just http.

The Pub/Sub bridge service can be configured with as many subscriptions or publishers as you wish (but for us it’s typically just one subscription or one publisher, our services are small after all).

It turns out we get quite a lot of benefits with this approach.

• The loose coupling means no transitive dependency nightmares.
• It also means that we could easily switch out Pub/Sub and use something else for message transportation.
• We become language agnostic, adding Pub/Sub integration for out Node.js services will be a breeze.
• Blackbox testing the business services becomes easier and more deterministic because we can now use synchronous http instead of asynchronous messaging.
• We can use the path based routing logic built into the web application framework which makes subscribing easier.

To support the last bullet point we have made it possible to use Pub/Sub message attributes in the url when configuring a subscriber, so you can use something like http://localhost:8080/{attribute:aggregateType}/events/{attribute:eventTypeType} and those placeholders will get replaced with the corresponding message attribute values.

So we can now have:

@PostMapping("/case/events/case_created_event")
fun caseCreated(@RequestBody event: CaseCreatedEvent) {
    // Do something
}

@PostMapping("/case/events/description_added_event")
fun descriptionAdded(@RequestBody event: CaseDescriptionAddedEvent) {
    // Do something
}

instead of something like:

override fun receiveMessage(message: PubsubMessage, consumer: AckReplyConsumer) {
    val eventType = message.getAttributesOrThrow("eventType")
    when(eventType) {
        "case_created_event" -> handleCaseCreated(consumer, message.attributesMap, convert(message.data, CaseCreatedEvent::class.java)) 
        "description_added_event" -> handleDescriptionAdded(consumer, message.attributesMap, convert(message.data, DescritionAddedEvent::class.java)) 
    }
}

A lot cleaner!

The Pub/Sub bridge service will ack/nack messages based on the response it gets from the http POST to the business service. Http 503 or connection issues and it will nack it (so the message will be re-delivered later), for any other error it will just log it and ack the message. You should always be wary when nacking messages, it can block up your topic if the message is in fact broken and can never be handled. Happened to us…

Ok, so that’s what we have done to make Pub/Sub provide all the value we want while being less in our face. We’re happy so far but we only just implemented it and we’ll see how we feel in a month.