Easyread
Published in

Easyread

Event-Driven with PubSub and Golang

Photo by pexels

Introduction

Hi everyone! Still with me to share some experience, thought, or opinion about technology-related with the software engineering field. In this article, I want to share how to create an Event-Driven with Pubsub and Golang!

But…what is the definition of Event-Driven?

Basically, unlike API-Driven which the request runs synchronously and in real-time, Event-Driver will be running based on an event, asynchronously and can have a delay in the request. Still confused? Let’s have an example in the Use Case section.

For the detail, maybe Event-Driven architecture definition by Wikipedia can give more context.

Use Case

In the first case, let’s talk about API-Driven architecture. For example, we have an App, that needs to call an API. The API needs to send a request on multiple services. This data flow will run synchronously, which means the response time will accumulate based on the response time of each service.

The advantages we can easily do tracking, the data flow is clear and the response will be returned in real-time. However, with this architecture, the response time might be not fast (depending on each service). Also, there’s a possibility hard to scale, for example adding a new Service, and might be affected the current relationship between Services.

This data flow below will help to understand how’s API-Driven works:

API-Driven Architecture Data-Flow

Next, with Event-Driven architecture, the differences there will be an Event between API and Services. API will publish an event, and each service will get the data from the event and this can be run asynchronously.

The advantages will be opposite from API-Driven architecture disadvantages such as the response time can be faster and easier to add new Service without breaking the others. However with this architecture, the data flow might be unclear, and there’s a possibility to have a delay in the request.

Event-Driven Architecture Data-Flow

PubSub Model

Because we will use PubSub for this example, better to take a look at PubSub architecture design. From the official documentation about Event-driven architecture with Pub/Sub, here is the architecture diagram that also describes the difference between the message-queue model and Pub/Sub model:

Event-driven architecture with Pub/Sub by Official Documentation

In the preceding diagram, a message-queue model is compared with a Pub/Sub event-stream model. In a message-queue model, the publisher pushes messages to a queue where each subscriber can listen to a particular queue. In the event-stream model using Pub/Sub, the publisher pushes messages to a topic that multiple subscribers can listen to. The differences between these models are described in the following sections.

Configure Service Accounts

Before implementing the code, we need to do some configuration such as generate the Service Accounts, create PubSub topic+subscriber in the GCP console.

  1. Go to Service Accounts from IAM & Admin menu.
Go to the Service Accounts submenu

2. Create a new Service Account.

Create a new Service Account

3. Fill in the service account name, modify the service account ID (if needed) then go to the next step.

Service Account Name & ID

4. Because we will only use the PubSub service, the role is also only for accessing PubSub. Then go to the next step by clicking Continue and Finisht the setup.

Grant Roles to Service Account
Go to the next step
Finish the setup

5. Next step is exporting the Private Key. Go to Manage keys from the Actions (ellipsis) menu.

Manage Keys

6. Create a new Key with type JSON the download it. Save the key in a safe place.

Create a new Private Key
Create Private Key JSON
Private Key Downloaded

Configure PubSub

After finishing with Service Actions, the next configuration is the PubSub configuration.

  1. Go to Topics submenu under Pub/Sub menu.
Go to Topics submenu

2. Create a new Topic ID, let’s give a simple name like medium . After the Topic ID is created, the Subscription ID with the name medium-sub will automatically be created.

Create a new Topic ID
Subscription ID automatically created

Code Implementation

After finishing configuring the Service Accounts and Pubsub topic+subscriber, it’s time to code!

The completed code can be found on my repository here:

Basically, the code for publisher copied from the official documentation in this link https://cloud.google.com/pubsub/docs/publisher#go_1

After some modification with the HTTP server, the final code for the publisher will be like this:

The important part of the publisher code is

pubsub.NewClient(ctx, projectID)

and

t.Publish(ctx, &pubsub.Message{
...
})

Both of the codes will open the client connection to PubSub and Publish the message.

The code for the publisher also copied from the official documentation in this link https://cloud.google.com/pubsub/docs/pull#go

The final code for living subscribing will be like this:

For the subscriber, the important code is quite the same, first for getting the client connection to PubSub:

pubsub.NewClient(ctx, projectID)

and for receiving the message and doing Ack action. Act action means for this subscriber, the message will be acknowledged and will not redeliver.

sub.Receive(ctx, func(_ context.Context, msg *pubsub.Message) {
...
msg.Ack()
})

Testing

Next to the testing part, we will be running both publisher and subscriber then publish a new message in order to check whether the message was successfully received or not.

Refer to my repository, copy and paste the Private Key of Service Accounts to the root of the project, rename it to service_account.json

Then configured the environment variable for storing the key path:

export GOOGLE_APPLICATION_CREDENTIALS=/path/service_account.json

Run the publisher and consumer with this command:

go run ./cmd/publisher/main.go
2022/03/24 11:50:09 server is starting at :8080...
go run ./cmd/subscriber/main.go
2022/03/24 11:50:35 subscriber is starting...

Trigger the publish message endpoint by opening the endpoint http://localhost:8080/publish on the web browser or by curl:

curl --location --request GET 'http://localhost:8080/publish'

After successfully publishing the message, it should be getting a response:

{
"message": "published message with custom attributes; msg id: 3923499137405184"
}

and the subscriber should be received:

2022/03/24 11:52:47 got data: "Hello world!"
2022/03/24 11:52:47 got attribute: map["origin":"golang" "username":"gcp"]
Publisher and Subscriber running and received a message

Yeay!! The PubSub works, and if we want to add more subscribers, just add a new subscription id to the topic through the Pub/Sub GCP console, and run another subscriber service with the different subID

Closing

By this experiment, implementing Event-Driven architecture with Pub/Sub isn’t really difficult. Pub/Sub already provided the system, and we just need to configure the topic, subscription, and the other configuration we need.

If we don’t want to manage the event system ourselves; deploying, maintaining, scaling, etc, choosing Pub/Sub is a good choice.

Thank you! I hope this article is useful for you!

--

--

--

Easy read, easy understanding. A good writing is a writing that can be understood in easy ways

Recommended from Medium

Clean Code 101

Leetcode-Graph

CS373 Fall 2021: Andrew Wu

How to Participate in Rainbow’s Testnet?

How to create our own text animation using SVG and CSS

Embracing Microservices

Using multiple player prefabs with UNET in Unity

ASP.NET Core with Docker — Installation guide and example

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Bismo Baruno

Bismo Baruno

Software Engineer | Traveler | Guitarist | J-Lovers

More from Medium

Writing Unit Tests with Testify

Go with gRPC, SQL, Dependency Injection(Wire) + standard project structure, and more: Introduction.

GoFrarme 101: Add ratelimit middleware

NSQ example application in golang