Last week, I was tackling a task subscribing to a PubSubTopic in our operator project, which I think is worth writing down. It gives me the same relief as reaching the top of a mountain. The bumpy path ahead, but you ride it out.
There were four “big stones” on the way, and I went on a smooth journey once I killed them. They were four “how to.”
- How to set up the Google PubSub subscription in Go?
- How to trigger the Kubernetes Operator after consuming the message?
- How to write the unit test?
- How to set the proper IAM?
Now, let’s go through the process with my code example of the solutions.
Google PubSub Subscription
PubSub is the message system on Google Cloud, similar to the commonly used Kafka in Java development, applying the producer/consumer mode to obtain messages in a completely decoupled manner, and implement the corresponding logic.
The detailed PubSub is so verbose that I will skip it. You can easily generalize from the messaging system you are using since their design ideas are basically the same. If you don’t know any messaging system, please read here.
I searched Google’s complex documentation system to seek some code examples. However, what I found was some related examples too simple to have any reference value. But there were still gains, the available Go library:
Let’s start with creating a client and a subscription object.
Then, we can call the
Receive method to consume the required event.
The above is just a template code, the actual logic must be much more complicated, but we can conclude as follows.
- Parse the message. It is normally JSON type and can be easily unmarshaled to specific ones.
- Handle errors. Some errors should have the re-try logic, and this is where we call
- Finish the process. Always it is calling
Ack()and making sure the system knows this message is delivered and processed.
To be noted, your code needs to be idempotent so that the result will not change no matter how many times you receive the same messages.
Trigger actions by new messages
This part involves Go’s channel, a potent tool.
Usually, consuming message logic will be run in a Goroutine. Therefore, there is nothing more reasonable than the design of triggering the business logic call through the channel when receiving new messages.
The second function is setting up the Kubernetes Operator controller, using a watcher to monitor the channel.
Test Pubsub Subscription
Test with emulator
After completing the business logic code, then it’s about writing unit tests.
Test PubSub is no easy task since ordinary mock doesn’t apply here when I want to test the whole workflow. Seeking help from Google documentation again, I find emulator. Unfortunately, only Java examples. And I failed to find the proper Go library even I dug into Google’s Github repositories. If anyone knows, please throw a link in the comment. Thanks in advance!
Eventually, I set up a test server for the unit test with
Upon getting the container instance, I could follow up on the
PubsubClient and create a PubSub topic.
Finally, I can write the test case code with everything set.
Test with built-in Server
In addition to the
emulator way, we can also rely on
SubscriberServerAPI to simulate the whole process, a solution I found in Google’s PubSub library and a better fit for cases when both publisher and subscriber tests are needed. See the complete code👉 here.
How to set the proper IAM
Now, I have the code and tests ready. What’s the next then? Oh right, the damn permissions.
I believe many may be confused by Google IAM permissions, and it is time to figure out PubSub related permissions and roles.
I spent some time scanning the crazy permission list to figure out what I need. My service account needs the
pubsub.subscriptions.consumepermission to subscribe to the topics. This permission is tied up to a set of roles starting with
roles/pubsub. Obviously, if you don’t own the topics, the suitable role is
roles/pubsub.subscriber. To be clear,
pubsub.subscriptions.get are not required here.
After figuring out what kind of roles needy, I can either ask the admin who owns the topic to add my service account manually in UI or simply run the below command in CLI:
gcloud projects add-iam-policy-binding project-id — member=’user:firstname.lastname@example.org’ — role=’roles/pubsub.subscriber
Because I already enabled declarative management in my cluster via Google Config Connector, it is super easy to add this IAMPolicy with the YAML below.
At the end
I listed all the crucial points of subscribing PubSub topics using Go, hoping it will be beneficial to anyone reading this article.
Thanks for reading!