Distributed Message Streaming in Golang using Nats JetStream

Hello there, in this post I will briefly talk about Nats JetStream, and then I will show an example of cross-service message streaming using Golang. This is my first Medium post and I’m so excited about it. I hope this and what I will write next will be useful to you.

What is Nats JetStream?

nats.io

To understand Nats JetStream, first let me talk about the Nats ecosystem and its evolution (Core Nats → Nats Streaming → Nats JetStream).

Core Nats is a cloud-native, high-performance PubSub messaging system. It uses the “at most once” delivery model. So, there is no guarantee of delivery or persistence of the published messages.

Nats Streaming (STAN) uses the “At least once” delivery model. Message persistence and delivery are guaranteed. But because of its architecture;

• Cannot scale horizontally without limits
• Cannot Nack messages
• Does not have “Pull-Subscribe” for consumers, etc.

The NATS Streaming Server is being deprecated. Critical bug fixes and security fixes will be applied until June of 2023. So Nats JetStream is recommended for new applications.

Nats JetStream includes new persistence features and message delivery policies. Horizontally scalable and optimized for very large data sets. You can also have a look at Design Goals in the Nats JetStream documentation.

We are actively using Nats JetStream in the production environment at the company (that have 150M+ users in worldwide) I work for now. We have not encountered a problem so far. Before you start using JetStream, I recommend that you take a look at the Nats Server and Nats Go Client issues. There may be situations that concern you.

Let’s start with the Nats JetStream installation. Then we will do a small demo using the Go client.

Installing and Running the Nats Server and JetStream

NATS philosophy is simplicity. Installation is just decompressing a zip file and copying the binary to an appropriate directory; you can also use your favorite package manager.

I will install it with brew. You can use other installation ways.

brew install nats-server

You can simply run the nats-server executable. We should use -js flag to enable JetStream and -m to set http monitoring port. You can also use a configuration file.

nats-server -js -m 8080

Nats Server is running with default port 4222

Setting Up Nats JetStream Go Client

Let’s start by creating a JetStream context to manage streams (create, publish, subscribe). I will use Nats Go Client v1.16.0 and Go v1.18.

PublishAsyncMaxPending sets the maximum outstanding async publishes that can be inflight at one time.
Default URL is nats://127.0.0.1:4222

Now we can create a new stream and publish a message to the stream. We created a REVIEWS stream to be used for REVIEWS.* subjects.

Let’s use a store’s dummy user review data. Assume we have two different services, one that publishes and the other one consumes user reviews.

Create a Review model:

Publishing Messages to Stream

Now we can publish dummy reviews using Nats JetStream. I put in random wait milliseconds to see the console output more clearly. We will publish all the review data on the REVIEWS.rateGiven subject. By the way, you can send Structured Data instead of marshaling and unmarshalling.

Understanding Consumer Types

Nats JetStream provides two types of consumers, pull-based and push-based.

Pull-based Consumers pull a batch of messages from a subject. It has to ask the system for the next available message, so you can scale it according to the availability of the services.

Push-based Consumers control is in Nats Server. The server sends messages to push consumers.

Nats JetStream also supports Wildcard Subscriptions:

jetStream.Subscribe(“REVIEWS.*”, func(m *nats.Msg)

Consuming Messages From Stream

Let’s create a push-based consumer and subscribe to the REVIEW stream. We will get the review dummy data from the subject REVIEWS.rateGiven which we published before. If we want, we can publish REVIEWS.rateAnswer as an answer.

Now it’s time to put the pieces together :) When we run both the consumer and publisher simultaneously, we should get something like this:

2022/06/28 00:24:19 Consumer => Subject: REVIEWS.rateGiven — ID:58c03ac18060197ca0b52d51 — Author: 58c039018060197ca0b52d4c — Rating:5
2022/06/28 00:24:19 Publisher => Message: I tried this place last week and it was incredible! 
2022/06/28 00:24:20 Consumer => Subject: REVIEWS.rateGiven — ID:58c03af28060197ca0b52d53 — Author: 58c03ada8060197ca0b52d52 — Rating:1
2022/06/28 00:24:20 Publisher => Message: hipsters everywhere

You can find the entire source code here. You can try wildcard subscription and pull subscription with minor changes in the code.

GitHub - ebubekiryigit/go-nats-jetstream-example

Summary

My favorite part is that it is simple and easy to use. It is also a big plus that it is horizontally scalable and high-performance. When I create a new microservice, Nats JetStream is the first technology that comes to my mind in the PubSub structure.

If you have a problem, they respond very quickly in Slack groups.

References

• https://docs.nats.io

Thank you! :)