Redis [Pub/Sub] Communication between microservices in Node.js

If you are not familiar with Event-Oriented Programming, I recommend that you read this basic article where I explain it

Event-oriented programming is an architectural concept in which systems are designed to respond to events or status changes asynchronously.

One way to apply this architectural concept is through the Publisher/Subscriber (Pub/Sub) design pattern.

This pattern involves event producers (publishers) emitting events on specific channels or topics, and event consumers (subscribers) subscribing to those channels or topics to receive and process the events of interest. This design pattern is a variation of the Observer pattern.

Redis is completely compatible with the Pub/Sub pattern.

It is a popular choice for implementing messaging and asynchronous communication systems in applications and distributed systems.

Key concepts to understand about Pub/Sub in Redis:

There are 4 key concepts:

1. Channel: A channel in Redis is a one-way communication medium to which clients can subscribe to receive messages. Channels are identified by their name, which is a string in Redis.
2. Publisher: A client that sends messages to a channel.
3. Subscriber: A client that subscribes to one or more channels. Subscribers receive messages published in the channels they are subscribed to.
4. Message: The information sent by the publisher to a channel. Messages are simply text strings, commonly sent in JSON format.

Below, you can find an illustrated example:

Disclaimer: Redis does not have different types of exchanges (Topic, direct, or Fanout) as found in, for example, RabbitMQ. In Redis, pub/sub is based solely on channels.
Note: In my next article I am going to tell you about RabbitMQ.

After this introduction, I am going to show you a basic practice example.

Requisites:

1. Have Node.js installed (preferably > 18)
2. Have Redis installed and this service is up.
   → I recommend that, if you have Docker installed, you run it. Once you’ve done that, I will provide you with a docker-compose.yml file, which you need to place in the root of the project to help you run Redis without the need to install it on your computer.

docker-compose.yml:

version: "3.7"

services:
  redis:
    image: redis
    restart: always
    ports:
      - "6379:6379"
    volumes:
      - redis_volumen:/bitnami/redis/data

volumes:
  redis_volumen:

Use the following command to start the Redis service:
→ This command should be executed in the root directory where the file named docker-compose.yml is located.

docker-compose up

Create a Productor (Service 1):

> mkdir publisher
> cd publisher
> npm -y init
> npm install express redis uuid

Afterward, we must create an index.js file, with the following code:

import express, { urlencoded, json } from 'express';
import { createClient } from 'redis';
import { v4 } from 'uuid';
const app = express();
const PORT = 3000;
const URL_REDIS_CONN = 'redis://localhost:6379';
const NAME_CHANNEL = "channel1";

app.use(urlencoded({ extended: false }));
app.use(json());

const redisClient = createClient({ url: URL_REDIS_CONN });
redisClient.on('error', err => console.log('Redis Client Error', err));

// Endpoint POST --> localhost:3000/testRedis
app.post('/testRedis', (req, res) => {
  try {

    if (!req.body?.message) {
      return res.status(400).json({
        error: 400,
        detail: "The message is mandatory"
      });
    }
    // Generate a message as an object with a unique ID, message, and date
    const message = {
      id: v4(),
      message: req.body.message,
      date: new Intl.DateTimeFormat('es-ES').format(new Date()),
    };

    redisClient.publish(NAME_CHANNEL, JSON.stringify(message));
    console.log(`Publishing an Event using Redis to: ${req.body.message}`);
    return res.json({ detail: 'Publishing an Event using Redis successful'});
  } catch (error) {
    return res.status(500).json({ detail: error.message });
  }
});

redisClient.connect().then(() => {
  console.log("Redis connected")
  app.listen(PORT, () => console.log(` Server listen on port ${PORT} `));
});

Create a Subscriber (Service 2):

> mkdir publisher
> cd publisher
> npm -y init
> npm install express redis uuid

Afterward, we must create an index.js file, with the following code:

import express from 'express';
import { createClient } from 'redis';
const app = express();
const PORT = 3001;
const URL_REDIS_CONN = 'redis://localhost:6379';
const NAME_CHANNEL = "channel1";

const messagesStorage = [];

const redisClient = createClient({ url: URL_REDIS_CONN });
redisClient.on('error', err => console.log('Redis Client Error', err));

redisClient.subscribe(NAME_CHANNEL, (message) => {
  console.log("Catching an Event using Redis to: " + message);
  messagesStorage.push(JSON.parse(message));
});

app.get("/messages", (req, res) => {
  try {
    return res.json({ messages: messagesStorage });
  } catch (error) {
    return res.status(500).json({ detail: error.message });
  }
});

redisClient.connect().then(() => {
  console.log("Redis connected")
  app.listen(PORT, () => console.log(` Server listen on port ${PORT}  `));
});

In both services, we set up an application using Node.js and Express. In the first service (Publisher), we include a ‘POST’ type endpoint that receives a ‘message’ in the ‘body’. Based on this message, it creates an object with the message, the date, and a unique ID to it. Then, it emits an event to Redis, passes the created object, and returns a JSON.

In the second service (Subscriber), we have a listener that captures all events with the same name as the channel. We also create a ‘GET’ type endpoint that returns an object with all the messages it has captured.

It’s important to note that this is an elementary example, and these events are stored in memory. If we restart the services, the data will be lost.

To test our application:

// Emit event - Execute from the terminal
curl --location 'http://localhost:3000/testRedis' \
--header 'Content-Type: application/json' \
--data '{
   "message": "Hi! I am Ale"
}'

/*
Response --> Publishing an Event using Redis to: Hi! I am Ale

Simultaneously, it has emitted an event to the channel named 'channel1' and
if service 2 is up and running, it listens for this event and executes 
the code of service 2.
*/

If you don’t want to execute curl from the terminal, feel free to use any tool like Postman.

Considerations when working with Redis using pub/sub.

If you subscribe to a channel in Redis from Node.js you must know that each event emitted from the publisher to this specific channel will be listened to by every subscriber to this channel.

For that, you must always take into account the following considerations:

1. Performance: Using Pub/Sub has a low cost in Redis, but if you have a large number of subscribers or messages, there could be an impact on the performance of the Redis server. It is advisable to have a thorough understanding of the requirements and the maximum quantity that we can handle.
2. Scalability: If you have to handle a heavy load of subscribers and messages, you can use a solution such as Redis clusters or high availability configurations to distribute the load.
3. Durable Messages: Redis does not provide persistent storage. We would need to use another mechanism for that purpose like RabbitMQ.
4. Message Loss: Redis does not guarantee message delivery in the event of server failure or outage. If you require this feature, you should use another mechanism with acknowledgment (ack) and retry logic, such as RabbitMQ.
5. Security: Ensure that only authorized clients can publish or subscribe to a channel by configuring security policies
6. Cleanup of Inactive Subscribers: Redis does not automatically perform cleanup of inactive subscribers. If you want to handle this, you must implement your logic to detect and close them.
7. Disconnects and Reconnections: If a user disconnects and then reconnects, the history of previous messages may be lost.
8. High Availability Scenarios: You can utilize a Redis cluster or replication.

You can download the code here.

Thank you for getting this far. If you like it, don’t forget to clap.

Subscribe to get more content related to JS, TS, and Node.js.