Rahasak Labs
Published in

Rahasak Labs

Reactive Microservices with Golang, Rabbitmq and Protobuf

Happy devOps

Scenario

We can build reactive, asynchronous and cross language microservices with Rabbitmq, Protobuf stack. In this post I’m gonna discuss about building microservices based document management service with Golang by using Rabbitmq and Protobuf. Protobuf use to serialized structured data(request response between microservices). Rabbitmq uses as message broker. The architecture of the system described in following figure.

There are two main services in the system, Gateway and Storage. Gateway service act as microservices api-gateway. Storage service do actual document management functions(ex: create documents in database). Client submit JSON documents to gateway service via REST api. Gateway service create Protobuf message based on the JSON data and pass it to storage service via Rabbitmq. Then storage service do what ever the operation(ex: create documents on a database) and sends the status back to gateway service as Protobuf message. Finally gateway service convert Protobuf message which receives from storage service into JSON object and send it back to client. Gateway service listen to Rabbitmq gateway queue and Storage service listen to storage queue. All the source code which related to this post available on gitlab. Please clone the repo and continue the post.

Protobuf spec

document.proto defines the Protobuf structures of document object and request response objects. Main thing to notice here is go_package="gitlab.com/rahasak-labs/rabbit/spec". When compiling the protobuf with protoc it generate the go file in gitlab.com/rahasak-labs/rabbit/spec package.

Following is the way to compile the Protobuf spec. It will generate output go file inside $GOPATH/src/gitlab.com/rahasak-labs/rabbit/spec.

Gateway service

Gateway service start REST api on /api/v1/documents:7654. When request comes to this api, it creates CreateDocumentMessage and forward it to storage service.

Rabbitmq publisher implemented with publisher.go. It gets the messages from go channel and publish to Rabbitmq.

Storage service process CreateDocumentMessage and send the CreateDocumentReply back to gateway service. Gateway service consumes the reply messages from Rabbitmq and send the response back to client as JSON object. The Rabbitmq consumer implemented with consumer.go.

The communication between publisher and consumer handles with go channels. I have written detailed blog post about go channel based communication and response aggregation in here.

Storage service

Storage services gets CreateDocumentMessage from Rabbitmq. The Rabbitmq consumer implemented with consumer.go.

Once message receives, it do what ever the operations(ex: create a document on database) and sends the CreateDocumentReply back to gateway service via Rabbitmq. The Rabbitmq publisher implemented with publisher.go.

Deploy services

I have dockerized all the services and available to deploy via docker-compose. Following is the docker-compose.yml.

The Rabbitmq connection informations defined with RABBIT_URI=amqp://admin:admin@host.docker.internal:5672/. host.docker.internal field routes to docker host in mac-os. In linux environment change it to local machines IP or add a host entry to /etc/hosts file by overriding host.docker.local with local machines IP. Services can be started in following order.

Test services

We can send the JSON request to gateway REST api via curl. Following is the way to test the service. The Rabbitmq admin console can be viewed from http://localhost:15672/#/queues.

Reference

  1. https://jbrandhorst.com/post/go-protobuf-tips/
  2. https://tutorialedge.net/golang/go-protocol-buffer-tutorial/
  3. https://kjanshair.github.io/2018/08/20/demystifying-protocol-buffers-and-grpc-with-go/
  4. https://medium.com/rahasak/microservice-response-aggregate-with-golang-goroutines-6aa84aea6fca
  5. https://medium.com/@masnun/work-queue-with-go-and-rabbitmq-b8c295cde861
  6. https://www.rabbitmq.com/tutorials/tutorial-one-go.html

--

--

--

Have less, be more

Recommended from Medium

Rewrite Your Elasticsearch Requests OnTheFly #

Explore Asynchrony Primer for Dart & Flutter

CMake Introduction

C9 vs TL W1D1

Tips to handle performance and coding way between different type of loop

Python vs Node.js for Tech Founders (Django vs Node.js)

Roadmap for Turbo-Geth

How Does Fleet Management Software Boost Functionality?

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
λ.eranga

λ.eranga

Ego = 1/Knowledge

More from Medium

Multiple Azure functions in Golang

Testing the main of a golang http server

gRPC proto files Best Practices

Series: Intro to Managing Golang Kafka Consumer Lag