Reactive Microservices with Golang, Rabbitmq and Protobuf
Happy devOps
Scenario
We can build reactive, asynchronous and cross language microservices with Rabbitmq, Protobuf stack. In this post I’m gonna discuss about building microservices based document management service with Golang by using Rabbitmq and Protobuf. Protobuf use to serialized structured data(request response between microservices). Rabbitmq uses as message broker. The architecture of the system described in following figure.
There are two main services in the system, Gateway and Storage. Gateway service act as microservices api-gateway. Storage service do actual document management functions(ex: create documents in database). Client submit JSON documents to gateway service via REST api. Gateway service create Protobuf message based on the JSON data and pass it to storage service via Rabbitmq. Then storage service do what ever the operation(ex: create documents on a database) and sends the status back to gateway service as Protobuf message. Finally gateway service convert Protobuf message which receives from storage service into JSON object and send it back to client. Gateway service listen to Rabbitmq gateway queue and Storage service listen to storage queue. All the source code which related to this post available on gitlab. Please clone the repo and continue the post.
Protobuf spec
document.proto defines the Protobuf structures of document object and request response objects. Main thing to notice here is go_package="gitlab.com/rahasak-labs/rabbit/spec". When compiling the protobuf with protoc it generate the go file in gitlab.com/rahasak-labs/rabbit/spec package.
Following is the way to compile the Protobuf spec. It will generate output go file inside $GOPATH/src/gitlab.com/rahasak-labs/rabbit/spec.
Gateway service
Gateway service start REST api on /api/v1/documents:7654. When request comes to this api, it creates CreateDocumentMessage and forward it to storage service.
Rabbitmq publisher implemented with publisher.go. It gets the messages from go channel and publish to Rabbitmq.
Storage service process CreateDocumentMessage and send the CreateDocumentReply back to gateway service. Gateway service consumes the reply messages from Rabbitmq and send the response back to client as JSON object. The Rabbitmq consumer implemented with consumer.go.
The communication between publisher and consumer handles with go channels. I have written detailed blog post about go channel based communication and response aggregation in here.
Storage service
Storage services gets CreateDocumentMessage from Rabbitmq. The Rabbitmq consumer implemented with consumer.go.
Once message receives, it do what ever the operations(ex: create a document on database) and sends the CreateDocumentReply back to gateway service via Rabbitmq. The Rabbitmq publisher implemented with publisher.go.
Deploy services
I have dockerized all the services and available to deploy via docker-compose. Following is the docker-compose.yml.
The Rabbitmq connection informations defined with RABBIT_URI=amqp://admin:admin@host.docker.internal:5672/. host.docker.internal field routes to docker host in mac-os. In linux environment change it to local machines IP or add a host entry to /etc/hosts file by overriding host.docker.local with local machines IP. Services can be started in following order.
Test services
We can send the JSON request to gateway REST api via curl. Following is the way to test the service. The Rabbitmq admin console can be viewed from http://localhost:15672/#/queues.
Reference
- https://jbrandhorst.com/post/go-protobuf-tips/
- https://tutorialedge.net/golang/go-protocol-buffer-tutorial/
- https://kjanshair.github.io/2018/08/20/demystifying-protocol-buffers-and-grpc-with-go/
- https://medium.com/rahasak/microservice-response-aggregate-with-golang-goroutines-6aa84aea6fca
- https://medium.com/@masnun/work-queue-with-go-and-rabbitmq-b8c295cde861
- https://www.rabbitmq.com/tutorials/tutorial-one-go.html
