Building a microservice framework in Golang
[UPDATE — 05/13/2018]
Added an gRPC based microservice example to the github repository. Check it out at rpc-microservice.
Start the gRPC server by running the /examples/rpc-microservice/main.go and invoke an RPC call using /examples/rpcclient.go
Added the code snippets on how the RPC server implements the Server interface.
Exactly one year ago I started my job at CSP Inc. to work on the ARIA product line. At the time we were a team of four engineers, all hailing from an embedded background. Having worked extensively in C/C++ in the past, we had little to no experience in Golang (heck I hadn’t even heard of Golang at that point). We were faced with a daunting task — to build the next biggest Software Defined Security product! The project had multiple components ranging from the web-service front-end to developing an agent software which would run on our Secure Intelligent Adapters (SIA).
I was tasked with designing and developing the Controlplane (which we refer to as the Software Defined Security Orchestrator — SDSo). The SDSo was to be developed as a collection of multiple microservices mostly written in Golang. Over the past year I have worked with multiple open-source libraries and implemented services using Kafka, gRPC, Redis, MySQL, HTTP, Celery, among several others. In this time, I have created over twenty microservices written exclusively in Golang. With the team growing from four to fifteen engineers working on the product, I have seen microservices being developed in a style as unique as the developer who worked on it. However, in principle, each one of these services are the same. Each one composed of one or more running servers, storage options and one or more upstream clients.
Due to the fast paced nature of our work, it was absolutely essential for me and my team of four engineers to focus on the business logic of our services. Rather than rehashing boilerplate code to setup our microservices such as initializing our server, database, clients etc., I came up with a simplified framework which I will be describing in this post.
Microservice interface
Each microservice should satisfy the Microservice interface. This is not imposed but only serves as a guideline.
An example of how we implement this interface would be :
We will get to the init() seen in line 12 above in a few moment. (*)
Server interface
Each server would need to implement the server interface which is composed of the following :
AStart() method which would start the would start the server as a go routine.
A Stop() method which would gracefully stop the server by sending a message on its quit channel returned at the time of starting
A RegisterNamespace() and RegisterService() to register a namespace which allows for using the same channel/topic for sending messages destined for different endpoints identified by their service type string.
An implementation of the server would look something like follows -
Dont get confused by the nomenclature [KafkaConsumer] below. At CSPi we use kafka not just as a pubsub framework but also as a server for our microservices. Typically used when the service being targeted needs to asychronous. (You may note that the response in the Handle() is not returned to the caller. Sending a response is an implementation detail when using kafka as a server).
Endpoint interface
The business logic of our service is encapsulated in the Endpoint interface within the Handle() method.
An implementation of which would look something like this :
The users job is to register this endpoint as a handler for the service identified by a string "hello" registered under the "greeter" namespace. In kafka the namespace is synonymous with the topic name.
That wraps up my post about microservices.
Keep watching this space for more edits, completed with snippets and gifs to come.
Doing this with an RPC Server instead of Kafka
The Server interface implementation would like below :
I generated the RPC server using the protoc tool using the following protobuf specification file.
which can be used to generate the proto server using the protoc tool provided by gRPC.
protoc --proto_path=./ rpc.proto --go_out=plugins=grpc:.You can find all the code hosted on github at https://github.com/nitishm/go-micro-framework/.
To test this framework out I have provided an example directory completed with a simple microservice running kafka as a server which can found in the /examples/ folder. Simple start the server by running the main.go in the /examples/kafka-microservice/ directory.
To send a message over the listening kafka topic, use the producer.go program provided under the /examples directory.
[A docker-compose file is in the works for using the example microservice. I will update the page as soon as I have that ready. If you are curious you can always manually startup your kafka server using the Dockerfile provided at https://hub.docker.com/r/spotify/kafka/].
