Golang & production-ready solution: Part 2 (Programming)

Golang developer cookbook

Konstantin Makarov
Dec 14, 2019 · 11 min read
ihippik ©

Golang & production-ready solution: Part 1 (Introduction)

Golang & production-ready solution: Part 3 (CI/CD)


In this article, let’s talk about the development of microservice and everything related to it.

What do we have at the output? A dozen microservices wrote using go-kit tools located behind API-Gateway.
Each service is placed in an Alpine container and docker-compose is orchestrated. So far, everything is simple.

Architecture & tools

Since we are creating a microservice architecture, let’s agree that each service is really small and performs only tasks within its authority, and communicates with brothers only through interservice interaction.
Yes, and I hope you are all ready familiar with the principles of clean architecture. Imagine how it should look in Go. By the way, this is not a bad article about this.
And remember that if you do not adhere to this concept, then you will have to rewrite your project every few months. You simply cannot test it and support it normally.
But since we use go-kit, we have already taken care of us. We simply implement each level using auxiliary tools.
The building element in the go-kit application is the endpoint. It is also an RPC method that exposes the service method (business logic) using the transport layer. Fortunately, their boxes are all the transports we need: HTTP, gRPC, and NATS. True, if you need NATS-streaming, then you will have to write the transport yourself. But it is not so difficult.
Another feature we need is that each endpoint can use several types of transport at the same time, and I just needed it. I set HTTP transport for a web application and gRPC for mobile.
So, you can feel the full power of this tool if you use middleware by wrapping the endpoint in them. With this trick, you get out of the box monitoring in the face of Prometheus and trace requests (for example jaeger).

A few more amenities will not hurt us. Any service must be configurable so that it is not constantly reassembled or rewritten. One of the universal solutions is Viper.
You will be able to store your config in any popular format, in environment variables, and perhaps in the service discovery. It all depends on your coolness ;)
As a bonus, since we use CI /CD to the fullest, we flash the version of your assembly into the binary. How to automatically increment the version for each assembly is your question, but I can tell you that if you do not come up with anything sensible, then look at the predefined Gitlab CI variables.

You can set the value variable during assembly like this:

go build -ldflags="-X 'package_path.variable_name=new_value’"


Well then, I will consider the most common PostgreSQL database option at the moment. But in principle, the approach is similar everywhere. By the way, if you collect, for example, a sufficient number of audits, then probably your case is a ClickHouse column database — I love it very much :)

Although perhaps you will have enough tables for these events in PostgreSQL with Write Ahead Log disabled.

But let’s go down from heaven to earth.

Guided by the principles of building microservices, we know that each of them must have its territory. In PostgreSQL, a scheme is best suited for these purposes.

That is, we are doing one for our entire project, and we add our schemes for each microservice and call them accordingly.

Next, we choose between ORM and pure SQL. Everything from tasks and loads. I don’t really like Gorm and use only PostgreSQL, so I settled on the pg, but if performance is critical for you, I advise you to use lower-level solutions like pgx, it has a lot of chips — try it.

He also has a library for testing, but it is too low-level, of the protocol level of Post communication with the database;) He has one more interesting feature.

Pgx supports logical replication, which means that by listening to events from WAL we can automatically publish them to the event bus.

Yes, it will not be as obvious as with the manual publication of events, but this magic also works well. By the way, in some of my microservices, where the value of events is not critical, I subscribe to events directly,

using the PostgreSQL notify/listener feature, which by the way is also implemented in ORM pg. There, unlike the WAL-listener, everything is quite simple ;)

Further, of course, we will use migrations, without this anywhere.

I use another sql-migrate convenient tool. It works with various databases, so the choice is yours.

Everything is simple to him. Create a migration using the create command, enter two instructions for it (up & down) in SQL

-- +migrate Up
login VARCHAR(255) NOT NULL,
-- +migrate Down

Later we roll them with the up command. Do not forget to configure the connection to the database using one of the available methods. Everything is simple too.

And that’s not all. There is a very nice bonus here.

We can store all migrations in our binary (packr is used), for which the DevOps engineer (we too) will thank us a lot. And it all works out of the box!

Interservice communication

Interservice communication is divided into two types. Synchronous and asynchronous.
We will use synchronous, for example, for requests receiving some data, without which it would not be possible to complete our task.
I really like Google’s protocol buffers, so we will use gRPC for synchronous inter-server communication. This transport is supported in the go-kit out of the box.
So, of course, you can use the more familiar REST.
For asynchronous interaction, we will use the event bus. We will use it if we want to notify another service about an event or we will carry out some resource-intensive and costly action, which is not necessary for the client initiating the request.
The publication and subscription system implemented using the NATS server is our choice. Imagine a situation in which one of the microservices changes a certain business model, and the other microservices that are really interesting just subscribe to this event and receive at any time (of course, with a payload). If it is critical for you to know about this event, then we use NATS-streaming — an event bus with a guarantee of delivery “at least once”.


API-Gateway is exactly what I have been missing for several years. Let’s talk a bit about what it is and what we will have advantages from using it.
What did it look like before? Several microservices running on the server, each working on its own port. Each of them has to be protected by authorization and other middleware.
Then it’s unlikely that some system administrator (that is, you) will like a lot of open ports. This increases the attack vector and brings other inconveniences. What can we do with this in our home project? Set up a firewall, write some rules for iptables and close internal ports, but was there a simpler way?
In Docker, all services went up, but their ports did not forward. Nginx also started on the same network and all the necessary traffic went to them through it. Directly they were not available.
Now we have one entry point — the gateway. All traffic goes through it and is routed by the config via our microservices.
I use the excellent open-source project KrakenD at my place. It is fast and reliable, and the possibilities in it are enough for your needs.

Here is a standard set of features that you will most often use.

  • A single entry point — everything else is closed from prying eyes.
    Here the benefits are clear. We solve all the necessary issues in one place, and not in each service individually. Be it CORS or authorization. I will dwell on authorization in more detail below.
GET https://api-gateway.io/users/1/profile

and already the gateway under the hood will make two requests:

GET service1.local/users/1 GET service2.local/userOrders/1

and it will not be noticeable for the end-user to combine them. Moreover, requests can be performed sequentially. For example, we need data from the first query when executing the second. Cool!
Here is a fragment of my config. Everything is very simple!

"version": 2,
"name": "My awesome API",
"port": 5000,
"cache_ttl": "3600s",
"timeout": "3s",
"extra_config": {
"github_com/ihippik/krakend-mw/relyingparty": {
"token_secret": "topsecret"
"endpoints": [
"endpoint": "/orders",
"method": "GET",
"output_encoding": "no-op",
"headers_to_pass": [
"extra_config": {
"github.com/ihippik/krakend-mw/relyingparty": {
"roles": [
"backend": [
"host": [
"encoding": "no-op",
"url_pattern": "/v1/orders",
"extra_config": {}

The project has excellent documentation, so I think you will figure it out yourself with the other, more specific features of it.


Now let’s dwell in more detail on the authorization. This was the first reason why I decided to use KrakenD. Of course, we could write middleware for authorization and copy it from service to service. But we are programmers! We better leave all our microservices defenseless, but limit the traffic to them on the gateway. At first, I decided to implement oauth2 and raise another go open-source ory/hydra as an authorization service, but remember that a small child would not give me so much free time I left this venture. He stopped at the authorization built on JWT tokens. Convenient and easy.
Inside the token, I put the user id and its role. Wrote a simple middleware for our gateway that did a few things

  • checked the token


Here, everything is quite simple and obvious. I use the most popular testify library for testing.
I test each layer with the table tests that the JetBrains IDE so beautifully generates for us.
The only thing you need to change all assert to testify assert.
Since we use a clean architecture (go-kit) in development, we should not have problems with tests.
The main thing is that everything that we add ourselves also adheres to this ideology.
We test each unit of abstraction with unit tests separately:

  • Transport layer (HTTP) — use the httptest classic. make testify.mock the service layer, which goes next.

The question is this. That we will test the real DB or mock it.
Of course, the second option is preferable for unit testing, but here it all depends heavily on which driver you chose to work with the database.
Sqlmock is probably the best choice if your driver implements the standard sql/driver interface.
I know there are fans of Gorm and it is perfect for them. Maybe you will have pq or sqlx.
You can test the repository simply by checking if your Gorm created the SQL query correctly or simply explicitly indicate which query you need to return, whether it is data or a regular error.
But everything becomes more complicated if you use more modern things, for example, such as ORM pg, pgx.
Here you have to use a real base, although in the era of docker it is not so scary.
And a good solution would be ory/dockertest, coupled with testify.suite. In general, you choose.

API Documentation

Any API must have documentation and our case is no exception.
After all, if you write API only for your front today, then tomorrow maybe you will provide it to third-party developers. For a small fee naturally :)
The industry standard in our business is Swagger.
Hardly these days someone else is not familiar with him, but all of a sudden. You can use it in two directions.
Generate documentation on annotations in the code, and generate code on the documentation.
The world already knows our attitude to the generated code, so let us dwell on the first option.
We are writing comments on each of our HTTP-endpoints according to the Swagger specification.

// @Title Create
// @Tags Users
// @Accept json
// @Produce json
// @Description create of new users
// @Summary create user
// @Success 201 {object} User
// @Failure 400 {object} Error
// @Failure 500 {object} Error
// @Router /users [post]
// @Param model body user.CreateRequest true "create user"

Next, we launch another great go-swagger tool that will generate documentation for our microservice and put it in the specified folder in JSON and YAML format.
Further, we can transfer this file, on-demand.
We can download it on the swagger website and get the opportunity not only to interactively view the specification but also send requests — the so-called Swagger UI.
Or, pack all these statics into our binary file, for example, packr, and distribute our microservice with a point showing our documentation. Who wants it.
Documentation as a code is a good approach. Everything is in one place and all your documentation is developing along with the project.
Earlier, by the way, I used the service https://apiary.io/ but it’s a little annoying to run the project in two places.


I think everyone understands the importance of logging, and we must undoubtedly pledge everything.
By the way, there is an opinion that it is necessary to prohibit all debuggers so that the developers finally begin to log everything normally :)
The go-kit has its logger, but I used logrus for so long that I dragged it into the project;)
You are of course free to use a more productive option.
The main thing is that it supports logging levels, json-format, and caller.

Let me remind you a bit about the basic rules of logging.

  • We use the Error level for errors due to which your service cannot work correctly.
    And we want to report it as soon as possible.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade