Golang & production-ready solution: Part 2 (Programming)
Golang developer cookbook
Golang & production-ready solution: Part 1 (Introduction)
Golang & production-ready solution: Part 3 (CI/CD)
Programming
In this article, let’s talk about the development of microservice and everything related to it.
What do we have at the output? A dozen microservices wrote using go-kit tools located behind API-Gateway.
Each service is placed in an Alpine container and docker-compose is orchestrated. So far, everything is simple.
Architecture & tools
Since we are creating a microservice architecture, let’s agree that each service is really small and performs only tasks within its authority, and communicates with brothers only through interservice interaction.
Yes, and I hope you are all ready familiar with the principles of clean architecture. Imagine how it should look in Go. By the way, this is not a bad article about this.
And remember that if you do not adhere to this concept, then you will have to rewrite your project every few months. You simply cannot test it and support it normally.
But since we use go-kit, we have already taken care of us. We simply implement each level using auxiliary tools.
The building element in the go-kit application is the endpoint. It is also an RPC method that exposes the service method (business logic) using the transport layer. Fortunately, their boxes are all the transports we need: HTTP, gRPC, and NATS. True, if you need NATS-streaming, then you will have to write the transport yourself. But it is not so difficult.
Another feature we need is that each endpoint can use several types of transport at the same time, and I just needed it. I set HTTP transport for a web application and gRPC for mobile.
So, you can feel the full power of this tool if you use middleware by wrapping the endpoint in them. With this trick, you get out of the box monitoring in the face of Prometheus and trace requests (for example jaeger).
A few more amenities will not hurt us. Any service must be configurable so that it is not constantly reassembled or rewritten. One of the universal solutions is Viper.
You will be able to store your config in any popular format, in environment variables, and perhaps in the service discovery. It all depends on your coolness ;)
As a bonus, since we use CI /CD to the fullest, we flash the version of your assembly into the binary. How to automatically increment the version for each assembly is your question, but I can tell you that if you do not come up with anything sensible, then look at the predefined Gitlab CI variables.
You can set the value variable during assembly like this:
go build -ldflags="-X 'package_path.variable_name=new_value’"Database
Well then, I will consider the most common PostgreSQL database option at the moment. But in principle, the approach is similar everywhere. By the way, if you collect, for example, a sufficient number of audits, then probably your case is a ClickHouse column database — I love it very much :)
Although perhaps you will have enough tables for these events in PostgreSQL with Write Ahead Log disabled.
But let’s go down from heaven to earth.
Guided by the principles of building microservices, we know that each of them must have its territory. In PostgreSQL, a scheme is best suited for these purposes.
That is, we are doing one for our entire project, and we add our schemes for each microservice and call them accordingly.
Next, we choose between ORM and pure SQL. Everything from tasks and loads. I don’t really like Gorm and use only PostgreSQL, so I settled on the pg, but if performance is critical for you, I advise you to use lower-level solutions like pgx, it has a lot of chips — try it.
He also has a library for testing, but it is too low-level, of the protocol level of Post communication with the database;) He has one more interesting feature.
Pgx supports logical replication, which means that by listening to events from WAL we can automatically publish them to the event bus.
Yes, it will not be as obvious as with the manual publication of events, but this magic also works well. By the way, in some of my microservices, where the value of events is not critical, I subscribe to events directly,
using the PostgreSQL notify/listener feature, which by the way is also implemented in ORM pg. There, unlike the WAL-listener, everything is quite simple ;)
Further, of course, we will use migrations, without this anywhere.
I use another sql-migrate convenient tool. It works with various databases, so the choice is yours.
Everything is simple to him. Create a migration using the create command, enter two instructions for it (up & down) in SQL
-- +migrate Up
CREATE TABLE users
(
id UUID
login VARCHAR(255) NOT NULL,
);
-- +migrate Down
DROP TABLE IF EXISTS users;Later we roll them with the up command. Do not forget to configure the connection to the database using one of the available methods. Everything is simple too.
And that’s not all. There is a very nice bonus here.
We can store all migrations in our binary (packr is used), for which the DevOps engineer (we too) will thank us a lot. And it all works out of the box!
Interservice communication
Interservice communication is divided into two types. Synchronous and asynchronous.
We will use synchronous, for example, for requests receiving some data, without which it would not be possible to complete our task.
I really like Google’s protocol buffers, so we will use gRPC for synchronous inter-server communication. This transport is supported in the go-kit out of the box.
So, of course, you can use the more familiar REST.
For asynchronous interaction, we will use the event bus. We will use it if we want to notify another service about an event or we will carry out some resource-intensive and costly action, which is not necessary for the client initiating the request.
The publication and subscription system implemented using the NATS server is our choice. Imagine a situation in which one of the microservices changes a certain business model, and the other microservices that are really interesting just subscribe to this event and receive at any time (of course, with a payload). If it is critical for you to know about this event, then we use NATS-streaming — an event bus with a guarantee of delivery “at least once”.
API-Gateway
API-Gateway is exactly what I have been missing for several years. Let’s talk a bit about what it is and what we will have advantages from using it.
What did it look like before? Several microservices running on the server, each working on its own port. Each of them has to be protected by authorization and other middleware.
Then it’s unlikely that some system administrator (that is, you) will like a lot of open ports. This increases the attack vector and brings other inconveniences. What can we do with this in our home project? Set up a firewall, write some rules for iptables and close internal ports, but was there a simpler way?
In Docker, all services went up, but their ports did not forward. Nginx also started on the same network and all the necessary traffic went to them through it. Directly they were not available.
Now we have one entry point — the gateway. All traffic goes through it and is routed by the config via our microservices.
I use the excellent open-source project KrakenD at my place. It is fast and reliable, and the possibilities in it are enough for your needs.
Here is a standard set of features that you will most often use.
- A single entry point — everything else is closed from prying eyes.
Here the benefits are clear. We solve all the necessary issues in one place, and not in each service individually. Be it CORS or authorization. I will dwell on authorization in more detail below. - Filter user request — you can filter requests, i.e. headers and query parameters can pass to microservices only those described in the config. We increase safety.
- Manage requests — cache requests, set timeouts for them. You probably have requests for directories, which are rarely updated and it is logical to cache them — we do this with one movement in the config.
- Formatting the response — you can get the response from the backend to reformat the object and return it to the end-user. Rename, hide, or copy any properties of the json object.
- Combine requests — cool things to merge requests. Imagine you have a page with a user profile on your site and it focuses on information from different endpoints or microservices. For example, information with personal data from the user's world service, and the history of his orders from the microservice orders. From the frontend, we can make one request to our gateway,
GET https://api-gateway.io/users/1/profileand already the gateway under the hood will make two requests:
GET service1.local/users/1 GET service2.local/userOrders/1
and it will not be noticeable for the end-user to combine them. Moreover, requests can be performed sequentially. For example, we need data from the first query when executing the second. Cool!
Here is a fragment of my config. Everything is very simple!
{
"version": 2,
"name": "My awesome API",
"port": 5000,
"cache_ttl": "3600s",
"timeout": "3s",
"extra_config": {
"github_com/ihippik/krakend-mw/relyingparty": {
"token_secret": "topsecret"
}
},
"endpoints": [
{
"endpoint": "/orders",
"method": "GET",
"output_encoding": "no-op",
"headers_to_pass": [
"Authorization",
"Content-Type"
],
"extra_config": {
"github.com/ihippik/krakend-mw/relyingparty": {
"roles": [
"admin",
"manager",
"member"
]
}
},
"backend": [
{
"host": [
"http://awesome_api:7000"
],
"encoding": "no-op",
"url_pattern": "/v1/orders",
"extra_config": {}
}
]
}
]
}The project has excellent documentation, so I think you will figure it out yourself with the other, more specific features of it.
Authorization
Now let’s dwell in more detail on the authorization. This was the first reason why I decided to use KrakenD. Of course, we could write middleware for authorization and copy it from service to service. But we are programmers! We better leave all our microservices defenseless, but limit the traffic to them on the gateway. At first, I decided to implement oauth2 and raise another go open-source ory/hydra as an authorization service, but remember that a small child would not give me so much free time I left this venture. He stopped at the authorization built on JWT tokens. Convenient and easy.
Inside the token, I put the user id and its role. Wrote a simple middleware for our gateway that did a few things
- checked the token
- saved the role of the user from the token with the role specified in the config to the endpoint
- put the user id in the headers (for the needs of microservices)
This is the minimum we need to limit access to API. We will not discuss the recall of tokens and their updating in this article.
Testing
Here, everything is quite simple and obvious. I use the most popular testify library for testing.
I test each layer with the table tests that the JetBrains IDE so beautifully generates for us.
The only thing you need to change all assert to testify assert.
Since we use a clean architecture (go-kit) in development, we should not have problems with tests.
The main thing is that everything that we add ourselves also adheres to this ideology.
We test each unit of abstraction with unit tests separately:
- Transport layer (HTTP) — use the httptest classic. make testify.mock the service layer, which goes next.
- The layer of endpoints (endpoints) — here we just test the functions of encoding requests and decoding responses.
- The service layer (business logic) we use testify.mock for the repository level and test it.
- The repository layer (DB) — here are the small problems and the eternal question of how ?!
The question is this. That we will test the real DB or mock it.
Of course, the second option is preferable for unit testing, but here it all depends heavily on which driver you chose to work with the database.
Sqlmock is probably the best choice if your driver implements the standard sql/driver interface.
I know there are fans of Gorm and it is perfect for them. Maybe you will have pq or sqlx.
You can test the repository simply by checking if your Gorm created the SQL query correctly or simply explicitly indicate which query you need to return, whether it is data or a regular error.
But everything becomes more complicated if you use more modern things, for example, such as ORM pg, pgx.
Here you have to use a real base, although in the era of docker it is not so scary.
And a good solution would be ory/dockertest, coupled with testify.suite. In general, you choose.
API Documentation
Any API must have documentation and our case is no exception.
After all, if you write API only for your front today, then tomorrow maybe you will provide it to third-party developers. For a small fee naturally :)
The industry standard in our business is Swagger.
Hardly these days someone else is not familiar with him, but all of a sudden. You can use it in two directions.
Generate documentation on annotations in the code, and generate code on the documentation.
The world already knows our attitude to the generated code, so let us dwell on the first option.
We are writing comments on each of our HTTP-endpoints according to the Swagger specification.
// @Title Create
// @Tags Users
// @Accept json
// @Produce json
// @Description create of new users
// @Summary create user
// @Success 201 {object} User
// @Failure 400 {object} Error
// @Failure 500 {object} Error
// @Router /users [post]
// @Param model body user.CreateRequest true "create user"Next, we launch another great go-swagger tool that will generate documentation for our microservice and put it in the specified folder in JSON and YAML format.
Further, we can transfer this file, on-demand.
We can download it on the swagger website and get the opportunity not only to interactively view the specification but also send requests — the so-called Swagger UI.
Or, pack all these statics into our binary file, for example, packr, and distribute our microservice with a point showing our documentation. Who wants it.
Documentation as a code is a good approach. Everything is in one place and all your documentation is developing along with the project.
Earlier, by the way, I used the service https://apiary.io/ but it’s a little annoying to run the project in two places.
Logging
I think everyone understands the importance of logging, and we must undoubtedly pledge everything.
By the way, there is an opinion that it is necessary to prohibit all debuggers so that the developers finally begin to log everything normally :)
The go-kit has its logger, but I used logrus for so long that I dragged it into the project;)
You are of course free to use a more productive option.
The main thing is that it supports logging levels, json-format, and caller.
Let me remind you a bit about the basic rules of logging.
- We use the Error level for errors due to which your service cannot work correctly.
And we want to report it as soon as possible. - We use the Warning level for errors processed by our program.
We also want the administrator of our system to find out about it, but our service works as usual, as we were able to process it. - We use the Info level to notify you of successful actions performed by the service.
The main thing is not to make dry comments that the method was successful — do not forget to add the call context to the fields. - With the Debug level, we can mark the train of thought of our service, mark every IF and put information in the context that helps to understand why a decision was made.
Well, that’s it.
Now, managing the logging levels from the service config, we can successfully enter the prod.
While the service is not yet stable, we exit production with a debug level.
Over time, we understand that the service is already quite stable and the logs of this level are already superfluous.
So gradually we reach the Error level and enjoy life.
