Working With Golang: Our First Steps

If you like to understand know why we choose golang in the first as an alternative to python, see https://medium.com/@anil.k.nayak8/gopher-in-healthifyme-25eb8494305b

In this article I would like to outline the main points we had in mind while creating an application using golang, the problems encountered and some of the packages that helped us. We created a simple microservice that sat on top of multiple databases and acted as a security/filtering layer.

We knew what we had to do: Create a JSON rpc server that would be consumed mainly by our python clients. But golang does not have a framework like Django or Ruby on Rails, which means there are no standards set on how things are to be done.

We had to make all decisions based on the project we were doing combined with the experince we had with python. So we made a note of all the things we needed to clear before going ahead:

• Folder structure
• Logging Setup
• Configuration with different environments with different sources of info like file, env etc
• Connecting to database and executing queries.
• Setting up rpc
• Writing tests

Deciding the folder structure

This was difficult at the beginning of a new project. So we took inspiration from https://github.com/golang-standards/project-layout and modified it a bit to suit our needs. We are now using the new go modules and they’re amazing! We can easily manage the dependencies with go plugins for IDEs. Go will automatically take care of dependencies and their versions. The only thing we should attend to is importing the third party module as url.

The following is the folder structure we are using:

build: development and production binaries.

cmd: the main go file. Ends up being lean as most of the initialization work happens here.

config: responsible for the configuration of the application. All the packages that need the current configuration will be in connection with this file.

pkg: contains all individual packages which can depend on each other.

Logging Setup

Golang comes with a very handy logging package.However we had debates on whether the native package should be used or the popular package ‘logrus’ or create a new custom interface all together. Logrus is an extension of the interface for the native logging package. Lorgus won as it was just what we needed, with all methods like Print, Debug, Panic etc. We could also change the logging Output.

We created a logger package in common that exposes an instance of logrus. And the same is used everywhere.

The final code was very simple:

var Log *logrus.Logger = logrus.New()

And for any logging:

logger.Log.Debug(“I am a debug message”)

Setup Configuration

Here we had to evaluate multiple factors. We had two options: either use a third party package or create our own configuration package. If we choose a third party, which one?

So we asked ourselves: what do we need from the package?

It should be able to read from a file like json/yaml. Since we needed sensitive information like database user, database password etc from a secret manager, it also should be able to read from the environment. And finally, it should do both simultaneously. For example for a single struct DatabaseConfig, host can come from env and the database name can come from the config file based on the environment.

We evaluated the ability of a third party package viper to do the same. However it could not function with multiple sources at the same time. So we tried a different package cleanenv. Bingo! It had all the features we needed.

A sample code would look something like:

Here we specify that the yaml file can be taken on preference, but it will fallback to env if the key is not available in the yaml file.

Database Connection and Querying

There are tons of libraries available for database interaction for go and each one has its own flavour. An added advantage here is that golang comes with a sql interface by default and most of the libraries implement the same. So now it’s easier to switch.

Initially we went with the native sql package to create database connections and execute queries. But it seemed very basic. Inorder to execute each and every query and convert into a struct, each column needs to be scanned, which would look like a whole lot of boilerplate code.

For example: suppose you are trying to select a row with 4 columns

This whole process has to be done for each db query.

This is where sqlx [github.com/jmoiron/sqlx] comes into the picture.It has a feature which will directly take the struct and do the scanning for us. It also gives an interface to specify what the db column names mapped to each struct field. The below example specifies the same:

And now it’s simple! Bonus: this package comes with an optional sql interface!

Setting up RPC

Golang comes with its own json rpc package. So RPC setup can be very simple.

But there is a glitch: it works on a TCP layer and it would make sense to write the rpc client also in golang. But our client is written in python. Hence using a tcp layer and maintaining consistent connection can create problems.

So we treated each rpc request as an independent http request.

So now, we used 2 packages github.com/gorilla/rpc/v2 and github.com/gorilla/rpc/v2/json.

Using this we could create a simple endpoint for example ‘/json’ and mount our RPC server on it.

The sample code would look something like:

We can also add different endpoints to handle different kinds of requests. Becomes handy when adding a healthcheck route. Any client would be able to use it as it is very flexible.

Setting up Tests.

Golang comes with its own test system. Great! Now we don’t have to use external setups unless absolutely necessary.

Since we are dealing with different databases and executing queries, we need a proper system where we can mock the database interaction. All we need to have is some struct, that has the same interface as the database object. We also need to have a global database instance which can be replaced by the mock database instance. Thus, we can mock all the database interactions and test different cases out.

We used two different packages to do this.

1. github.com/DATA-DOG/go-sqlmock which creates a mock database based on sql.

2. github.com/jmoiron/sqlx which has the feature to create a database having the same interface as sqlx from other database instances. This is extremely useful as we use sqlx to interact with databases.

And now the final step: replace our global database instance with the mock database we created. We wrapped this behaviour in a single function and used it in all our testing methods. This made sure that we have a new fresh database for each test.

Now the sample code looks like:.

Handling Errors

Golang gives no special status to errors. Error is treated as a normal value, just like any other interface. Now it is very important that all kinds of errors are handled.

If any error is left out it can crash production. Fortunately there is a package to check if any errors are missed.https://github.com/kisielk/errcheck helps in finding error cases which should have been checked but has been left out. We also needed to make sure the errors have contextual information. For e.g a rpc can fail due to several reasons: database connection error, query execution error, rows scanning error etc. The client that uses a function to execute a database query needs more meta information, including the error trace to understand the error in detail. For this we have used the package “github.com/pkg/errors” which helps in wrapping an error with additional parameters. E.g: errors.Wrap(err, “Error in statement execution”). This become handy while debugging.

Bonus lessons from our experience:

1. Not to use the ‘init’ feature unless absolutely necessary. The dependency system might break as there is no way of knowing which package will run first. This will also cause problems while writing tests.
2. Not to use Panic unless the application cannot run when the error occurs. Always try to handle errors gracefully.
3. Always wrap errors to give them more meta information.
4. Always defer to cleanups like closing database connection, closing a file etc.
5. Keep interfaces as lean as possible. This would bring out the most from golang’s powerful interface concept.

Conclusion.

Though a low level one, Go is a great language. Unlike others of its kind, memory management is very flexible. It has a proper way to format code. Gofmt does all the work and kind of universally accepted*. The type system is also simple. The code we write becomes very clear and understandable with very little effort. Most importantly, it creates a single binary. It can run anywhere* with no setup and no dependencies. Also the compiler is superfast with a great IDE support. When we created a build using docker to deploy the docker image to production size, it was just 8MB, a very small size and hence easy to handle.

However there is no proper standard for code organisation yet, maturity of different third party packages etc. But there is hope it will become the next go to language for server side development once it establishes its ground. It was worth while experimenting in golang and we will definitely extend its use for new projects and in some of our older projects as well.