Go Microservices Using Gin And GORM

A blog entry delving into the basics of the Gin Web Framework and the GORM ORM library. Along with some digging into type embedding, interfaces and receivers.

Wish I Was Vodka! 😋

The Gin Web Framework — Briefly

The Gin Web Framework is a versatile web framework for go. It provides a nice wrapper around an incoming HTTP request and the corresponding response with easy to use functions that facilitate the binding of incoming JSON or XML request body, Form data or query parameters to structs including input validation followed by functions to easily serialise interfaces as JSON or XML and render them as response or just render a simple template driven HTML response.

Gin also provides ways to write middlewares, likes of which we will see in the upcoming stories dealing with cross cutting concerns like JWT based authentication and population of the request context with parsed claims or attaching trace and span id headers for distributed tracing and logging.

In this instalment -

1. We will set up a go microservice using Gin.
2. And set up GORM or Golang Object Relational Mapping against a Postgres database.

The User Microservice

That is what we are going to build! A user microservice that will expose a REST endpoint for user signup. It will accept a well defined JSON payload, validate it, and then transform and persist the data using GORM. We will use postgres as our backend. We will also look into error handling. Later on we will enhance this service to facilitate authentication. So let’s begin.

Initializing The Module And Adding Gin and GORM

That’s simple! Open up a terminal and fire

mkdir user-service && \
cd user-service && \                                                 
go mod init github.com/prithuadhikary/user-service

That will create and initialise the user-service go module in a directory named ‘user-service. You should already be inside the user-service directory with a go.mod file by now. So let’s add Gin and GORM.

go get -u github.com/gin-gonic/gin

And

go get -u gorm.io/gorm 

We will also need the postgres driver to connect to our backend.

go get -u gorm.io/driver/postgres

Now we are ready to wire things up!

Initialising GORM

Before we can write the repository that we will use to persist our data, we would need to initialise GORM, that will include:

• Define our domain classes (with appropriate fields and tags so that GORM knows the schema).
• Connect to the database.
• Do an automigration — i.e. ask GORM to create the required tables and primary key and referential constraints etc.

Application Architecture — The Layers

In traditional enterprise applications, it is a common practice to break the application logic into three manageable layers, namely:

• Controller — This is the layer that receives the incoming requests(primarily HTTP), deconstructs the request body, request parameters, headers, path variables etc. to create a model object that the service methods act on. This also is responsible to render the response back to the response stream.
• Service — This is where the business logic resides and interacts with the repository layer.
• Repository — This handles the lower level details of persisting a domain object or structs.

Let us start building bottom up with the repository layer being the bottom one.

Defining The User Domain — User Struct Type

The repository layer generally deals with the domain entities. The domain entities are what get mapped to the underlying database by the ORM library in the form of tables.

Let’s keep it simple for now. Let’s create a directory named domain under the module root and create a file user.go containing the following.

Well, it’s pretty self explanatory. We have four fields (at least which are visible to the naked eye). But what is that gorm.Model doing there? In Go, we don’t have inheritance, instead we have composition. It works like this. If you have two types TypeA and TypeB like so:

type Vodka struct {
   AlcoholPercent float32
}

func (vodka Vodka) Hangover()  {
   fmt.Println("Tasteless Hangover!")
}

type BloodyMary struct {
   Vodka
   Ingredients []string
}

Then, upon creating a struct variable of BloodyMary.

var myPeg BloodyMary

You can do,

myPeg.AlcoholPercent

This is called Type Embedding. Actually you can do this too:

myPeg.Vodka.AlcoholPercent

Because, we embedded Vodka in BloodyMary 👻. But, myPeg.AlcoholPercent also works because fields and methods of an embedded type get promoted to the containing type. Similarly, myPeg.Hangover() is also valid. But there is a caveat to this, promoted only if they are not shadowed!

So, if you declare a separateHangover method for BloodyMary type like so,

func (bloodyMary BloodyMary) Hangover()  {
   fmt.Println("Hangover yet tasty!")
}

this Hangover will shadow the Hangover method declared for the Vodka type! And invoking myPeg.Hangover() will print out Hangover yet tasty!.

Ok! We digressed a bit but this will help you understand what will be the effect of embedding the gorm.Model in our domain.User structure. If you look up the contents of the gorm.Model structure, you will see the following:

The documentation sums it up already. Even the User struct 😉. But then you would ask, why did we declare ID again in our User domain?

The aim was to shadow the unsigned int ID declared in the gorm.Model with a UUID ID. Reason is obvious, integer primary keys are guessable and often generated sequentially(often using Sequences). UUID is unique all the time, at least within the same table residing in one of the schemas in your database.

Important Note: If you encountered a UUID collision within the a single table of your application among all the gazillion tables on earth, be sure that either God probably wants you in Hell or it is a message from Morpheus and you are going to exit the Matrix really soon! 🤪 Be sure to choose the RED pill just in case.

Jokes apart, using randomly generated non guessable UUID primary keys is essential to the security of your software because it often gets embedded in HTTP GET request parameters or path variables. Keeping them guessable means someone can do DELETE /credit-card/123.

OK! I hope things are making sense till now. Let’s move ahead with the initialisation of the DB connection.

Connecting To The DB

It is just simple and idiomatic. Here’s a function that does it and returns a pointer to gorm.DB using which, we can interact with the underlying database.

Again, pretty self explanatory. We create a data source name or dsn in the prescribed format mentioning host, port, username etc. and pass it to the gorm driver library to return a Dialector pointer that we pass on to gorm library along with some *gorm.Config giving us back a *gorm.DB that lets us interact with the backend using the domain structs. You would notice mapstructure tags, because I was using viper to load in the configuration, but for simplicity, we won’t delve into that in this entry. Additionally, we are enabling GORM to log the generated queries and configuring it with a NamingStrategy to compute the target table names to be used in the generated queries.

We will invoke InitialiseDB from main like so:

db, err := InitialiseDB(&DbConfig{
   User:     "postgres",
   Password: "password",
   DbName:   "groot",
   Host:     "localhost",
   Port:     "5432",
   Schema:   "users",
})
if err != nil {
   panic(err)
}
err = db.AutoMigrate(&domain.User{})
if err != nil {
   panic(err)
}

The db.AutoMigrate(&domain.User{}) will cause GORM to inspect the current schema and attempt to migrate the current schema to match what is desired. Though, one thing to note, it will attempt to do so without causing data loss. It won’t drop anything e.g. columns from existing table not present in the domain structure.

Since, now we have the gorm.DB pointer, i.e. the db variable. It is time to write the repository layer!

The UserRepository

We need a method to persist our User. So, let’s start by defining an interface that our repository will adhere to. Let’s create a package under the root of our user-service module and create a user.repository.go.

package repository

import "github.com/prithuadhikary/user-service/domain"

type UserRepository interface {
   Save(user *domain.User)
}

Plain and simple! A Save method (behavior) within a UserRepository interface. Just keep in mind that there is a reason behind declaring the interface and the method name in caps. In go, caps means exported. Any name starting with capital letters are called exported and accessible outside the package they are defined.

But it is just the interface right? We need the implementation. So let’s implement the interface within the same user.repository.go file by adding -

type userRepository struct {
   db *gorm.DB
}

But does it implement the interface? Nope! Not yet. Let’s add the method to the struct.

func (repository *userRepository) Save(user *domain.User) {
   // Implement the logic to save here.
}

In go, implementing an interface means just adhering to the contract it defines. In other words, the interface implementation is implicit and does not require you to define whether your type implements any. And the underpinning rule that a UserRepository interface variable should be able to hold the reference to a variable implementing the interface also holds. So the following is valid:

func NewUserRepository(db *gorm.DB) UserRepository {
   var repository UserRepository
   repository = &userRepository{
      db: db,
   }
   return repository
}

And we have our first constructor function! And that is why declared userRepository struct beginning with lowercase u keeping it unexported and keeping it from directly being constructed outside the package and controlling its instantiation by exposing the NewUserRepository constructor returning the userRepository pointer as UserRepository interface.

But before going further, let’s discuss something called method sets. Instead of assigning a reference(pointer) to the userRepository to the repository interface variable, if you attempted the following:

repository = userRepository{
      db: db,
}

This wouldn’t compile. Because, our structure doesn’t implement the interface, rather it is like its pointer does!

Receivers And Interfaces Under The Hood — Skippable

Let’s take a detour if you are willing! You can skip this if you want to get the repository implementation directly.

If we revisit the declaration of the save method:

func (repository *userRepository) Save(user *domain.User) {
   // Implement the logic to save here.
}

The *userRepoitory is called a receiver. The receiver receives the method call from the caller and is accessible inside the method body, assume it to be the automatic first argument to the declared method. A receiver can be a reference to a value of a type or a value of a type. So, the following declaration is also valid and results in userRepository structimplementing the UserRepository interface.

func (repository userRepository) Save(user *domain.User) {
   // Implement the logic to save here.
}

And since repository is available inside the method body and actually the first argument that gets passed by go, the principles of pass by reference or value hold for it to. A value receiver causes a copy of itself to be created and passed whereas a pointer receiver well, is a pointer to the actual receiver. So, any changes made to the pointer receiver will result in the actual receiver being modified. Actually it is always pass by value, even the pointer gets copied, coz for:

func printPointerAddress(aPtr *int32) {
        println("Address of pointer aPtr", &aPtr)
}
And then in main, 
a := int32(10)
ptr := &a
println("Addres of pointer ptr", &ptr)
printPointerAddress(ptr)

The output you would get will be similar to:

Addres of pointer ptr 0xc000084db8
Address of pointer aPtr 0xc000084dd0

ptr is a pointer but is a variable too! So, when you pass the ptr to the printPointerAddress function, the pointer gets copied and assigned to the pointer aPtr and since aPtr is a pointer, we get a handle to the original memory address which we might want to dereference and make changes to. But, it is never a true pass by reference! 😁 Because even a pointer to pointer to pointer will also get copied. It is by the virtue of the nature of pointers, it is a pass by reference.

Coming back to the declaration:

func (repository userRepository) Save(user *domain.User) {
   // Implement the logic to save here.
}

There is an additional implication that might not be seem obvious unless stated. Now both of the following will compile:

var repository UserRepository

//This is implied.
repository = userRepository{
   db: db,
}

//This is also valid.
repository = &userRepository{
   db: db,
}

So, the rule of thumb is, if the method of an implementing type is declared against:

• a pointer receiver, only the pointer to the implementing type can be assigned to the interface variable.
• a value receiver, then both, value and the pointer of the implementing type, can be assigned to the interface variable.

It might be easier to remember it like this, interface variables store receivers. If the receiver is a pointer, then interface must store the pointer receiver. But if the receiver is a value type, interface can store the address as well as the value itself. And that is what method sets encompass.

Given a pointer, you can always find out the value it points to, but provided a value, you can not always determine the address. It becomes more evident if you re-declare a primitive type as a user defined type or use it directly and try to take its address just after creating an instance of it, like so:

type Number interface {
   Increment() Number
}

type number int64

func (num *number) Increment() Number {
   *num++
   return num
}
n := number(5)
// The above declaration does three things:
// 1. Allocate memory
// 2. Give it the name 'n'
// 3. Assign the value number(5) to it.
// 4. Since there exists a named memory location,
//    we can take its address by &n (address of n).

nptr := &number(6)
// On the other hand, &number(6) isn't feasible
// even though number(6) could have caused an 
// allocation,
// we haven't got a named handle to its location.
// Likewise, &6 is invalid.

Under the hood, go interfaces are implemented as a two word data structure. The first word itab holds a pointer to something called Itab struct(from the world of C) holding metadata about the type followed by function pointers (say fun[0], fun[1]..) pointing to the functions of the implementing type that correspond to the functions declared by the interface type. The second word data holds a pointer to the data held by the implementing type. When we assign a concrete type to an interface variable, the underlying data structure, i.e. data and itab get populated. And when we invoke a method through the interface variable say iface, go does something in the lines of

Depending upon the kind of receiver fun[0] has,
iface.itab->fun[0](iface.data,...additional parameters) 
or 
iface.itab->fun[0](*iface.data, ...additional parameters)

Do remember that fun[0] means (*(fun+0)).

The problem starts when go can not derive the function pointers to populate the itab struct with, because it does not have an address to start with. For example, &number(5). It is similar to constants and literals. Though modern C allows something called compound literals that enable you to declare such constants and take their address. Go also allows you to create and take address of a struct instance but not with constants like 6 or primitive values.

Anyways, enough digging in, let’s get back to our microservice stuff, shall we?

Implementation Of The Save() Method

It is very simple. gorm.DB provides a Save(interface{}) method that takes a pointer to the domain we want to save, and thus:

func (repository userRepository) Save(user *domain.User) {
   repository.db.Save(user)
}

But we need to handle one thing! If we create a struct variable of domain.User, the ID(declared as [16]byte) field will have the value uuid.Nil(zeroed [16]byte array). Our ID needs to be a valid UUID value and if it doesn’t have one, we should generate one and set it. GORM Hooks to the rescue! GORM enables us to declare hook methods for our domain types. These hooks are executed before or after various CRUD operations that we may perform on our domain using GORM. Currently, these are the available hooks for create, as per the GORM documentation:

// begin transaction
BeforeSave
BeforeCreate
// save before associations
// insert into database
// save after associations
AfterCreate
AfterSave
// commit or rollback transaction

For an exhaustive list, please refer the GORM Hooks documentation.

Let’s install Google’s uuid package

go get -u github.com/google/uuid

and define a BeforeCreate hook for the User domain like so:

func (u *User) BeforeCreate(tx *gorm.DB) (err error) {
  u.ID = uuid.New()

  return
}

But that’s not all! We have a Password field as well. The right way to store a password in a database is to hash it(along with a salt) and store it. We can store a hex encodedsha256 digest of the password for now(leaving the salting for a later discussion). Let’s add that bit to the BeforeCreate hook too! Should look something like:

func (user *User) BeforeCreate(tx *gorm.DB) error {
   user.ID = uuid.New()
   
   // Digest and store the hex representation.
   digest := sha256.Sum256([]byte(user.Password))
   user.Password = hex.EncodeToString(digest[:])

   return nil
}

And that should do it! Next we will write the service layer which will just transform the incoming model to a domain.User and pass it on to the repository.

The User Service

Let’s start by defining the model struct which will act as the input to the Signup method to be defined in our service. Let’s name it SignupRequest and define it in signup-request.go under a directory model beneath the root of our go module.

package model

type SignupRequest struct {
   Username             string `json:"username"`
   Password             string `json:"password"`
   PasswordConfirmation string `json:"passwordConfirmation"`
}

Pretty self explanatory, the json tags though, they map the the json fields to the struct fields when we bind the incoming request body to it. Gin provides other tags that enable binding to form POST, GET request and for applying input validation using predefined or custom validators. We will look into it in more detail when we write the controller layer.

Let’s define the UserService interface in user.service.go under the package service.

package service

import "github.com/prithuadhikary/user-service/model"

type UserService interface {
   Signup(request *model.SignupRequest) error
}

And let’s implement the interface by defining a struct and implementing the Save method in the same user.service.go file.

Very easy to understand, we compare the password and the confirm password fields, and if valid, we check if there already exists a user for the same username/password combination. If not, we save the user and return. On any validation failure, we are returning an error. The error is an interface with a single method Error() string defined in the standard go library. We will look into how we can declare our own custom error type with additional fields along with the middleware to handle it in an upcoming entry.

You would notice the use of a new method

service.repository.ExistsByUsername(request.Username)

I added it to the repository struct because I kinda forgot about it. 😐 But here it is:

func (repository *userRepository) ExistsByUsername(username string) bool {
   var count int64
   repository.db.Model(&domain.User{}).Where("username = ?", username).Count(&count)
   return count > 0
}

Again, pretty self explanatory. It checks if a user exists by a username. We use the *gorm.DB's Count method which takes a pointer to an int64 to set its find in it.

Ohk!! A hell lot of info. Let’s quickly wrap this up with the controller.

Finally! The User Controller

The interface(defined in user-service/controller/user.controller.go) :

package controller

import "github.com/gin-gonic/gin"

type UserController interface {
   Signup(ctx *gin.Context)
}

The Signup method takes in a gin.Context pointer. It conforms to a function type declaration called the HandlerFunc defined in the gin library:

type HandlerFunc func(*Context)

HandlerFunc -> Handler Function! It declares the signature of the function that will Handle an incoming HTTP request. Its basically a callback that if configured against a route, will get invoked upon receiving an HTTP request at that route.

Let’s look at the implementation of the Signup method.

The Signup method attempts to bind the incoming request body to a model.SignupRequest struct by passing in its address to the ctx.ShouldBind method provided by the gin.Context struct. The ShouldBind method inspects the incoming Content-Type header, selects a strategy to deserialise the request body and then bind and validate the parsed parameter map according to the binding tags if any, present against the struct fields pointed by the pointer passed to it. In our case, if it found application/json, it would attempt to map the incoming JSON request body to the model.SignupRequest struct pointed by the pointer it was passed. But!! We forgot to add the binding tags as far as I can remember!

So, let’s add the binding tags to the fields of the SignupRequest structure.

In addition to the json tags which help map the incoming json properties to the target struct fields, the binding tags specify the constraints against each field that must be respected in order for the binding to succeed. Going back to the Signup method implementation, if the binding succeeded, it would just invoke the Signup method of the UserService method, which if returns an error, will be rendered in the response, otherwise a 200 OK is returned to the client.

The constructor function NewUserController is also straight forward. It accepts as parameters, the *gorm.Engine and the UserService it needs. It creates a gin.RouterGroup with the path /api and registers the userController.Sigup handler function at path /users under the /api group, thus essentially making /api/users the effective path for the Signup handler function.

Improving The Validation Error Response

ShouldBind returns an error implementation of type validator.ValidationErrors , iterating which we can find and construct an error response that would contain the fields and their corresponding tags that failed the validation. If we just render a JSON with a message “Validation failed”, the client won’t know what failed. So, Let’s improve the validation error response a bit.

Have a look at the following function:

It just does that, it iterates over the field errors and constructs an array to be sent as json response to the client, along with an HTTP status of 400 Bad Request.

And thus our Signup method changes a bit:

errors.As is a builtin function that returns true if it succeeded in unwrapping its first argument to the target error implementation. You can find more on the errors package available in the standard library from the documentation.

Wiring It Altogether

Here is the main function.

Which does the following in sequence:

• Initializes GORM by connecting to postgres.
• Automigrates — Applies the desired schema.
• Creates the user respository using the NewUserRepository constructor function passing the *gorm.DB obtained in the first step.
• Creates the user service using the NewUserService constructor passing in the user repository.
• Obtains a pointer to agin.Engine with default configuration and creates the user controller using the NewUserController constructor passing the engine and the user service.
• Finally starts the gin engine to listen on port 8080.

And the logs:

...
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:   export GIN_MODE=release
 - using code:  gin.SetMode(gin.ReleaseMode)
[GIN-debug] POST   /api/users                --> github.com/prithuadhikary/user-service/controller.userController.Signup-fm (3 handlers)
[GIN-debug] [WARNING] You trusted all proxies, this is NOT safe. We recommend you to set a value.
[GIN-debug] Listening and serving HTTP on :8080

And:

CURL Output

OHK!!! It was a big one, but I hope it was informative.

Github Repo: https://github.com/prithuadhikary/user-service

Adios!