How to Architect Good Go Backend REST API Services

Let me start by asking a question, what is the best way to write software?

You will definitely say this is a vague question. It will depend on what you are building, how much you can look into the future, and most importantly how much time and money you can spend in developing the best possible solution for the given problem.

This iterative approach has resulted in some rule of thumb. My article and the goserve backend framework which is the topic of this article is about those learnings which will help you to reduce the iterations while developing REST API services from scratch.

Now, you may argue that the best solution is to work on microservices and prepare for the scale that eventually needs to be addressed. For that I will say “Yes”. I will also quote “You don’t need a bomb to kill an ant”. For the remaining part of this article, I will discuss the solution that is typically needed when you start building your product to get PMF (product market fit).

Let’s begin with those requirements in mind. So, you have to write a backend API service, now which framework to choose, which pattern to follow, and most of all which language to adopt. Or simply why Go?

I have worked on many languages and frameworks, and until recently I have been writing backend services in Javascript/Typescript and I see myself writing many more such services in the future with them. I have also written extensively in Java and Kotlin. So, I can surely tell you, that no one language is superior to another, but each one has its beauty. But we might choose one over the other based on many considerations, maybe job availability or performance. Go is unique in this respect, it is relatively new and adopts simplicity over a bunch of features. It does not have similar generics capabilities like Java or freedom like Javascript. But it is a language that you will appreciate when you experience the simplicity of writing programs and freedom from the ocean of libraries which other languages heavily rely on. Go is compiled but feels like an interpreted language and its fast in both compilation and execution. So, here is the reason to give Go a try.

Now, let’s come back to the topic, what are the main requirements for a common REST API service?

1. Database to store states
2. Logic to mutate those states
3. Help clients interface with the end user
4. Data access control
5. Ease of adding new features
6. Minimal DevOps or deployment on servers
7. Caching to reduce redundant operations

Few more things to consider

1. Consistency of your code base
2. Reusability of already developed features
3. Reduce code conflicts when many developers work together
4. Writing tests to ensure stability
5. Reducing redundancy in code

“For all these points a framework is very useful.”

A Quick tour of the goserve backend framework — GitHub Repo Link

Sample Project Structure:

.
├── .extra
│   └── setup
│       └── init-mongo.js
├── api
│   └── sample
│       ├── dto
│       │   └── create_sample.go
│       ├── model
│       │   └── sample.go
│       ├── controller.go
│       └── service.go
├── cmd
│   └── main.go
├── config
│   └── env.go
├── keys
│   ├── private.pem
│   └── public.pem
├── startup
│   ├── indexes.go
│   ├── module.go
│   ├── server.go
│   └── testserver.go
├── utils
│   └── convertor.go
├── .env
├── .test.env
├── .gitignore
├── .dockerignore
├── Dockerfile
├── docker-compose.yml
├── go.mod
└── go.sum

Let’s take bottom to top approach. I am going to discuss the goserve example for blog service implementation.

Note: The blog service example has more apis implemented as compared to the sample project structure given above.

You will start the program from cmd/main.go. The main function just calls the Server function in the package startup

package main

import "github.com/unusualcodeorg/goserve/startup"

func main() {
 startup.Server()
}

What will the Server function do?

It will assemble all the things required to enable the server to do its job.

Who are the main characters in this story?

1. Controller — It receives a request and based on the logic sends a response.
2. Service — It is the logic part of the program. It takes an input, processes it through a database, cache, etc, and return an output. It is essentially the logic component that assists a controller.
3. Model — It is a representation of data that is stored in a database. It defines the fields and validations over those data. It is passed around the program for running logic.
4. DTO — This is a data representation of the information exchanged over the network. Basically, it represents the data being received in a request and sent in a response. It also defines the validations of those data to make them safe for processing.
5. Database — It is a client that helps to communicate with the database server. In most cases, you have to create a utility to make database operation simpler in your project.

Who are the side characters of the story?

1. Request — A helper to extract data received over the network
2. Response — A helper to simplify the process of returning the data over the network
3. Router — A helper to attach an appropriate controller to a request
4. Query — A helper to make database operation simpler
5. Store — A helper to perform cache operations
6. Context — It holds values in the request processing sequence

Who are the villains in the story?

1. Key Protection Middleware — It only allows the request to pass if it has a secret key with it.
2. Authentication Middleware — It only allows the request to pass if it has a valid identity or token. It guards a few endpoints to be accessed only by the approved clients.
3. Authorization Middleware — If block a client from accessing any restricted resource. The business can control who can access how much of the information.

Let’s see the request-response flow for the public, private, and protected APIs.

Now let’s come back to the Server

package startup

import (
 "context"
 "time"

 "github.com/gin-gonic/gin"
 "github.com/unusualcodeorg/goserve/arch/mongo"
 "github.com/unusualcodeorg/goserve/arch/network"
 "github.com/unusualcodeorg/goserve/arch/redis"
 "github.com/unusualcodeorg/goserve/config"
)

type Shutdown = func()

func Server() {
 env := config.NewEnv(".env", true)
 router, _, shutdown := create(env)
 defer shutdown()
 router.Start(env.ServerHost, env.ServerPort)
}

func create(env *config.Env) (network.Router, Module, Shutdown) {
 context := context.Background()

 dbConfig := mongo.DbConfig{
  User:        env.DBUser,
  Pwd:         env.DBUserPwd,
  Host:        env.DBHost,
  Port:        env.DBPort,
  Name:        env.DBName,
  MinPoolSize: env.DBMinPoolSize,
  MaxPoolSize: env.DBMaxPoolSize,
  Timeout:     time.Duration(env.DBQueryTimeout) * time.Second,
 }

 db := mongo.NewDatabase(context, dbConfig)
 db.Connect()

 if env.GoMode != gin.TestMode {
  EnsureDbIndexes(db)
 }

 redisConfig := redis.Config{
  Host: env.RedisHost,
  Port: env.RedisPort,
  Pwd:  env.RedisPwd,
  DB:   env.RedisDB,
 }

 store := redis.NewStore(context, &redisConfig)
 store.Connect()

 module := NewModule(context, env, db, store)

 router := network.NewRouter(env.GoMode)
 router.RegisterValidationParsers(network.CustomTagNameFunc())
 router.LoadRootMiddlewares(module.RootMiddlewares())
 router.LoadControllers(module.Controllers())

 shutdown := func() {
  db.Disconnect()
  store.Disconnect()
 }

 return router, module, shutdown
}

The first thing the server needs is an environment variable holder i.e config/env.go. It simply reads .env file and populates Env object.

package config

import (
 "log"

 "github.com/spf13/viper"
)

type Env struct {
 // server
 GoMode     string `mapstructure:"GO_MODE"`
 ServerHost string `mapstructure:"SERVER_HOST"`
 ServerPort uint16 `mapstructure:"SERVER_PORT"`
 // database
 DBHost         string `mapstructure:"DB_HOST"`
 DBName         string `mapstructure:"DB_NAME"`
 DBPort         uint16 `mapstructure:"DB_PORT"`
 DBUser         string `mapstructure:"DB_USER"`
 DBUserPwd      string `mapstructure:"DB_USER_PWD"`
 DBMinPoolSize  uint16 `mapstructure:"DB_MIN_POOL_SIZE"`
 DBMaxPoolSize  uint16 `mapstructure:"DB_MAX_POOL_SIZE"`
 DBQueryTimeout uint16 `mapstructure:"DB_QUERY_TIMEOUT_SEC"`
 // redis
 RedisHost string `mapstructure:"REDIS_HOST"`
 RedisPort uint16 `mapstructure:"REDIS_PORT"`
 RedisPwd  string `mapstructure:"REDIS_PASSWORD"`
 RedisDB   int    `mapstructure:"REDIS_DB"`
 // keys
 RSAPrivateKeyPath string `mapstructure:"RSA_PRIVATE_KEY_PATH"`
 RSAPublicKeyPath  string `mapstructure:"RSA_PUBLIC_KEY_PATH"`
 // Token
 AccessTokenValiditySec  uint64 `mapstructure:"ACCESS_TOKEN_VALIDITY_SEC"`
 RefreshTokenValiditySec uint64 `mapstructure:"REFRESH_TOKEN_VALIDITY_SEC"`
 TokenIssuer             string `mapstructure:"TOKEN_ISSUER"`
 TokenAudience           string `mapstructure:"TOKEN_AUDIENCE"`
}

func NewEnv(filename string, override bool) *Env {
 env := Env{}
 viper.SetConfigFile(filename)

 if override {
  viper.AutomaticEnv()
 }

 err := viper.ReadInConfig()
 if err != nil {
  log.Fatal("Error reading environment file", err)
 }

 err = viper.Unmarshal(&env)
 if err != nil {
  log.Fatal("Error loading environment file", err)
 }

 return &env
}

Next, it connects to a Mongo database, and then to a Redis database. goserve provides a simple abstraction for these operations.

It then creates the instances of all the characters mentioned above. In this list Module and Router have the main role.

The Router is part of the goserve framework and it creates functions over the Gin to simplify the process of creating and mounting handlers.

Let’s check the module. It implements the framework’s Module interface, and create services, controllers, and middlewares. It acts as the dependency manager to create and distribute object instances. Here RootMiddlewares is attached to all the requests. But the interesting part is AuthenticationProvider and AuthorizationProvider. They basically are passes to each controller so that they can decide which endpoints need them.

type Module[T any] interface {
 GetInstance() *T
 RootMiddlewares() []RootMiddleware
 Controllers() []Controller
 AuthenticationProvider() AuthenticationProvider
 AuthorizationProvider() AuthorizationProvider
}

The main idea here is to make controllers independent in deciding over the access control.

package startup

import (
 "context"

 "github.com/unusualcodeorg/goserve/api/auth"
 authMW "github.com/unusualcodeorg/goserve/api/auth/middleware"
 "github.com/unusualcodeorg/goserve/api/blog"
 "github.com/unusualcodeorg/goserve/api/blog/author"
 "github.com/unusualcodeorg/goserve/api/blog/editor"
 "github.com/unusualcodeorg/goserve/api/blogs"
 "github.com/unusualcodeorg/goserve/api/contact"
 "github.com/unusualcodeorg/goserve/api/user"
 coreMW "github.com/unusualcodeorg/goserve/arch/middleware"
 "github.com/unusualcodeorg/goserve/arch/mongo"
 "github.com/unusualcodeorg/goserve/arch/network"
 "github.com/unusualcodeorg/goserve/arch/redis"
 "github.com/unusualcodeorg/goserve/config"
)

type Module network.Module[module]

type module struct {
 Context     context.Context
 Env         *config.Env
 DB          mongo.Database
 Store       redis.Store
 UserService user.Service
 AuthService auth.Service
 BlogService blog.Service
}

func (m *module) GetInstance() *module {
 return m
}

func (m *module) Controllers() []network.Controller {
 return []network.Controller{
  auth.NewController(m.AuthenticationProvider(), m.AuthorizationProvider(), m.AuthService),
  user.NewController(m.AuthenticationProvider(), m.AuthorizationProvider(), m.UserService),
  blog.NewController(m.AuthenticationProvider(), m.AuthorizationProvider(), m.BlogService),
  author.NewController(m.AuthenticationProvider(), m.AuthorizationProvider(), author.NewService(m.DB, m.BlogService)),
  editor.NewController(m.AuthenticationProvider(), m.AuthorizationProvider(), editor.NewService(m.DB, m.UserService)),
  blogs.NewController(m.AuthenticationProvider(), m.AuthorizationProvider(), blogs.NewService(m.DB, m.Store)),
  contact.NewController(m.AuthenticationProvider(), m.AuthorizationProvider(), contact.NewService(m.DB)),
 }
}

func (m *module) RootMiddlewares() []network.RootMiddleware {
 return []network.RootMiddleware{
  coreMW.NewErrorCatcher(), // NOTE: this should be the first handler to be mounted
  authMW.NewKeyProtection(m.AuthService),
  coreMW.NewNotFound(),
 }
}

func (m *module) AuthenticationProvider() network.AuthenticationProvider {
 return authMW.NewAuthenticationProvider(m.AuthService, m.UserService)
}

func (m *module) AuthorizationProvider() network.AuthorizationProvider {
 return authMW.NewAuthorizationProvider()
}

func NewModule(context context.Context, env *config.Env, db mongo.Database, store redis.Store) Module {
 userService := user.NewService(db)
 authService := auth.NewService(db, env, userService)
 blogService := blog.NewService(db, store, userService)

 return &module{
  Context:     context,
  Env:         env,
  DB:          db,
  Store:       store,
  UserService: userService,
  AuthService: authService,
  BlogService: blogService,
 }
}

The Router interface is provided by the goserve framework, and its default implementation is also provided by the framework in network.NewRouter

type Router interface {
 GetEngine() *gin.Engine
 RegisterValidationParsers(tagNameFunc validator.TagNameFunc)
 LoadControllers(controllers []Controller)
 LoadRootMiddlewares(middlewares []RootMiddleware)
 Start(ip string, port uint16)
}

Let’s now focus on the apis.

The main goal is to make each api independent and isolated as much as possible. I call it feature encapsulation. Basically, one developer working on an api can work without much worry about other developer working on some other api.

For most part, these are one-time setups and you can use the starter template generator for goserve Framework namely goservegen (GitHub Repo Link)

Now let’s check a model. It implements the interface Document provided by the goserve framework mongo.

type Document[T any] interface {
 EnsureIndexes(Database)
 GetValue() *T
 Validate() error
}

Message Model implementation

package model

import (
 "time"

 "github.com/go-playground/validator/v10"
 "github.com/unusualcodeorg/goserve/arch/mongo"
 "go.mongodb.org/mongo-driver/bson/primitive"
)

const CollectionName = "messages"

type Message struct {
 ID        primitive.ObjectID `bson:"_id,omitempty" validate:"-"`
 Type      string             `bson:"type" validate:"required"`
 Msg       string             `bson:"msg" validate:"required"`
 Status    bool               `bson:"status" validate:"required"`
 CreatedAt time.Time          `bson:"createdAt" validate:"required"`
 UpdatedAt time.Time          `bson:"updatedAt" validate:"required"`
}

func NewMessage(msgType string, msgTxt string) (*Message, error) {
 time := time.Now()
 m := Message{
  Type:      msgType,
  Msg:       msgTxt,
  Status:    true,
  CreatedAt: time,
  UpdatedAt: time,
 }
 if err := m.Validate(); err != nil {
  return nil, err
 }
 return &m, nil
}

func (message *Message) GetValue() *Message {
 return message
}

func (message *Message) Validate() error {
 validate := validator.New()
 return validate.Struct(message)
}

func (*Message) EnsureIndexes(db mongo.Database) {

}

Let’s come to the Dto Interface provided by the goserve framework

type Dto[T any] interface {
 GetValue() *T
 ValidateErrors(errs validator.ValidationErrors) ([]string, error)
}

Dto implementation at api/contact/dto/create_message.go

Here ValidateErrors is needed to send personalised errors, and it is called when the controller parses the body into a Dto.

package dto

import (
 "fmt"

 "github.com/go-playground/validator/v10"
)

type CreateMessage struct {
 Type string `json:"type" binding:"required,min=2,max=50"`
 Msg  string `json:"msg" binding:"required,min=0,max=2000"`
}

func EmptyCreateMessage() *CreateMessage {
 return &CreateMessage{}
}

func (d *CreateMessage) GetValue() *CreateMessage {
 return d
}

func (d *CreateMessage) ValidateErrors(errs validator.ValidationErrors) ([]string, error) {
 var msgs []string
 for _, err := range errs {
  switch err.Tag() {
  case "required":
   msgs = append(msgs, fmt.Sprintf("%s is required", err.Field()))
  case "min":
   msgs = append(msgs, fmt.Sprintf("%s must be min %s", err.Field(), err.Param()))
  case "max":
   msgs = append(msgs, fmt.Sprintf("%s must be max%s", err.Field(), err.Param()))
  default:
   msgs = append(msgs, fmt.Sprintf("%s is invalid", err.Field()))
  }
 }
 return msgs, nil
}

The Controller interface provided by the goserve framework is quite involved, and I will give only a brief introduction here. In later articles, I will talk in detail about it. The goserve defines the following interfaces for the Controller functionality.

type SendResponse interface {
 SuccessMsgResponse(message string)
 SuccessDataResponse(message string, data any)
 BadRequestError(message string, err error)
 ForbiddenError(message string, err error)
 UnauthorizedError(message string, err error)
 NotFoundError(message string, err error)
 InternalServerError(message string, err error)
 MixedError(err error)
}

type ResponseSender interface {
 Debug() bool
 Send(ctx *gin.Context) SendResponse
}

type BaseController interface {
 ResponseSender
 Path() string
 Authentication() gin.HandlerFunc
 Authorization(role string) gin.HandlerFunc
}

type Controller interface {
 BaseController
 MountRoutes(group *gin.RouterGroup)
}

Let’s see our controller implementation at api/contact/message/controller.go

package contact

import (
 "github.com/gin-gonic/gin"
 "github.com/unusualcodeorg/goserve/api/contact/dto"
 "github.com/unusualcodeorg/goserve/arch/network"
 "github.com/unusualcodeorg/goserve/utils"
)

type controller struct {
 network.BaseController
 service Service
}

func NewController(
 authProvider network.AuthenticationProvider,
 authorizeProvider network.AuthorizationProvider,
 service Service,
) network.Controller {
 return &controller{
  BaseController: network.NewBaseController("/contact", authProvider, authorizeProvider),
  service:        service,
 }
}

func (c *controller) MountRoutes(group *gin.RouterGroup) {
 group.POST("/", c.createMessageHandler)
}

func (c *controller) createMessageHandler(ctx *gin.Context) {
 body, err := network.ReqBody(ctx, &dto.CreateMessage{})
 if err != nil {
  c.Send(ctx).BadRequestError(err.Error(), err)
  return
 }

 msg, err := c.service.SaveMessage(body)
 if err != nil {
  c.Send(ctx).InternalServerError("something went wrong", err)
  return
 }

 data, err := utils.MapTo[dto.InfoMessage](msg)
 if err != nil {
  c.Send(ctx).InternalServerError("something went wrong", err)
  return
 }

 c.Send(ctx).SuccessDataResponse("message received successfully!", data)
}

1. NewController — Creates the controller instance by receiving the required instances through the module. It also creates a base instance using network.NewBaseController which defines the base path for the controller.
2. MountRoutes — This is called by the Router when the server is created. It loads the handler functions and other middleware for the endpoints.
3. createMessageHandler — The controller defined many handlers to process the request for the endpoints.

How to process the request?

1. network.ReqBody is one of the utility functions that enables parsing the body into our Dto.
2. Send error response using sender. The base controller instance provides a Send function which exposes other functions to send appropriate responses. The error processing is abstracted by the goserve framework.
3. Translation between Model to Dto using utils.MapTo

Now let’s see the api/contact/service.go

It defines an interface and implements it. This is very useful while testing using mock.

package contact

import (
 "github.com/unusualcodeorg/goserve/api/contact/dto"
 "github.com/unusualcodeorg/goserve/api/contact/model"
 coredto "github.com/unusualcodeorg/goserve/arch/dto"
 "github.com/unusualcodeorg/goserve/arch/mongo"
 "github.com/unusualcodeorg/goserve/arch/network"
 "go.mongodb.org/mongo-driver/bson"
 "go.mongodb.org/mongo-driver/bson/primitive"
)

type Service interface {
 SaveMessage(d *dto.CreateMessage) (*model.Message, error)
 FindMessage(id primitive.ObjectID) (*model.Message, error)
 FindPaginatedMessage(p *coredto.Pagination) ([]*model.Message, error)
}

type service struct {
 network.BaseService
 messageQueryBuilder mongo.QueryBuilder[model.Message]
}

func NewService(db mongo.Database) Service {
 return &service{
  BaseService:         network.NewBaseService(),
  messageQueryBuilder: mongo.NewQueryBuilder[model.Message](db, model.CollectionName),
 }
}

func (s *service) SaveMessage(d *dto.CreateMessage) (*model.Message, error) {
 msg, err := model.NewMessage(d.Type, d.Msg)
 if err != nil {
  return nil, err
 }

 result, err := s.messageQueryBuilder.SingleQuery().InsertAndRetrieveOne(msg)
 if err != nil {
  return nil, err
 }

 return result, nil
}

func (s *service) FindMessage(id primitive.ObjectID) (*model.Message, error) {
 filter := bson.M{"_id": id}

 msg, err := s.messageQueryBuilder.SingleQuery().FindOne(filter, nil)
 if err != nil {
  return nil, err
 }

 return msg, nil
}

func (s *service) FindPaginatedMessage(p *coredto.Pagination) ([]*model.Message, error) {
 filter := bson.M{"status": true}

 msgs, err := s.messageQueryBuilder.SingleQuery().FindPaginated(filter, p.Page, p.Limit, nil)
 if err != nil {
  return nil, err
 }

 return msgs, nil
}

Here messageQueryBuilder provides functions to make mongo queries.

For fyi lets see the Query builder interface provided by the framework

type QueryBuilder[T any] interface {
 GetCollection() *mongo.Collection
 SingleQuery() Query[T]
 Query(context context.Context) Query[T]
}

type Query[T any] interface {
 Close()
 CreateIndexes(indexes []mongo.IndexModel) error
 FindOne(filter bson.M, opts *options.FindOneOptions) (*T, error)
 FindAll(filter bson.M, opts *options.FindOptions) ([]*T, error)
 FindPaginated(filter bson.M, page int64, limit int64, opts *options.FindOptions) ([]*T, error)
 InsertOne(doc *T) (*primitive.ObjectID, error)
 InsertAndRetrieveOne(doc *T) (*T, error)
 InsertMany(doc []*T) ([]primitive.ObjectID, error)
 InsertAndRetrieveMany(doc []*T) ([]*T, error)
 UpdateOne(filter bson.M, update bson.M) (*mongo.UpdateResult, error)
 UpdateMany(filter bson.M, update bson.M) (*mongo.UpdateResult, error)
 DeleteOne(filter bson.M) (*mongo.DeleteResult, error)
}

Note: A Service takes in Dto in most cases and sends out Dto as well.

Phew!! It was quite a long writing session. I will write individually on these topics in the next articles. So, do follow me to receive them once I publish.

Oh, I forgot, Authentication and Authorization. You can find them in the repo — Link

Let’s see how to use them in api/blog/author/controller.go

func (c *controller) MountRoutes(group *gin.RouterGroup) {
 group.Use(c.Authentication(), c.Authorization(string(userModel.RoleCodeAuthor)))
 group.POST("/", c.postBlogHandler)
 group.PUT("/", c.updateBlogHandler)
 group.GET("/id/:id", c.getBlogHandler)
 group.DELETE("/id/:id", c.deleteBlogHandler)
 group.PUT("/submit/id/:id", c.submitBlogHandler)
 group.PUT("/withdraw/id/:id", c.withdrawBlogHandler)
 group.GET("/drafts", c.getDraftsBlogsHandler)
 group.GET("/submitted", c.getSubmittedBlogsHandler)
 group.GET("/published", c.getPublishedBlogsHandler)
}

You call the Authentication and Authorization provider function and receive the handler function.

Now, you can explore the repo in detail and I am sure you will find it a good time-spending exercise.

Recommended Article: How to Create Microservices — A Practical Guide Using Go

Thanks for reading this article. Be sure to share this article if you found it helpful. It would let others get this article and spread the knowledge. Also, putting a clap will motivate me to write more such articles

Find more about me on janisharali.com

Let’s become friends on Twitter, Linkedin, and Github