Golang Microservice Design

Article by Weave, written by Kas Buunk
Part 2 of 3 articles

Now that the theory of a layered architecture has been laid out, it is time to turn towards implementing this design. All the following will purely contain Go-specific code and terminology. However, keep in mind that many programming languages have sufficiently similar concepts to apply the same concepts, even though they may have different names or methods of abstraction.

Keep in mind that, for brevity, this article refrains from test files, error handling and other due diligence that distracts from illustrating the key take-aways.

Packages

As one may have anticipated, the aforementioned layers will be reflected as Go packages, or directories containing packages. We will design a User service, whose responsibility is to manage users’ accounts in our system. To be quite pragmatic, let’s jump right in with the directory structure of a complete microservice.

user
└─app # App layer
│ └─ app.go # implements domain logic
└─event
│ └─ streams.go # defines owned streams & subjects
└─internal # contains adapters to plug in as internal dependencies
│ └─ config # immutable configuration loaded at startup
│ └─ othersvc # client to svc in cluster or other third-party
│ └─ publish # publish events
│ └─ repository # storage and retrieval adapter
└─model # Domain core models
│ └─ user.go
└─transport # special adapters to expose app to external environment
│ └─ gql
│ └─ grpc
│ └─ subscribe # eventbus subscription
└─proto # Protobuf schema + generated code
│ └─ user.proto
│ └─ user.pb.go
└─ main.go # construct app & adapters and start transport processes

Directory and package names

Obviously, directory and package names can freely be changed. The directory hierarchy has little meaning, other than to reflect the package’s relation towards one another. It is a common Go convention to have little or no directories that don’t contain any Go files themselves. This is reflected in the standard library. `internal` is the exception, because it serves a special role. The Go compiler enforces some import restrictions if packages are contained under `internal`. Read more in Dave Cheney’s post.

This article chooses a middle ground. Under internal, we include the packages that expose an adapter. All non-internal packages can be imported by other microservices, but this can be dealt with via code review or a simple tool that enforces restrictions that packages with certain names are restricted from importing packages with particular names.

main

The microservice’s dependencies are tied together in the main.go file. This file:

• constructs a Config instance, which loads in configuration through environment variables.
• calls other adapter constructors with their respective configuration, which each may require other adapters’ contracts.
• constructs the domain logic’s app.App, injecting the implementation of its dependencies.
• constructs the service’s transports, injecting the app.App, and starts their process. For example, for an http server transport, this is the method http.ListenAndServe(address).

config

Each of these Configs declare the data they need to initialise. The config.Config struct collects these and provides the constructor that retrieves the values in some way, once, from the environment or file system. Potentially, some validation is done to exit the service initialisation at startup-time.

package config

import (
  // the packages containing their Config structs.
)

// Config contains data for other service data structures to be constructed.
type Config struct {
  GQLServer gql.Config
  DB storage.Config
  Postmark emailclient.Config
}

func (c *Config) validate() error {
  // Perform checks, e.g. parse urls of apis, etc.
  return nil
}

func New() (*Config, error) {
  var conf Config
  
  // Initialise with environment variables, file system, command-line arguments, etc.
  err := conf.validate()
  // Handle err.
  
  return &conf, nil
}

model

The model package provides domain models through type aliases and data structures that model the domain objects. This package imports no other packages internal to the microservice, but may import some generic reusable types to model e.g. uuids, addresses or decimal fields of the data structures.

package model

import "github.com/third-party/uuid"

type User struct {
  ID uuid.UUID // Third-party type.
  Email string // Possibly such fields have their own type alias, to model validation through their constructor.
  Password string // Not stored.
  PasswordHash string // Stored.
}

app

The app package contains a concrete App struct, which implicilty implements the transport.App interface, which is the set of methods that we define to be the domain behaviour of the service. To fullfil its domain behaviour, it needs to perform State Changes and Effects, which is where its dependencies come into play. One such dependency may be a UserRepository field, containing another contract interface implemented by an adapter.

package app

// Specify what the App needs from a Repository. This happens to
// be implemented in the repository adapter, but the App need not know 
// this. This pattern is called "Dependency Inversion".
type UserRepository interface {
  CreateUser(ctx context.Context, email, passwordHash string) (*model.User, error)
}

type App struct {
  userRepository UserRepository
}

func (a *App) SignUp(ctx context.Context, email, password string) (*model.User, error) {
  // Validate input.
  err := validateEmail(email)
  // Handle err.
  err = validatePassword(password)
  // Handle err.
  // Domain logic.
  passwordHash := hashPassword(password)
  // Let adapter manage Effects.
  user, err := a.userRepository.CreateUser(ctx, email, passwordHash)
  // Handle err.
  return user, nil
}

func (a *App) SignIn(ctx context.Context, email, password string) (model.User, error) {
  // …
}

internal/**

An example of an adapter instance is the repository package, located in a subdirectory of internal. It exports the UserRepository which implicitly implements the app.UserRepository interface. It imports the service’s model package, because its responsibility is to store and retrieve domain model entities. It serves the domain to abstract away these responsibilities behind the contract interface it implements. The app will get the interface injected in the main package, rather than the UserRepository instance itself.

package repository

import (
  "database/sql"
  
  "lab.weave.nl/project/svc/user/model"
)

type UserRepository struct {
  db *sql.DB
}

func (u *UserRepository) CreateUser(
  ctx context.Context, email string, passwordHash string,
) (*model.User, error) {
  var user model.User
  // Execute query that inserts the user, using u.db.
  return &user, nil
}

Readers aware of the ports and adapters pattern will recognise the interfaces to play the role of ports here, and the internal packages’ exported data structures play the role of the adapters.

transport/**

The transport packages may just as well live inside internal. It’s just that these happen to invoke domain-specific behaviour. But there are times where the dependency direction goes both ways. For instance, an event bus implementation can both subscribe to events and call methods on the app.App, and perhaps the app publishes events back after some domain behaviour has executed.

Anyway, typical examples of transport packages can be grpc, rest and graphql. It is any package that exposes an interface from outside the program to input the domain understands, and provides error handling accordingly.

package rest

import (
  "net/http"

  "github.com/cockroachdb/errors"

  "lab.weave.nl/project/svc/user/contract"
)

// Again, App is implicitly implemented by app.App, but the transport 
// package need not know this.
type App interface {
  SignUp(ctx context.Context, email, password string) (*model.User, error)
  SignIn(ctx context.Context, email, password string) (*jwt.Token, error)
  // ChangePassword, UpdateUser, VerifyEmailAddress, …
}

// Server is the http server that provides the transport to the application.
type Server struct {
  app App // Dependency injection of the app layer.
  // fields for additional configuration
}

// New constructs a new server.
func New(app contract.App) Server {
  return Server{
    app: app,
  }
}

// ListenAndServe starts the http server. It should probably run in a goroutine.
func (s *Server) ListenAndServe(address string) error {
  http.HandleFunc("/", s.myRoute)
  err := http.ListenAndServe(address, nil)
  // Return error, since ListenAndServe should only return in case of an error.
  return errors.Wrap(err, "starting http server")
}

func (s *Server) myRoute(w http.ResponseWriter, r *http.Request) {
  // Parse user details from request.
  response, err := s.app.SignUp(r.Context(), user, password)
  // Handle error.
  // Translate domain response to make it RESTful.
  _, err := w.Write(restResponse)
  // Handle error.
}

This concludes a simple population of a Go microservice architecture. The same architecture actually scales to higher complexity in terms of more adapters and domain behaviour. A better article name could have been Go Program Design, because the same patterns and directory structure easily fit a CLI or daemon process. Basically, a program with any API to communicate with an external environment can be appropriate for the layered design.

Next up

In the third and last part, we introduce how one microservice can be modularised into multiple app modules, such that the same layered architecture is shown to be applicable to modular monoliths.