Crash course in Go kit — Part I: the basics
2021 was an interesting year for many of us as well as the last one, and the pandemic changed a lot in our lives. Lockdowns, stay-home policies, etc. lead into tremendous increase in use of e-commerce, q-commerce as well as delivery services.
Technically speaking, working in an environment where we process around 100 orders/sec and aiming 99.99% uptime SLA comes with a constant challenge to increase the scale and robustness of our services. Performance tests are usually showing us are we around our limits or not, and we can often achieve great results by doing some smart tweaks/refactoring — allowing the services to handle more. However, that’s not always the case since the bottlenecks may arise due to monolithic approaches as well.
Just to clarify — are these monolithic approaches are pathological or not? The answer is indeed no! We always start building our services using the monolith-first pattern, allowing us to move fast and avoid premature optimisation, since tearing out and maintaining yet another service always comes with some overhead.
If you do it otherwise you create a risk of never getting there or being too late to penetrate the market. But also, over some time we gather better knowledge about new requirements: sometimes it’s about new features to support the business, and sometimes it’s about scaling more to support further growth.
In my case, the journey stared with reviewing our systems architecture. We identified bottlenecks caused by monolithic approaches — that worked well to create/grow the product — and planned to break it down into product-owned microservices — fostering an architecture that scales. It also affected the squad topology: so instead of working on the same backlog with a team of 20+ talented engineers, we managed to facilitate stream-aligned and subsystem squads with less cognitive load — less meal on each’s plate.
Right after defining the product ownership and squad topologies, the next big thing was making tech decisions. Our teams had solid track success delivering cloud-native distributed systems using Python and Go so we decided to keep build in Go.
The decision was not just about having shipped battle-tested Go services (e.g. a country announces a lockdown, orders soar, imagine the spikes on service dashboards) but it was also fairly simple/easy to learn and adapt for newcomers, and although not anywhere near Python — still has a strong ecosystem. But a rather simpler ecosystem since in most cases the standard libraries were quite OK to support such capabilities using different serialisation formats/transports (e.g. gRPC as first-class citizen) as well as logging, distributed tracing, instrumentation rather than using frameworks in Python for the same job and dealing with opinions on their underlying logic. And plenty of simple libraries — e.g. you don’t like the default MUX then just use another one. All lightly opinionated, since the language itself was designed for interoperability.
Being really strong opinionated against using frameworks, I give a shot at Go kit (official repo https://github.com/go-kit/kit), rather a set of abstractions, libraries and interfaces filling the gaps in standard libraries based on a clean architecture, best practices and idioms, and thus leading into technical excellence by helping engineers to standardise implementation our services.
Well I am definitely neither a Go guru, nor have years of production experience using Go kit. I rather like to read, hear out from my peers and then build some toy programs so that I can start mastering on certain topics and evangelise the outcomes with my teams. And I decided to come up with writing this article not just guiding my teams building microservices using Go kit, but also to share this experience with the community.
In this post, I will make an introduction to Go kit by showcasing a very basic todo application. You can find the source code of this tutorial on https://github.com/fulls1z3/go-kit-example.
And in this very first example focus is on its concepts/the big picture, don’t expect real-world examples from day one (storage/persistence, validations, containerising, etc.) but the direction is towards creating a series of posts incrementally covering these concepts as well — so stay tuned.
The todo application
As the name says, we’re going to work on a simple todo application where the client can create a new item by sending free text but can also remove it by sending the id. And finally, expose all items stored by the service.
We will achieve this by creating a microservice and expose the logic using a RESTful API.
Endpoints
- GET /items, returns a list of todo items stored by the service
- POST /items, creates a new todo item
- DELETE /items/:id, deletes an existing todo item
Getting started
You’ll need to install Docker and Go tools in your system first. Then, we start by creating a git repository and then init go modules.
$ mkdir go-kit-example
$ cd go-kit-example
$ git init
$ go mod init github.com/fulls1z3/go-kit-exampleApplication entrypoint
We’re not building a sophisticated application, rather a simple RESTful service. Let’s keep things simple — it’s OK to have our application entrypoint located at the root directory.
Let’s create there a new main.go file merely printing a “Hello world” message in the main function and nothing more.
package main
import "fmt"
func main() {
fmt.Println("Hello world")
}We will re-visit this again to instantiate things, so don’t expect anything special at this point — we don’t need it now.
I’d rather follow a bottoms-up approach that moves up across first the service, then the endpoint and transport layers, explain these tiers, what they do, etc. and eventually let the main function interact with these.
The service layer
In Go kit, the business logic is implemented by services. Services only contain the core business functions nothing more. They are totally agnostic about other concepts — such as exposing the logic over HTTP/gRPC or handling requests and responses as well as logging and instrumentation.
That’s what I love the most about Go kit, since it encourages a very clean architecture — letting your services focus on doing one thing and doing it well rather than try awkwardly doing 10 things at a time.
Another thing I love about Go kit is that it perfectly encourages the dependency-inversion principle so dependencies are not outwards-facing and any domain that needs the service can just depend on its contract — so called the service interface — rather than the actual implementation.
Enough talk, let’s create a service.go file and begin modelling our service having the following functions: add, remove literally adding/removing todo items into an imaginary store (for now) and getAll that returns all items in that store.
And therefore, we start with a “todo” domain entity, and an interface defining the service’s contract.
package main
type model struct {
ID int `json:"id"`
Name string `json:"name"`
}
type Service interface {
add(name string) error
remove(id int) error
getAll() ([]model, error)
}Next step is to build a factory function that constructs the service — actually returning its interface.
type svc struct{}
func NewService() Service {
return &svc{}
}And now it’s the best time to create a service_test.go file and write some unit tests before we implement the service. Sounds TDD’ish — I know :)
Will be a long one…
package main
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestAdd(t *testing.T) {
type args struct {
model *model
}
testCases := []struct {
name string
args args
err error
}{
{
name: "should add items",
args: args{
model: &model{
Name: "test item",
},
},
err: nil,
},
}
s := NewService()
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
err := s.add(tc.args.model.Name)
assert.Equal(t, tc.err, err)
})
}
}
func TestRemove(t *testing.T) {
type args struct {
id int
}
testCases := []struct {
name string
args args
err error
}{
{
name: "should remove items",
args: args{
id: 1,
},
err: nil,
},
}
s := NewService()
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
err := s.remove(tc.args.id)
assert.Equal(t, tc.err, err)
})
}
}
func TestGetAll(t *testing.T) {
testCases := []struct {
name string
expected []model
err error
}{
{
name: "should get all items",
expected: []model{},
err: nil,
},
}
s := NewService()
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
actual, err := s.getAll()
assert.Equal(t, tc.expected, actual)
assert.Equal(t, tc.err, err)
})
}
}The testing structure above may lead to yet another article. But if to briefly talk about it, I just used a pattern that tests each function against a list of cases/args, all by reusing a single service instance — thanks to pure functions.
You can also notice that test domain is the first consumer of this service in this project, just depends on the abstraction rather than implementation — which is not there yet.
At the moment we only have the service contract and its constructor. We also have now a set of tests that are failing. So we need to make them pass by writing the service functions. Let’s go back to service.go file, and add the implementation of service.
func (s *svc) add(name string) error {
return nil
}
func (s *svc) remove(id int) error {
return nil
}
func (s *svc) getAll() ([]model, error) {
return []model{}, nil
}For the sake of keeping it simple — and iterative — I hereby return empty results from all these three functions. We can indeed validate the input, etc. and eventually increase the complexity, but we will do it in later posts. I want to move fast across more important concepts for now.
The endpoint layer
Handling the incoming requests that comes from the transport (HTTP, gRPC, etc), calling the service functions and returning responses to the transport again by interacting with the service layer is done by endpoints,
Let’s go by example: the “todo” service has three functions: add, remove, and getAll. We want to hit these endpoints first — if applicable — with some payload, our starting point is handling incoming HTTP requests to these functions. So let’s create the endpoint.go file and define the request types.
package main
type addRequest struct {
Name string `json:"name"`
}
type removeRequest struct {
ID int `json:"id"`
}You may ask the request type for getAll function is missing out, but since that function accepts no arguments no payload is sent — so let’s skip it.
The next step is to define response types. getAll function is the only one exposing a concrete response whereas the add and remove functions just return an error when things go wrong. And things will eventually go wrong at some point. Nevertheless, let’s define response types for all these service functions.
type addResponse struct {
err error
}
type removeResponse struct {
err error
}
type getAllResponse struct {
payload []model
err error
}And finally, let’s put it altogether by wrapping these service functions with endpoint functions.
func makeAddEndpoint(s Service) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (interface{}, error) {
input := request.(addRequest)
err := s.add(input.Name)
return &addResponse{
err: err,
}, nil
}
}
func makeRemoveEndpoint(s Service) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (interface{}, error) {
input := request.(removeRequest)
err := s.remove(input.ID)
return &removeResponse{
err: err,
}, nil
}
}
func makeGetAllEndpoint(s Service) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (interface{}, error) {
res, err := s.getAll()
return &getAllResponse{
payload: res,
err: err,
}, nil
}
}These endpoint functions are exposing the service interfaces as the Go kit generic abstraction of “endpoint.Endpoint” declaring how the endpoint would behave and the transport can seamlessly integrate to the service.
The transport layer
The outermost layer in Go kit is the transport, where endpoints are bound to concrete transports (it natively supports HTTP, gRPC, Thrift, and net/rpc) — the layer decoding requests, calling the endpoints and encoding the response for the client.
In this example, we will demonstrate the HTTP transport to expose our services to outside world — by using the Chi router. Then let’s create transport.go and start mapping incoming HTTP requests to the request types we defined earlier alongside with the endpoints functions, as well as validating them and returning custom errors to be later handled as transport errors.
package main
import (
"context"
"encoding/json"
"errors"
"net/http"
"strconv"
"github.com/go-chi/chi/v5"
)func decodeAddRequest(_ context.Context, r *http.Request) (interface{}, error) {
var request addRequest
if err := json.NewDecoder(r.Body).Decode(&request); err != nil {
return nil, ErrBadRequest
}
return request, nil
}
func decodeRemoveRequest(_ context.Context, r *http.Request) (interface{}, error) {
id, err := strconv.Atoi(chi.URLParam(r, "ID"))
if err != nil {
return nil, ErrInvalidId
}
return removeRequest{
ID: id,
}, nil
}
func decodeGetAllRequest(_ context.Context, r *http.Request) (interface{}, error) {
return struct{}{}, nil
}var ErrBadRequest = errors.New("bad request")
var ErrInvalidId = errors.New("invalid id")
And then continue with mapping the responses into their JSON payloads.
func encodeAddResponse(_ context.Context, w http.ResponseWriter, response interface{}) error {
res := response.(*addResponse)
return json.NewEncoder(w).Encode(res.err)
}
func encodeRemoveResponse(_ context.Context, w http.ResponseWriter, response interface{}) error {
res := response.(*removeResponse)
return json.NewEncoder(w).Encode(res.err)
}
func encodeGetAllResponse(_ context.Context, w http.ResponseWriter, response interface{}) error {
res := response.(*getAllResponse)
if res.err != nil {
return json.NewEncoder(w).Encode(res.err)
}
return json.NewEncoder(w).Encode(res.payload)
}We said things would eventually go wrong. And below is how we do define a generic function to encode any kind of errors as transport errors, by mapping them into JSON responses as well.
func encodeError(_ context.Context, err error, w http.ResponseWriter) {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
switch err {
case ErrBadRequest:
w.WriteHeader(http.StatusBadRequest)
case ErrInvalidId:
w.WriteHeader(http.StatusNotFound)
default:
w.WriteHeader(http.StatusInternalServerError)
}
_ = json.NewEncoder(w).Encode(map[string]interface{}{
"error": err.Error(),
})
}Now we are able to handle request and responses as well as errors, the remaining step is to create the makeHandler function wrapping the endpoints to HTTP transport.
func makeHandler(s Service) http.Handler {
options := []httptransport.ServerOption{
httptransport.ServerErrorEncoder(encodeError),
}
addHandler := httptransport.NewServer(
makeAddEndpoint(s),
decodeAddRequest,
encodeAddResponse,
options...,
)
removeHandler := httptransport.NewServer(
makeRemoveEndpoint(s),
decodeRemoveRequest,
encodeRemoveResponse,
options...,
)
getAllHandler := httptransport.NewServer(
makeGetAllEndpoint(s),
decodeGetAllRequest,
encodeGetAllResponse,
options...,
)
r := chi.NewRouter()
r.Route("/items", func(r chi.Router) {
r.Get("/", getAllHandler.ServeHTTP)
r.Post("/add", addHandler.ServeHTTP)
r.Get("/remove/{ID}", removeHandler.ServeHTTP)
})
return r
}Application entrypoint re-revisited
Let’s wrap up the things: since the transport is registered in a Mux object, it’s time adapt our main function to run our HTTP server. Let’s re-revisit the main.go file, the main function.
package main
import (
"fmt"
"net/http"
"log"
"os"
"os/signal"
"syscall"
"time"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
)
var (
httpAddr = ":8080"
httpTimeout = 60 * time.Second
)
func main() {
var service Service
service = NewService()
handler := makeHandler(service)
r := chi.NewRouter()
r.Use(middleware.Recoverer)
r.Use(middleware.Timeout(httpTimeout))
r.Mount("/api/v1", handler)
errs := make(chan error)
go func() {
c := make(chan os.Signal, 1)
signal.Notify(c, syscall.SIGINT, syscall.SIGTERM)
errs <- fmt.Errorf("%s", <-c)
}()
go func() {
log.Printf("listening on %s", httpAddr)
errs <- http.ListenAndServe(httpAddr, r)
}()
log.Printf("exit", <-errs)
}Running the app
In a matter of minutes we got our service ready to handle the requests and produce meaningful responses. Let’s test the behavior by running it and seeing the service is listening on port 8080.
$ go run main.go service.go endpoint.go transport.go
# 2021/12/10 06:01:09 listening on :8080- returning a list of todo items stored by the service
$ curl --location --request GET 'localhost:8080/api/v1/items'
# []- creating a new todo item
$ curl --location --request POST 'localhost:8080/api/v1/items' \
$ --header 'Content-Type: application/json' \
$ --data-raw '{"name": "test item"}'
# null- deleting an existing todo item
$ curl --location --request DELETE 'localhost:8080/api/v1/items/1'
# nullWrapping it up
In focused on simplicity and being straight to point since when I begin this journey I suffered from the lack of a simplified approach. I hope I explained well why I chose to work using Go kit, how it is organised/works and revealed another myth showcasing this example app. And I hope the next people after me will be more lucky.
Cheers!
Next steps
- Focusing on the power of endpoint layer by showcasing the middleware pattern, candidates are authentication and logging.
- Adding persistence, more validations/error handling as well as integration testing.
- Containerising the todo microservice and deploying to kubernetes.
