Full-stack monorepo - Part I: Go services

Published in

Burak Tasci

7 min readJan 4, 2020

Techniques involved in creating web applications have continuously evolved, since web applications evolved from simple static websites into complex multi-layered applications and API first systems.

Over time, monolithic systems became too complex to deal with: they grew into huge platforms and many teams are drawn to decompose a monolith to an ecosystem of microservices by isolating features into separate repositories.

Although separation of concerns with this approach allowed teams to deliver value in parallel and independently of each other, dependencies between the projects slowed them down a lot: even a small change required a PR in multiple repositories, figuring out in which order to build changes and waiting for each other’s deployment.

The bottleneck around working across projects in a multi-repo workspace has been leading teams to adopt the concept of a monorepo, making it super easy to deal with cross-project dependencies as well as CI/CD configuration and allowing teams to implement features with atomic commits — since there’s one source of truth for everything while still having scalability, separation of concerns, code sharing and a lot more.

This series is an attempt to explain how to build a modern architecture using microservices — in different programming languages — and single-page applications (SPA), containerize them using Docker, add the necessary CI pipelines/deployment scripts and organize everything in a monorepo.

You can meanwhile fork and have a look at this repository containing the source code.

Getting started

You’ll need to install Docker, Go tools and Node.js runtime in your system first.

Then, we start by creating a git repository.

$ mkdir fullstack-monorepo
$ cd fullstack-monorepo
$ git init

Monorepo structure

Since we’re going to work polyglot, the separation takes place on the programming language at the first level. Below is the directory structure of our monorepo.

├── .circleci
├── golang
│   └── ...
├── nodejs
│   └── ...
├── scripts
│   ├── make
│   └── ...
├── Makefile
├── docker-compose.test.yml
├── docker-compose.yml
└── ...

We build Docker images for each service since each of them are separate applications and Docker Compose allows to group and run these services together.
We use the make utility to run tasks which build Docker images, start/stop containers and run test suites.
In this series, we stick to the generous free-plan of CircleCI to automate tests, builds and deployments.

Part I: Adding Go services

I’d like to begin with the first set of microservices in Go since it provides exciting features and has more or less a mature ecosystem nowdays, thus has been powering an extensive number of popular projects.

Init Go modules

First we create the golang directory since we want to locate our Go code there, and then init go modules.

$ mkdir golang
$ cd golang
$ go mod init github.com/{username}/{repo}

Where {username} is your Github username and {repo} is the repository name. In this example, we have it as below:

$ go mod init github.com/fulls1z3/fullstack-monorepo

Docker images for the services

We start by creating a base Dockerfile with the instructions below, in order to automate image creation for services.

# Start from golang base image
FROM golang:1.13-alpine as builder

# Set the current working directory inside the container
WORKDIR /build

# Copy go.mod, go.sum files and download deps
COPY go.mod go.sum ./
RUN go mod download

# Copy sources to the working directory
COPY . .

# Build the Go app
ARG project
RUN GOOS=linux CGO_ENABLED=0 GOARCH=amd64 go build -a -v -o server $project

# Start a new stage from busybox
FROM busybox:latest

WORKDIR /dist

# Copy the build artifacts from the previous stage
COPY --from=builder /build/server .

# Run the executable
CMD ["./server"]

And the following Dockerfile.test to execute tests in our libraries and services.

# Start from golang base image
FROM golang:1.13-alpine

# Set the current working directory inside the container
WORKDIR /test

# Copy go.mod, go.sum files and download deps
COPY go.mod go.sum ./
RUN go mod download

# Copy sources to the working directory
COPY . .

# Run the test suite
ARG project
RUN CGO_ENABLED=0 go test -v -cover $project \
    | sed ''/PASS/s//$(printf "\033[32mPASS\033[0m")/'' \
    | sed ''/FAIL/s//$(printf "\033[31mFAIL\033[0m")/'' \

Directory structure

I prefer such a lean structure as Peter Bourgon explained in his article.

The basic idea is to have two top-level directories, pkg and cmd. Underneath pkg, create directories for each of your libraries. Underneath cmd, create directories for each of your binaries.

At this point, we have the following directory structure for Go services.

/golang
├── cmd
│   └── ...
├── pkg
│   └── ...
├── Dockerfile
├── Dockerfile.test
├── go.mod
└── go.sum

Adding a library

We’re going to create a simple library holding a simple function to return some kind of Hello world.

Since the pkg directory contains all shared code and libraries (and must not
contain any service specific code), we’re going to work there and hereby create a hello directory.

$ cd pkg
$ mkdir hello
$ cd hello

And finally we create two files: hello.go and hello_test.go, with the content provided below.

hello.go

package hello

import "fmt"

func GetHello(s string) string {
   return fmt.Sprintf("Hello World from %s", s)
}

hello_test.go

package hello

import (
   "testing"
   "github.com/stretchr/testify/assert"
)

func TestGetHello(t *testing.T) {
   expected := "Hello World from TEST"
   actual := GetHello("TEST")

   assert.Equal(t, actual, expected)
}

Adding Go services

The cmd directory contains entry points — each folder representing the service name and holding the main package.

In this example, we will add two Go services called calypso and echo, so we start with creating directories for each of them.

$ cd ./golang/cmd
$ mkdir calypso && mkdir echo

And in each directory, we create main.go files with the content below.

package main

import (
   "fmt"
   "github.com/fulls1z3/fullstack-monorepo/pkg/hello"
   "log"
   "net/http"
   "os"
)

func main() {
   var PORT string

   if PORT = os.Getenv("PORT"); PORT == "" {
      log.Fatal("PORT not defined")
   }

   var APP_NAME string

   if APP_NAME = os.Getenv("APP_NAME"); APP_NAME == "" {
      log.Fatal("APP_NAME not defined")
   }

   http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
      msg := hello.GetHello(APP_NAME)
      fmt.Fprintf(w, "%s: %s\n", msg, r.URL.Path)
   })

   http.ListenAndServe(":" + PORT, nil)
}

I’m aware that both services are identical — performing the same task — and I can hear that many are now saying we’re repeating ourselves. It’s just to make things less complicated since the focus is on the architectural side.

Before we proceed with Docker Compose configuration, let’s confirm our directory structure.

/golang
├── cmd
│   ├── calypso
│   │   └── main.go
│   └── echo
│       └── main.go
├── pkg
│   └── hello
|       ├── hello.go
|       └── hello_test.go
├── Dockerfile
├── Dockerfile.test
├── go.mod
└── go.sum

Docker Compose configuration

We need two Docker containers since we have two Go services (calypso and echo). Therefore, we’ll use Docker Compose to orchestrate that group of containers.

At the repository root, we create docker-compose.yml file to define the services, so they can be run together inside of a single sandbox.

version: '3'
services:
  calypso:
    build:
      context: ./golang
      dockerfile: ./Dockerfile
      args:
        project: ./cmd/calypso/
    environment:
      - PORT=3001
      - APP_NAME=calypso
    ports:
      - 8001:3001
    restart: on-failure
    volumes:
      - calypso_vol:/usr/src/calypso/
    networks:
      - monorepo_net
  echo:
    build:
      context: ./golang
      dockerfile: ./Dockerfile
      args:
        project: ./cmd/echo/
    environment:
      - PORT=3001
      - APP_NAME=echo
    ports:
      - 8002:3001
    restart: on-failure
    volumes:
      - echo_vol:/usr/src/echo/
    networks:
      - monorepo_net

volumes:
  calypso_vol:
  echo_vol:

networks:
  monorepo_net:
    driver: bridge

And then the docker-compose.test.yml to define the services that are required to run the tests.

version: '3'
services:
  pkg_test:
    build:
      context: ./golang
      dockerfile: ./Dockerfile.test
      args:
        project: ./pkg/...
    volumes:
      - testing_vol:/usr/src/pkg/
    networks:
      - monorepo_net
  calypso_test:
    build:
      context: ./golang
      dockerfile: ./Dockerfile.test
      args:
        project: ./cmd/calypso/...
    depends_on:
      - pkg_test
    volumes:
      - testing_vol:/usr/src/calypso/
    networks:
      - monorepo_net
  echo_test:
    build:
      context: ./golang
      dockerfile: ./Dockerfile.test
      args:
        project: ./cmd/echo/...
    depends_on:
      - pkg_test
    volumes:
      - testing_vol:/usr/src/echo/
    networks:
      - monorepo_net

volumes:
  testing_vol:

networks:
  monorepo_net:
    driver: bridge

Build and test scripts

Even though it’s possible to build Docker images, and manage containers using native Docker commands, we’ll use the make utility — a very simple and easy-to-understand mechanism to call the shell scripts.

Still at the repository root, we create base Makefile file, which includes the build, make and test scripts.

SOURCE := $(shell git rev-parse --show-toplevel)

include $(SOURCE)/scripts/make/build.mk
include $(SOURCE)/scripts/make/dev.mk
include $(SOURCE)/scripts/make/test.mk

Since make scripts are located in scripts/make directory, we create build.mk, dev.mk and test.mk, with the content provided below.

$ mkdir scripts
$ cd scripts
$ mkdir make
$ cd make

build.mk

.PHONY: build

build: ## Build docker image
 docker-compose build

dev.mk

.PHONY: status logs start stop clean

status: ## Get status of containers
 docker-compose ps

logs: ## Get logs of containers
 docker-compose logs -f

start: ## Start docker containers
 docker-compose up -d

stop: ## Stop docker containers
 docker-compose stop

clean:stop ## Stop docker containers, clean data and workspace
 docker-compose down -v --remove-orphans

test.mk

.PHONY: test

test: ## Run tests
 docker-compose -f docker-compose.test.yml up --build --abort-on-container-exit
 docker-compose -f docker-compose.test.yml down --volumes

Running the app

As you could follow, you might run the tests by using make test, and run the containers by using make start.

Then, just visit http://localhost:8001/ and http://localhost:8002/ on your browser (or send GET requests using cURL) and you should be able to see Hello World from both APIs.

Wrapping it up

I tried to keep things quite simple in this article, but at the time we were challenging with this setup there wasn’t much sample and documentation — while most of the articles were focused on Node.js and were not intended to be polyglot. We’ve gone through excruciating pain to evolve this architecture. But I think it’s all worth now.

Meanwhile, keep in mind that this fullstack-monorepo project is currently very much WIP and still more is underway. I created this tag to keep the point where we are with everything explained in this article.

Next steps

Implementing services in Node.js and consuming the API with SPA

Burak Tasci (fulls1z3)
https://www.linkedin.com/in/buraktasci
http://stackoverflow.com/users/7047325/burak-tasci
https://github.com/fulls1z3