Full-stack monorepo - Part I: Go services

Burak Tasci
Jan 4, 2020 · 7 min read

Techniques involved in creating web applications have continuously evolved, since web applications evolved from simple static websites into complex multi-layered applications and API first systems.

Over time, monolithic systems became too complex to deal with: they grew into huge platforms and many teams are drawn to decompose a monolith to an ecosystem of microservices by isolating features into separate repositories.

From monolithic systems to monorepos

Although separation of concerns with this approach allowed teams to deliver value in parallel and independently of each other, dependencies between the projects slowed them down a lot: even a small change required a PR in multiple repositories, figuring out in which order to build changes and waiting for each other’s deployment.

The bottleneck around working across projects in a multi-repo workspace has been leading teams to adopt the concept of a monorepo, making it super easy to deal with cross-project dependencies as well as CI/CD configuration and allowing teams to implement features with atomic commits — since there’s one source of truth for everything while still having scalability, separation of concerns, code sharing and a lot more.

This series is an attempt to explain how to build a modern architecture using microservices — in different programming languages — and single-page applications (SPA), containerize them using Docker, add the necessary CI pipelines/deployment scripts and organize everything in a monorepo.

You can meanwhile fork and have a look at this repository containing the source code.

Getting started

You’ll need to install Docker, Go tools and Node.js runtime in your system first.

Then, we start by creating a git repository.

Monorepo structure

Since we’re going to work polyglot, the separation takes place on the programming language at the first level. Below is the directory structure of our monorepo.

  • We build Docker images for each service since each of them are separate applications and Docker Compose allows to group and run these services together.
  • We use the make utility to run tasks which build Docker images, start/stop containers and run test suites.
  • In this series, we stick to the generous free-plan of CircleCI to automate tests, builds and deployments.

Part I: Adding Go services

I’d like to begin with the first set of microservices in Go since it provides exciting features and has more or less a mature ecosystem nowdays, thus has been powering an extensive number of popular projects.

Init Go modules

First we create the golang directory since we want to locate our Go code there, and then init go modules.

Where {username} is your Github username and {repo} is the repository name. In this example, we have it as below:

Docker images for the services

We start by creating a base Dockerfile with the instructions below, in order to automate image creation for services.

And the following Dockerfile.test to execute tests in our libraries and services.

Directory structure

I prefer such a lean structure as Peter Bourgon explained in his article.

The basic idea is to have two top-level directories, pkg and cmd. Underneath pkg, create directories for each of your libraries. Underneath cmd, create directories for each of your binaries.

At this point, we have the following directory structure for Go services.

Adding a library

We’re going to create a simple library holding a simple function to return some kind of Hello world.

Since the pkg directory contains all shared code and libraries (and must not
contain any service specific code), we’re going to work there and hereby create a hello directory.

And finally we create two files: hello.go and hello_test.go, with the content provided below.

  • hello.go
  • hello_test.go

Adding Go services

The cmd directory contains entry points — each folder representing the service name and holding the main package.

In this example, we will add two Go services called calypso and echo, so we start with creating directories for each of them.

And in each directory, we create main.go files with the content below.

I’m aware that both services are identical — performing the same task — and I can hear that many are now saying we’re repeating ourselves. It’s just to make things less complicated since the focus is on the architectural side.

Before we proceed with Docker Compose configuration, let’s confirm our directory structure.

Docker Compose configuration

We need two Docker containers since we have two Go services (calypso and echo). Therefore, we’ll use Docker Compose to orchestrate that group of containers.

At the repository root, we create docker-compose.yml file to define the services, so they can be run together inside of a single sandbox.

And then the docker-compose.test.yml to define the services that are required to run the tests.

Build and test scripts

Even though it’s possible to build Docker images, and manage containers using native Docker commands, we’ll use the make utility — a very simple and easy-to-understand mechanism to call the shell scripts.

Still at the repository root, we create base Makefile file, which includes the build, make and test scripts.

Since make scripts are located in scripts/make directory, we create build.mk, dev.mk and test.mk, with the content provided below.

  • build.mk
  • dev.mk
  • test.mk

Running the app

As you could follow, you might run the tests by using make test, and run the containers by using make start.

Then, just visit http://localhost:8001/ and http://localhost:8002/ on your browser (or send GET requests using cURL) and you should be able to see Hello World from both APIs.

Wrapping it up

I tried to keep things quite simple in this article, but at the time we were challenging with this setup there wasn’t much sample and documentation — while most of the articles were focused on Node.js and were not intended to be polyglot. We’ve gone through excruciating pain to evolve this architecture. But I think it’s all worth now.

Meanwhile, keep in mind that this fullstack-monorepo project is currently very much WIP and still more is underway. I created this tag to keep the point where we are with everything explained in this article.

Next steps

Burak Tasci (fulls1z3)

In depth articles about software technologies