Go Starter

How fast can you build a (micro-)service? It always takes time to capture requirements, write business logic, connect all the pieces and get a production-ready application to the public. But there is this first step which normally takes a bit of routine work to just get started, roll-out that first bit into production environment so you can start building upon. This overhead of the first release is even more impactful when you build microservices, deploy new ones and re-write existing ones regularly.

As you build more and more services at some point you start to repeat yourself, copy-pasting project layout, server bootstrapping code, infrastructure specification files, etc. And then, most annoying thing, update all artifact names, references, import paths, etc. Furthermore, different team members may do things slightly different and you will soon start to see misalignment, naming inconsistency and other nasty things, which may look harmless, but will eventually make project maintenance complicated.

So, once again, asking myself a question, how do I install dependencies in this project, was it dep ensure? make dep? Ah, wait make deps! I figured we should get some sort of template. In a bare minimum, it should help to bootstrap a project, a new micro-service with a standard Makefile, an HTTP server, some default middleware, and endpoints and other things every project should have. Even better, if we could get a repository, CI/CD pipeline and deploy this new project onto our integration environment right away.

Going further, it would be great if we could even update repositories after they have been created, have a tool that would allow patching projects as we change template. For example, we decide make deps is a bad choice to install dependencies? Let’s just run one command which would automatically patch all Makefiles, create pull requests and get projects up-to-date. Oh, that would be great! But to make it work you need all projects to follow the same template, otherwise, you are just doomed going one by one and making changes manually.

After checking some existing project bootstrapping tools (boilr, yarn and some others) I got a bit frustrated. Some of them, do not really provide any tools to build a template. I mean, for example, with GitHub templates you can copy repository, but then what? If you can not replace placeholders, is it really a template? Others provide a rather limited way to extend what they do. Boilr came close combining ease of creating and distributing templates and simple interface. Although it requires to checkout templates locally and there was still no way to get CI/CD pipeline running.

I still was missing some flexibility in how a new project is created. It’s nice when you have features A and B, but it’s better if you have an extensible platform where you can build A, B and C.

Meet go-starter

I started with a simple bash script, which was cloning a repository and replacing placeholders. It worked pretty well and quickly grew into a standalone go command-line application.

The idea is very simple: clone repository and run batch of ordinary command-line utilities to do whatever post-processing you need.

In essence, here is what go-starter does:

• Clone given repository (a template) from a given URL
• Ask questions from .starter.yml to the user
• Execute tasks — commands listed in .starter.yml.

Based on this simple design we can create additional tools that will serve as tasks. For example, go-starter comes with go-starter-replace utility, it traverses files in the current folder and replaces <placeholder>s in the folder and file names as well as in the content of files. Don’t like <placeholder> format for placeholders? Not a problem, just create your version and use a different format.

Configuration file (.starter.yml) is stored along with the template. It defines two sections: questions and tasks.

Questions are prompted to the user after cloning repository. They provide input for tasks. For example, questions may ask user to enter: application name, which environment it should be deployed by default, who will be code owners, etc. Answers to these questions will be available to tasks via environment variables.

What’s in the box

Go-starter by itself does not do much. But, it’s shipped with additional binaries which can be used as tasks in the .starter.yml. For the moment, there are 3 extra binaries which take care of replacing placeholders, creating GitHub repository and activating this repository in Drone CI. With time more integrations can be added, for example for Bitbucket and Jenkins.

go-starter-replace

This task helps to replace placeholders in the template with project-specific values. It traverses file system starting with the current folder and replaces placeholders in folder and file names as well as in the content of the files.

go-starter-github

This task helps to create local and remote git repository. It creates GitHub repository using API, initializes git repository locally, creates initial commit and runs git push.

go-starter-drone

This task helps you to start first build of your application. It connects to Drone CI server, activates project build pipeline and creates an empty commit which triggers first build.

Starter in action

In case you want to try commands shown below, make sure you have go-starter installed. Follow instructions from README file to install go-starter.

Hello world

Let’s start a simple go project. We will use hello-world-starter template. It creates a simple HTTP server and GitHub repository for it.

Let’s run go-starter starter-template/hello-world-starter awesome-project.

That’s it.

We can enter awesome-project directory and start hello-world service locally:

cd ./awesome-project
go run ./cmd/hello-world

We can also see GitHub repository for the project skolodyazhnyy/hello-world.

Now, you can start implementing some additional logic, commit and push.

This is already something, but we are missing pipeline, so let’s try to integrate with CI/CD system and build an artifact for our application.

Build pipeline

We are going to use a different template which has additional files to configure CI pipeline.

Let’s try go-starter starter-template/pipeline-starter awesome-cicd.

Similar to the previous template, we got awesome-cicd folder with local copy of the project, GitHub repository, as well as go-starter, has triggered a build in drone.

Go-starter has activated pipeline in Drone CI and made an empty commit to trigger first build. This build has executed tests, linter and, most importantly, created a docker image. We can try to pull this image locally and run the application:

docker run --rm -it skolodyazhnyy/awesome-cicd:0.0.0-a8903e5

Our next step will be adding some deployment steps to our pipeline to get our application to the integration environment.

Next steps

Next, I want to extend our pipeline to actually deploy service to Kubernetes cluster. But since it will be rather a long story, I want to publish it separately.

In short, we are using Helm charts and GitOps approach to deploy applications to our Kubernetes cluster.

Drone pipeline creates Helm Chart and uploads it to Artifactory. Then, an automated script creates a pull request to GitOps repository. Then, it gets reviewed and merged. Finally, flux picks up changes and deploys new version of the chart to an actual environment.

In the pipeline from the previous section, we are missing two bits: we need a step to create helm chart and step to create a pull request to GitOps repository. These are simple Python scripts, which run in docker containers as drone steps.

But, more on this in the next story.