Go tools & GitLab — how to do Continuous Integration like a boss

Julien Andrieux
Nov 6, 2017 · 10 min read

At Pantomath, we use GitLab for all our development work. The purpose of this paper is not to present GitLab and all its features, but to introduce how we use these tools to ease our lives.

So what is it all about? To automate everything that is related to your development project, and let you focus on your code. We’ll cover the lint, unit tests, data race, memory sanitizer, code coverage, and build.

All the code shown in this post is available at https://gitlab.com/pantomath-io/demo-tools. So feel free to get the repository, and use the tags to navigate in it. The repository should be placed in the src folder of your $GOPATH:

$ go get -v -d gitlab.com/pantomath-io/demo-tools
$ cd $GOPATH/src/gitlab.com/pantomath-io/demo-tools

Go tools

Package list

$ go list ./...

Note that we want to avoid applying our tools on external resources, and restrict it to our code. So we need to get rid of the vendor directories:

$ go list ./... | grep -v /vendor/

Lint

This linter is not part of go per se, so you need to grab it and install it by hand (see official doc).

The usage is fairly simple: you just run it on the packages of your code (you can also point the .go files):

$ golint -set_exit_status $(go list ./... | grep -v /vendor/)

Note the -set_exit_status option. By default, golint only prints the style issues, and returns (with a 0 return code), so the CI never considers something went wrong. If you specify the -set_exit_status, the return code from golint will be different from 0 if any style issue is encountered.

Unit test

$ go test -short $(go list ./... | grep -v /vendor/)

Data race

$ go test -race -short $(go list ./... | grep -v /vendor/)

Memory sanitizer

$ go test -msan -short $(go list ./... | grep -v /vendor/)

Code coverage

To calculate the code coverage ratio, we need to run the following script:

$ PKG_LIST=$(go list ./... | grep -v /vendor/)
$ for package in ${PKG_LIST}; do
go test -covermode=count -coverprofile "cover/${package##*/}.cov" "$package" ;
done
$ tail -q -n +2 cover/*.cov >> cover/coverage.cov
$ go tool cover -func=cover/coverage.cov

If we want to get the coverage report in HTML format, we need to add the following command:

$ go tool cover -html=cover/coverage.cov -o coverage.html

Build

$ go build -i -v gitlab.com/pantomath-io/demo-tools

Makefile

Image for post
Image for post
Photo by Matt Artz on Unsplash

Now we have all the tools that we may use in the context of Continuous Integration, we can wrap them all in a Makefile, and have a consistent way to call them.

The purpose of this doc is not to present make, but you can refer to official documentation to learn more about it.

PROJECT_NAME := "demo-tools"
PKG := "gitlab.com/pantomath-io/$(PROJECT_NAME)"
PKG_LIST := $(shell go list ${PKG}/... | grep -v /vendor/)
GO_FILES := $(shell find . -name '*.go' | grep -v /vendor/ | grep -v _test.go)
.PHONY: all dep build clean test coverage coverhtml lintall: buildlint: ## Lint the files
@golint -set_exit_status ${PKG_LIST}
test: ## Run unittests
@go test -short ${PKG_LIST}
race: dep ## Run data race detector
@go test -race -short ${PKG_LIST}
msan: dep ## Run memory sanitizer
@go test -msan -short ${PKG_LIST}
coverage: ## Generate global code coverage report
./tools/coverage.sh;
coverhtml: ## Generate global code coverage report in HTML
./tools/coverage.sh html;
dep: ## Get the dependencies
@go get -v -d ./...
build: dep ## Build the binary file
@go build -i -v $(PKG)
clean: ## Remove previous build
@rm -f $(PROJECT_NAME)
help: ## Display this help screen
@grep -h -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'

What do we have, now? One target for any tool previously presented, and 3 more targets for:

  • installation of dependencies (dep);
  • housekeeping of the project (clean);
  • some nice and shiny help (help).

Note that we also had to create a script for the code coverage work. This is because implementing loops over files in a Makefile is a pain. So the work is done in a bash script, and the Makefile only triggers this script.

You can try the Makefile with the following commands:

$ make help
$ make lint
$ make coverage

Continuous Integration

Image for post
Image for post
Photo by Max Panamá on Unsplash

Now the tools are in place, and we can run various tests on our code, we’d like to automate these, on your repository. Luckily, GitLab offers CI pipelines just for this. And the setup for this is pretty straightforward: all you create is a .gitlab-ci.yml file at the root of the repository.

The full documentation on this Yaml file presents all the options, but you can start with this .gitlab-ci.yml:

image: golang:1.9cache:
paths:
- /apt-cache
- /go/src/github.com
- /go/src/golang.org
- /go/src/google.golang.org
- /go/src/gopkg.in
stages:
- test
- build
before_script:
- mkdir -p /go/src/gitlab.com/pantomath-io /go/src/_/builds
- cp -r $CI_PROJECT_DIR /go/src/gitlab.com/pantomath-io/pantomath
- ln -s /go/src/gitlab.com/pantomath-io /go/src/_/builds/pantomath-io
- make dep
unit_tests:
stage: test
script:
- make test
race_detector:
stage: test
script:
- make race
memory_sanitizer:
stage: test
script:
- make msan
code_coverage:
stage: test
script:
- make coverage
code_coverage_report:
stage: test
script:
- make coverhtml
only:
- master
lint_code:
stage: test
script:
- make lint
build:
stage: build
script:
- make

If you break down the file, here are some explanations on its content:

  • The first thing is to choose what Docker image will be used to run the CI. Head to the Docker Hub to choose the right image for your project.
  • Then, you specify some folders of this image to be cached. The goal here is to avoid downloading the same content several times. Once a job is completed, the listed paths will be archived, and next job will use the same archive.
  • You define the different stages that will group your jobs. In our case, we have two stages (to be processed in that order): test and build. We could have other stages, such as deploy.
  • The before_script section defines the commands to run in the Docker container right before the job is actually done. In our context, the commands just copy or link the repository deployed in the $GOPATH, and install dependencies.
  • Then come the actual jobs, using the Makefile targets. Note the special case for code_coverage_report where execution is restricted to the master branch (we don’t want to update the code coverage report from feature branches for instance).

As we commit/push the .gitlab-ci.yml file in the repository, the CI is automatically triggered. And the pipeline fails. How come?

The lint_code job fails because it can’t find the golint binary:

$ make lint
make: golint: Command not found
Makefile:11: recipe for target 'lint' failed
make: *** [lint] Error 127

So, update your Makefile to install golint as part of the dep target.

The memory_sanitizer job fails because gcc complains:

$ make msan
# runtime/cgo
gcc: error: unrecognized argument to -fsanitize= option: 'memory'
Makefile:20: recipe for target 'msan' failed
make: *** [msan] Error 2

But remember we need to use Clang/LLVM >=3.8.0 to enjoy the -msan option in go test command.

We have 2 options here:

  • either we setup Clang in the job (using before_script);
  • or we use a Docker image with Clang installed by default.

The first option is nice, but that implies to have this setup done for every single job. This is going to be so long, we should do it once and for all. So we prefer the second option, which is a good way to play with GitLab Registry.

git tag: use-own-docker

We need to create a Dockerfile for the container (as usual: read the official documentation for more options about it):

# Base image: https://hub.docker.com/_/golang/
FROM golang:1.9
MAINTAINER Julien Andrieux <julien@pantomath.io>
# Install golint
ENV GOPATH /go
ENV PATH ${GOPATH}/bin:$PATH
RUN go get -u github.com/golang/lint/golint
# Add apt key for LLVM repository
RUN wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add -
# Add LLVM apt repository
RUN echo "deb http://apt.llvm.org/stretch/ llvm-toolchain-stretch-5.0 main" | tee -a /etc/apt/sources.list
# Install clang from LLVM repository
RUN apt-get update && apt-get install -y --no-install-recommends \
clang-5.0 \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
# Set Clang as default CC
ENV set_clang /etc/profile.d/set-clang-cc.sh
RUN echo "export CC=clang-5.0" | tee -a ${set_clang} && chmod a+x ${set_clang}

The container built out of this Dockerfile will be based on golang:1.9 image (the one referenced in the .gitlab-ci.yml file).

While we’re at it, we install golint in the container, so we have it available. Then we follow official way of installing Clang 5.0 from LLVM repository.

Now we have the Dockerfile in place, we need to build the container image and make it available for GitLab:

$ docker login registry.gitlab.com
$ docker build -t registry.gitlab.com/pantomath-io/demo-tools .
$ docker push registry.gitlab.com/pantomath-io/demo-tools

The first command connects you to the GitLab Registry. Then you build the container image described in the Dockerfile. And finally, you push it to the GitLab Registry.

Take a look at the Registry for your repository, you’ll see your image, ready to be used. And to have the CI using your image, you just need to update the .gitlab-ci.yml file:

image: golang:1.9

becomes

image: registry.gitlab.com/pantomath-io/demo-tools:latest

One last detail: you need to tell the CI to use the proper compiler (i.e. the CC environment variable), so we add the variable initialization in the .gitlab-ci.yml file:

export CC=clang-5.0

Once the modification are done, next commit will trigger the pipeline, which now works:

https://gitlab.com/pantomath-io/demo-tools/pipelines/13497136

Badges

Image for post
Image for post
Photo by Jakob Owens on Unsplash

Now the tools are in place, every commit will launch a test suite, and you probably want to show it, and that’s legitimate :) The best way to do so is to use badges, and the best place for it is the README file.

Edit it and add the 4 following badges:

  • Build Status: the status of the last pipeline on the master branch:
[![Build Status](https://gitlab.com/pantomath-io/demo-tools/badges/master/build.svg)](https://gitlab.com/pantomath-io/demo-tools/commits/master)
  • Coverage Report: the percentage of code covered by tests
[![Coverage Report](https://gitlab.com/pantomath-io/demo-tools/badges/master/coverage.svg)](https://gitlab.com/pantomath-io/demo-tools/commits/master)
  • Go Report Card:
[![Go Report Card](https://goreportcard.com/badge/gitlab.com/pantomath-io/demo-tools)](https://goreportcard.com/report/gitlab.com/pantomath-io/demo-tools)
  • License:
[![License MIT](https://img.shields.io/badge/License-MIT-brightgreen.svg)](https://img.shields.io/badge/License-MIT-brightgreen.svg)

The coverage report needs a special configuration. You need to tell GitLab how to get that information, considering that there is a job in the CI that displays it when it runs.
There is a configuration to provide GitLab with a regexp, used in any job’ output. If the regexp matches, GitLab consider the match to be the code coverage result.

So head to Settings > CI/CD in your repository, scroll down to the Test coverage parsing setting in the General pipelines settings section, and use the following regexp:

total:\s+\(statements\)\s+(\d+.\d+\%)

You’re all set! Head to the overview of your repository, and look at your README:

Image for post
Image for post

Conclusion

Many thanks to Charles Francoise who co-wrote this paper and https://gitlab.com/pantomath-io/demo-tools.

We are currently working on Pantomath. Pantomath is a modern, open-source monitoring solution, built for performance, that bridges the gaps across all levels of your company. The well-being of your infrastructure is everyone’s business. Keep up with the project

Pantomath

Pantomath is a cutting edge monitoring solution

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store