Building a web app: Golang, GitLab & Docker
Part 1: Environment: CI & static analysis
Quite a big part of modern software development is knowing what tools to use and how to glue them together. The services and the apps we usually build don’t contain many novel techical problems. We just need to choose the right tools for already solved problems and to work with them.
In this blog series I will try to overview the common problems of a web app development and tools to solve them. It is obviously useful for the project at the very beginning. If your project is already up and running, the series might help to uninvent a few wheels too.
We will use git and GitLab for source control (gitlab.com). It could be installed locally at it has a CI built in.
If you plan to use GitHub and TravisCI, most of the steps still apply anyway, so keep reading! ;)
Let’s begin!
We begin with setting up a CI runner with static analysis tools. Why?
It is much easier to protect the codebase from inflow of an obviously wrong code, than to fix all the linter issues afterwards. Linters can slow down the development for a few days. However they prevent potential bugs on a very early phase, it saves a lot of debugging time.
Static analyzers can also find security related issues. This feature is very useful, if you don’t have a person who can pen-test your app.
Also, static analyzers can be set up at any phase of the project, even before any code is written.
For Golang, we will use 4 static analysis tools that complement each other.
gofmt & goimports (golang.org/x/tools/cmd/goimports)
These tools help with the file formatting and correct imports. I strongly recommend to hook running goimports when you save the file, but having an additional check when the merge request is submitted is very useful.
Golint (github.com/golang/lint)
Golint will catch all the style errors that aren’t caught with goimports or go fmt, such as variable naming, comments, etc. It finds more issues like ignored errors and many more.
goconst (github.com/jgautheron/goconst)
goconst tries to save you from hardcoding the strings over and over again. It finds the repetitions and asks you to extract it to constants.
GoASTScanner (github.com/GoASTScanner/gas)
This one is a very powerful and flexible scanner. Among other things, it reports security issues. It is very important to catch these issues on a very early phase.
Huh, that’s a lot of linters!
Luckily, there is one tool run all of these tools together. It is called gometalinter (github.com/alecthomas/gometalinter). We will use it in our article.
Step 1: Create a docker image with linters
To abstract away the nuances of the platform, where our CI runner is started, we will enclose all our environment into a Docker container. You can read up about Docker in The Docker Book.
Create the file called Dockerfile (no extension) in the root directory of your code.
# Inherit the container from a pre-built `golang` image.
FROM golang:latest
# Install the static analysis tool(s)…
RUN go get -u -v github.com/GoASTScanner/gas
RUN go get -u -v golang.org/x/tools/cmd/goimports
RUN go get -u -v github.com/golang/lint/golint
# ...and one tool to rule them all!
RUN go get -u -v github.com/alecthomas/gometalinter
WORKDIR /go/src/<your-gitlab-host>/<path-to-the-repo>
COPY . .This creates a directory /go/src/<your-gitlab-host>/<path-to-the-repo> inside the container, makes it current and copies all the current folder contents there. So we will end up with an image with the golang environment, our linters and our code.
Step 2: Make a shell script to build and run tests in the container
Now we will make a shell script which will run the linters (run-static-analysis.sh). Put it in the same directory where the Dockerfile is.
#/bin/sh
docker build -t my-project-linters .docker run my-project-linters gometalinter \
--exclude=".+should have comment or be unexported \(golint\)" \
./... \
--disable-all \
--enable="gas" \
--enable="golint" \
--enable="gofmt" \
--enable="vet" \
--enable="goimports"
Let’s dive into the contents a little bit.
docker build -t my-project-linters — build a Docker image from the Dockerfile in current directory and tags it as my-project-linters (needed for the next step).
docker run my-project-linters <command> — runs <command> inside a container tagged as my-project-linters.
gometalinter --exclude=".+should have comment or be unexported \(golint\)" ./... --disable-all -enable=… - this excludes not very useful warning from golint that every exported method should have a comment. It also specifies that only gofmt, golint, gas, go vet and goimports should be launched. Otherwise there will be an error that the certain linter is not installed.
Try running it locally:chmod a+x run-static-analysis.sh
./run-static-analysis.sh
You should see the output like this:
Sending build context to Docker daemon 7.378MB
Step 1/7 : FROM golang:latest
latest: Pulling from library/golang
Digest: sha256:b8bf21db6cf238db5f79cc3b2d773d0982367876d03dc5c8b8da5c336e85e612
Status: Downloaded newer image for golang:latest
---> 6d9bf2aec386
Step 2/7 : RUN go get -u -v github.com/GoASTScanner/gas
---> Running in c336ef1584e1
---> 8bd8ee562081
Removing intermediate container c336ef1584e1
Step 3/7 : RUN go get -u -v golang.org/x/tools/cmd/goimports
---> Running in 884c9ee8f6b0
---> 84476743f1ff
Removing intermediate container 884c9ee8f6b0
Step 4/7 : RUN go get -u -v github.com/golang/lint/golint
---> Running in 9fb39dc6568e
---> d2d81b8cffec
Removing intermediate container 9fb39dc6568e
Step 5/7 : RUN go get -u -v github.com/alecthomas/gometalinter
---> Running in 140a331c5c94
---> 8becaefcafcc
Removing intermediate container 140a331c5c94
Step 6/7 : WORKDIR /go/src/github.com/mandrigin/kbk2/backend
---> 6102093267da
Removing intermediate container 2f47d3b6cdaf
Step 7/7 : COPY . .
---> 24558cb5a437
Removing intermediate container e62e68a9f1cd
Successfully built 24558cb5a437
Successfully tagged my-project-linters:latest…and the errors actually found in the code you are checking:
common/gdrive.go:23::error: no formatting directive in Printf call (vet)
main.go:32:1:warning: comment on exported type Session should be of the form "Session ..." (with optional leading article) (golint)
api/comments.go:12:2:warning: struct field Id should be ID (golint)
api/comments.go:20:2:warning: don't use ALL_CAPS in Go names; use CamelCase (golint)
api/comments.go:20:2:warning: exported const COMMENT_TYPE_POSITIVE should have comment (or a comment on this block) or be unexported (golint)Now, we are ready to set it up for the CI environment.
Step 3: Run static analysis on every Merge Request
We need to install and register a CI runner. The instructions can be found in the GitLab documentation.
Keep in mind that the runner executor (step 8) should be set up to docker and share the docker socket (see more here). We need share the socket because we will need to start a new container (aka our my-project-linters) from another container (a CI runner container). Sounds complicated, but it is very simple to do. Just run this code on the CI machine to register the runner:
sudo gitlab-ci-multi-runner register -n \
--url https://gitlab.com/ \
--registration-token REGISTRATION_TOKEN \
--executor docker \
--description "My Docker Runner" \
--docker-image "docker:latest" \
--docker-volumes /var/run/docker.sock:/var/run/docker.sockMore information about why we need to use the socket sharing is discussed in this article.
Now go back to your repository and create a file named .gitlab-ci.yml at the root directory of the repo.
image: docker:latest
static_analysis:
stage: test
script:
- ./run-static-analysis.shNow commit your branch and make a merge request. You should see that it got a nice blue progress indicator.
Step 4: Don’t allow to merge requests that fail static analysis
Go to the GitLab settings and check the checkbox titled “Only allow merge requests to be merged if the pipeline succeeds”.
Now you are set up.
In the future posts, I will show how to add unit and functional tests to our CI environment and solve some typical security-related issues.
Stay tuned and thank you for reading this!
