Test Coverage of Go Services during Integration Tests
In Golang, getting code coverage with
go test is easy. But it's still rather hard for integration tests.
Here I want to introduce the method we used at Manabie to collect and measure code coverage on our servers from integration tests.
About our integration tests
At Manabie we use Kubernetes for container orchestration. To perform integration tests, we deploy our services, and then run a test container with a go program with a whole lot of integration tests.
On dev environments, we start up minikube, deploy the services, and then run the test container as well. In CI, we start a vcluster instead of a minikube.
Collecting coverage in this setting is a little more complicated, but doable. The main idea is to compile the services with
go test -cover instead of
go build, and then get the services to exit in a timely manner after testing is finished.
What’s Coming Up
- compile services with
go test -c,
- add an http killswitch
- run services & run tests
- stop the services by calling the http killswitch endpoint
- collect coverage reports from containers
- merge them with gocovmerge.
- get a code coverage percent. Conclusion & Example code
Part I: Compiling services with go test instrumentation
We want to run our service with
go test so we can use it's
cover flags to enable code coverage output.
To do this, we need to first create a test function that works like our
func main(), so that we can use
go test to run our server.
so that we can write a test like this that starts the server:
For the coverage to be output,
TestRun function needs to finish. We can't just kill the process, or go test will not output a coverage profile.
Because the service does not know when the tests are complete, we have to set up a mechanism to stop it remotely. We can set up a simple HTTP server. When it gets a request, it will gracefully terminate our service. Later, we can call it with curl.
Our Kill HTTP server calls a context’s
CancelFunc when it receives a request.
Our TestRun uses the same context to run the service.
To test out our killswitch locally, we can compile the service with:
go test -c ./ -cover -covermode=count -coverpkg=./... to create a
.test binary, and run it with:
In another terminal, we run some tests and then kill it with the HTTP signal.
Finally, a file named
my-service.out should be created with coverage information from our server.
go test -c to build test binaries is a nice little trick I learned about recently. It plays nicely with containers too.
Part 2: Running in our Kubernetes Cluster
Now we can add them to our Dockerfile. Also, we’re going to add a little wrapper script to restart them endlessly. If we let the entrypoint process end, our container and coverage files will be deleted after the killswitch is called, which is not what we want.
We need a Dockerfile with the
curl, and the restart script.
Next, build the new docker image and deploy it in your kubernetes cluster, possibly with helm. Make sure you don’t so override the Dockerfile’s entrypoint. However, you can pass args as usual into the deployment container, which will pass into the server.test command.
Next, run your tests. In our case, we built an integration testing program named
gandalf that we run in our cluster:
Coverage will be recorded by the server.test.
Part 3: Collecting Coverage
Now to generate, download and merge our coverage.
To output the coverage we just have to send an HTTP request to our killserver on port 19999.
Then copy the file to our local filesystem:
Do this for each service being tested, and then merge the coverage profiles together.
Note: We use a
.cov extention on the
merged.cov to make it easy to use
cover/*.out in scripting.
Inspect the contents of your new
merged.cov file and try not to get drunk with power.
Now our build pipeline prints out our integration test coverage percent collected from all our go services, and combined into a final figure.
For us at Manabie, we output this percent in our CI logs, and we set a rule that prevents pull requests from being mergeable if they decrease this percent.
Originally published at https://blog.manabie.io on December 27, 2021.