Golang Microservice Starter Kit

TL;DR — check out our gosample repo.

This article was originally posted on the SYNQ blog.

Here at SYNQ, we’ve built our Video API service on Golang. While we move from a monolithic microservice architecture (more on that in a future story) to individual self contained services, we’re creating a lot more repositories.

When I was tasked with building our new “v2” API service (codenamed “Aerico”), which will provide a more flexible Video JSON object, I was about to use the same “SYNQ scaffolding” from another repo (Hydra) when I decided to build a little Gosample app first. This will allow us to

• Start building future Go services more easily and quickly
• Act as a living code-based docs for our tools and processes
• Quick reference guide to various Golang techniques and testing methodologies (useful for Golang newbies like myself!)

Key Capabilities

This sample app contains several capabilities and it may not all make sense for a new service, and that’s ok! The point of this is to cover a multitude of integrations so any new service can pick and choose what it needs. The main features are

• Continuous Integration with CircleCI and Coveralls
• Postgres support with migrations using Flyway
• SYNQ API integration using our Golang SDK
• Gin Web framework
• 100% test coverage via various mocking and code organization techniques

Let’s do a deeper dive.

It’s nice to test you

We use CircleCI and Coveralls. I am not going to go through our entire .circleci/config.yml file but I will highlight a few sections.

Below, you’ll see that we use our own circleci docker image and then the base library/postgres docker image. That docker image will provision the db, user and password if you set the right ENV vars.

The next section below shows a simple “wait” script as Postgres isn’t guaranteed to be up and provisioned by the time you use it, so we have to wait for the db and user to be ready.

This last piece is where we run the tests:

First thing you see is that we use something called overalls. This will create an aggregated .coverprofile file; this is useful for repos with multiple sub-directories. We also use goveralls, which is the Go SDK for coveralls (both of these are baked into our circleci image). Finally, you’ll see that I only run it when the ${COVERALLS_TOKEN exists and this is because we can’t pass along Env variables in the local circleci build and we wouldn’t want the coverage to be posted for local builds anyways.

Migrations over easy

As you saw in the config.yml, we use Postgres for our data store. But we need some convention for creating and managing database migrations. Here at Synq we use Flyway, a language independent migration tool that makes it easy to manage both one time and repeatable database migrations.

First, notice our directory structure:

sql/
  common/
    dbhelper
  environments/
    prod/
      .gitignore
      flyway.conf.in
      Makefile
      secrets.json
    test/
      flyway.conf
  migrations/
    V000__Init.sql

We organize our migrations and the environments in separate directories. The flyway.conf is located in the environments section. Notice how we have flyway.conf.in and secrets.json for our prod environment. This is because we use ansible-vault to manage our secrets so we never create the flyway.conf file directly. Below you can see what the unencrypted secrets.json looks like

We also have the Makefile which makes it easier to run migrations and we can simply type make migrate and it will handle the decryption of the secrets, generation of the flyway.conf file and run flyway migrate

The Code, The Code, The Code

Ok, this is what all (two) of you came for :). First, a quick overview of what this thing even does. It’s fairly straight forward, the service offers two APIs, /v1/details and /v1/status. Details will accept a video_id parameter and make a call to Synq’s API to retrieve the video details. It will also save the call information to the database. Status will list out all the API calls made through our service.

First, let’s look at the ApiCall struct:

So just a couple of things, since we will marshal the data into the database and then also return it in JSON form, you can pass in a “space” delimited list after the type to say what the parameters names will be for the respective formats. You’ll also notice the json:"-" field, which just means don’t marshal this in JSON.

Next, let’s look at setting up the Gin router:

We use the Group concept and version these APIs as v1. Also, we make details take a POST and status take a GET. You might also ask, “why have a setupRouter method in the first place when I could’ve just put this in main()”, and we go over that later, and its really about code coverage.

Now for the details method:

This is fairly straightforward, you are given the pointer to a gin.Context which gives you convenience functions like BindJSON and JSON for easily processing incoming JSON and return a JSON response. The “video_id” parameter is parsed when we run BindJSON since the ApiCall struct is looking for the “video_id” to bind to.

Now for the actual Call method:

Two things here, we use the defer method to save this call even if it fails. Second, the a.Video will get saved properly because we override the Value and Scan for the Video struct that exists in our Golang SDK.

If you don’t do this, you will get an error trying to save the struct to the database.

0 to 100% in 60 seconds

Ok, it definitely took longer than 60 seconds and getting to 100% code coverage is no easy task. There a lot of little nooks and crannies in your code that can be hard to get to. You have to test the Gin methods a certain way and you might actually have to refactor some of your code to make it easier to test. No fear, this Gosample will help with all that.

Let’s first look at some helper methods we created in our main_test.go

Every test will call our setupTest method. First you’ll see that we use require.Assertions which is from the popular Golang assertions toolkit testify. You notice we use require.New instead of assert.New. This is so that our tests will fail at the first failed assertion, where as assert.New will run all your tests. Second, you’ll see a curious test_helper.SetupServer method call. This is coming from our test_helper repo which has some convenience functions for things like setting up a mock server or issuing a test Gin request.

Now let’s look at the actual TestDetails unit tests:

You’ll see these test_helper.RunSimplePost methods. We use this because it’s not easy to create the gin.Context object so testing the actual details method without having to start the Gin server can be a bit difficult. These convenience functions makes that easy.

Conclusion

First, I am happy (and slightly surprised) you stuck with us this far! I know 5min+ blog posts don’t get a ton of reads but hopefully this was useful for you and we’d be happy for you to try out the gosample repo for yourself and see if it helps you in the ways it helps our team!

//TODO(batmany13)

There’s still quite a few things we could add, and here’s the short list

• Sample Dockerfile and docker-compose.yml
• Heroku Integration
• Convenience setup scripts for a new repo
• “Done” channel for long running goroutines
• Add repeatable migration example

You can also see the full list of issues on GitHub