Help! My house is on fire but I need to create an API fast!

This guide is for people that don’t necessarily want to take a deep dive into complex technical details but would rather create and deploy an API without being overloaded by so many buzzwords they don’t know where to start. Later on you will find few code examples that will help you jump-start into the world of automated software releasing.

What will you learn after reading this article?

  • What is continuous integration
  • What is continuous delivery
  • What is continuous deployment
  • How to get rid of a GOPATH using Go GB
  • How to handle Go vendor dependencies
  • How to deploy an Go application to Heroku for free
  • How to restrict an endpoint with a JWT authorization token
  • How to deploy to test and production servers

Things you should know

What’s the difference between Continuous X and Continuous Y

There is a chance that you are probably wondering “what’s the difference between all of this Continuous stuff?” so in this section I will try to briefly introduce you to these 3 basic concepts. To make it easier to grasp I will cite a blogpost titled Continuous Delivery vs Continuous Deployment vs Continuous Integration: Key Definitions by Sergiy Golub

Continuous Integration

Continuous integration is the practice of constantly merging development work with a Master/Trunk/Mainline branch so that you can test changes and test that those changes work with other changes.[1]

Let’s imagine that we have a masterbranch and we want to merge our fresh feature branch onto it. To do so we have to create a Pull Request that has to:

  • be up to date with our master branch
  • all its tests need to pass
  • it should be reviewed (if we have a strict policy on it as we should)

Then if all of these points are met we are able to merge our branch and move on to the next task.

Continuous Delivery

Continuous delivery is the continual delivery of code to an environment once the developer feels the code is ready to ship — this could be UAT, staging or production. The idea behind continuous delivery is that you’re constantly delivering code to a user base, whether it be QA or directly to customers for continual review and inspection.[1]

For example:

  • master branch where the latest commit is going to the Staging server
  • feature branches that could be deployed to their own environments hosted under their very own endpoints

This will help us make sure that all our features are working fine on their own and with the current masterbranch that represents the Staging. I’ve met the approach with more than one test server so let’s say:

  • pre-alpha stage
  • alpha stage
  • beta stage
  • production

It’s really up to you and your team to figure out which approach would work best and make the whole process as bug free as possible!

Continuous Deployment

Continuous deployment is the deployment or release of code to production as soon as it’s ready.[1]
So that’s it! No tests because this branch should be stable and all tests should pass before merging anything to it. Simply take your latest tag; build it and then deploy to the production server.

Git

So what is this Git I keep hearing about?

Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. [3] Long story short Git makes it extremely convenient for multiple people to work on the same projects or even the same file without worrying about conflicts and overwriting each other’s work.

What do I really need to know?

  • How to create and push commits
  • What are branches and what to do with them
  • Create
  • Checkout
  • Merge
  • Delete

Where can I learn it?

A good place to learn Git is the official free interactive tutorial you can find on try.github.io.

What’s GitLab?

Imagine GitLab as a platform that combines some cool features from GitHub, Jira, Jenkins and Docker Hub. That’s pretty much it. GitLab allows you to host your git repositories, manage issues, plan milestones and releases, build/test/deploy your projects and it even host a docker images for you. I couldn’t recommend it more so go ahead and take a look at their official website: about.gitlab.com.

Heroku

Heroku is a cloud platform as a service (PaaS) supporting several programming languages that is used as a web application deployment model.[4] In simple words Heroku allows you to build, deploy and run your apps without the need to manage your own servers on bare metal. One of the biggest benefits is their free plan that allows you to test your apps or build prototypes for free.

Golang

Go GB

Go GB is a project based build tool for the Go programming language.[5] The most well known benefit of using GB to build your projects is that you don’t have to set any environmental variables such as the GOPATH.

Thanks to that you don’t have to worry about dependency conflicts when working on a multiple projects at the same time (such as bunch of microservices).

Minimal Go GB application

Before we start we need to create a repository on gitlab.com with the following folder structure:

repo-name
└───src
└───project-name
└───main.go

Next we should install GB by typing go get github.com / constabulary / gb...in our terminal. When we have GB installed and it we have GOBIN linked to our PATH then we should install all necessary dependencies using GB:

gb vendor fetch github.com/labstack/echo/…
gb vendor fetch github.com/spf13/viper

after that we should end up with the similar project structure:

repo-name
└───src
│ └───project-name
│ └───main.go
└───vendor
│ └───…
└───pkg
│ └───…
└───bin
└───…

As you can see we are using github.com instead of a gitlab.com. That’s because these Go packages are using GitHub to host their repositories. (Go is using git for package management).

main.go file contains our example application (slightly modified example from Echo’s official cookbook) and it should look like this:

package main
import (
"net/http"
"github.com/labstack/echo"
"github.com/spf13/viper"
)
func main() {
viper.BindEnv("port")
envPort := viper.GetString("port")
e := echo.New()
e.GET("/", func(c echo.Context) error {
return c.String(http.StatusOK, "Hello, World!")
})
e.Logger.Fatal(e.Start(":" + envPort))
}

Pretty straightforward isn’t it? That’s all it takes to create a basic API with one GETendpoint under / using Echo framework. I will show you how to add another endpoints under JSON Web Tokens (JWT and restricted endpointssection later on but for now let’s deploy our hello-world API.

Manual Deployment

First of all you need to go to the Heroku dashboard and click the New button then select Create new app it and select a region where you want the server to be located.

Empty list of the Heroku’s buildpacks

When you create your app go to the Settingstab click Add buildpackand then select go just like you can see on the screenshots below.

Selecting a Go buildpack

Heroku’s go buildpack supports GB out of the box so it will fetch all of your dependencies and do the build for you. If you have more complicated build process you can use .gitlab-ci.yml file for this (either invoke your scripts manually there, use a makefile or a Docker).

On the Settings page you will also find a domain under your API will work (see the Domains and certificates section). To manually deploy your API please go to the Deploy page and look for a Deploy using Heroku Git. Everything is nicely explained there.

If you followed all the steps you should be able to see a familiar Hello, World! under your newly created endpoint. Congratulations! New let’s move forward to the fun part.

Automatic Deployment

To do the automatic deployment we need two things:

  • Heroku API Key that we can find on the Account page
  • App name we want to deploy from the Settings page of our app on Heroku

When you obtain these two things go to your GitLab repository click on Settingsup on the top next go to Pipelinestab and then look for a Secret Variablessection. Add a new variable called HEROKU_APP_NAME_STAGINGand enter your app name here then add another variable called HEROKU_API_KEY and past your API Key. In this place you can set a limit on which scope this variables will be passed to so for example a stagingor productionenvironments. No need to change that just now so we will leave it as it so * (it will pass our variables to every job we run no matter of the environment name).

Variables passed down to the pipelines

At this point I would advise you to create another app to simulate a production server. If you do that please remember to add another variable called HEROKU_APP_NAME_PRODUCTION. When you are done create a new file called .gitlab-ci-ymlat the root of your project structure with the following content:

image: ruby:latest
stages:
- deploy
before_script:
- gem install dpl
staging:
stage: deploy
environment:
name: staging
url: https://mz-simple-api-test.herokuapp.com
only:
- master
script:
- dpl --provider=heroku --app=$HEROKU_APP_NAME_STAGING --api-key=$HEROKU_API_KEY
production:
stage: deploy
environment:
name: production
url: https://mz-simple-api.herokuapp.com
only:
- tags
script:
- dpl --provider=heroku --app=$HEROKU_APP_NAME_PRODUCTION --api-key=$HEROKU_API_KEY

That’s it! Now you have to simply push your changes to the GitLab’s origin and the application should deploy itself to the staging server. If you want to do some basic continuous integration you can do it in this file as well. To do so modify your .gitlab-ci-yml like this:

image: ruby:latest
stages:
- test
- deploy
before_script:
- gem install dpl
tests:
stage: test
script:
- // do your tests here
staging:
stage: deploy
environment:
name: staging
url: https://mz-simple-api-test.herokuapp.com
only:
- master
script:
- dpl --provider=heroku --app=$HEROKU_APP_NAME_STAGING --api-key=$HEROKU_API_KEY
production:
stage: deploy
environment:
name: production
url: https://mz-simple-api.herokuapp.com
only:
- tags
script:
- dpl --provider=heroku --app=$HEROKU_APP_NAME_PRODUCTION --api-key=$HEROKU_API_KEY

This way when the developer creates a Pull Request you can make it mandatory to allow merging to the masterbranch only when all tests specified in this file are passing.

Alright so how do I deploy to the production now?

It is as simple as creating a new tag. Go to the Repository → Tags page and click the New Tagbutton.

Tagging production releases

When you see the New Tag page fill the Tag name field with the version you want to deploy so for example 0.1.5and select a branch you want to create a tag from (for the most cases it will be your staging branch like master). You can fill Messageand Release notesas you wish. It doesn’t affect the deployment process unless you want to have a version number or release notes included in your application (you can do this in the .gitlab-ci.yml file using environment variables probieded by GitLab you can find on their official documentation docs.gitlab.com/ce/ci/variables/README.html).

Tags view with one tag already created

If you want to check how the build is doing and if nothing has failed you can go to the Pipelinespage. If you see something failing simply click on the status icon and you will be redirected to the Job that is failing. You will have a logs file similar to the one Jenkins have.

Pipeline view showing staging and tag builds

On the Pipelines → Environments page you can see your staging and production environments to find out what commit is currently deployed on which server.

Environments view showing what’s currently deployed on our servers

Congratulations! You have automated your build and deployment process.

JSON Web Tokens (JWT) and restricted endpoints

Sometimes you don’t want everyone to access your endpoints. To quickly restrict an access to them we can use JWT.

When we are using JSON Web Tokens we need a secretvariable to sign our tokens. A good practice is to not hardcode them and pass them using environmental variables or config files that are not versioned in the git repository.

To create an environment variable that we will have access to on the runtime as opposed to the ones we used during our build we have to go to the Settingspage of our Heroku application then click the Reveal config Varsbutton and add a variable with a key named secretand most likely randomly generated hash as a value. This value will be later used to sign our tokens and validate existing ones so it would be a good idea to have a separate set for staging and production servers.

Adding a environment variable to an app in Heroku

When you have secretenvironment variable up and running modify your application as shown below:

package main
import (
"net/http"
"time"
"github.com/dgrijalva/jwt-go"
"github.com/labstack/echo"
"github.com/labstack/echo/middleware"
"github.com/spf13/viper"
)
func main() {
viper.BindEnv("port")
viper.BindEnv("secret")
envPort := viper.GetString("port")
secret := viper.GetString("secret")
e := echo.New()
e.Use(middleware.Logger())
e.Use(middleware.Recover())
e.GET("/", func(c echo.Context) error {
return c.String(http.StatusOK, "Hello, World!")
})
e.POST("/login", login)
r := e.Group("/restricted")
r.Use(middleware.JWT([]byte(secret)))
r.GET("", restricted)
e.Logger.Fatal(e.Start(":" + envPort))
}
func login(c echo.Context) error {
username := c.FormValue("username")
password := c.FormValue("password")
if username == "jon" && password == "shhh!" {
// Create token
token := jwt.New(jwt.SigningMethodHS256)
// Set claims
claims := token.Claims.(jwt.MapClaims)
claims["name"] = "Jon Snow"
claims["admin"] = true
claims["exp"] = time.Now().Add(time.Hour * 72).Unix()
// Generate encoded token and send it as response.
t, err := token.SignedString([]byte("secret"))
if err != nil {
return err
}
return c.JSON(http.StatusOK, map[string]string{
"token": t,
})
}
return echo.ErrUnauthorized
}
func restricted(c echo.Context) error {
user := c.Get("user").(*jwt.Token)
claims := user.Claims.(jwt.MapClaims)
name := claims["name"].(string)
return c.String(http.StatusOK, "Welcome "+name+"!")
}

When you’re done all it’s left to do is push your changes to the GitLab’s origin and watch it deploying itself.

Conclusion

As you can see it is very easy to create an automated software delivery these days and the best part of it is that you can do all of this for free in just a few minutes. In my opinion it is crucial for modern web applications to be built and deployed automatically. I’ve seen too many times that developers (including myself) incorrectly released something after coming back to the project after a long time or working under a serious time pressure when deploying this post-release hot-fix.

With the Docker and GitLab’s Review Apps on the other hand you can even create the whole environments that you can host for your feature branches to make a lifes of your testers a bit more pleasant. But that’s a topic for a whole blogpost for itself.

Resources

[1] Sergiy Golub. Continuous delivery vs continuous deployment vs continuous integration: Key definitions. Retrieved August 17, 2017 from https://blog.assembla.com/assemblablog/tabid/12618/bid/92411/Continuous-Delivery-vs-Continuous-Deployment-vs-Continuous-Integration-Wait-huh.aspx

[2] Chris Richardson. What are microservices? Retrieved August 6, 2017 from http://microservices.io/

[3] Git’s official website. Retrieved August 6, 2017 from https://git-scm.com/

[4] Heroku on Wikipedia. Retrieved August 21, 2017 from https://en.wikipedia.org/wiki/Heroku

[5] Go gb official website. Retrieved August 6, 2017 from https://getgb.io/

[6] The go programming language — command go. Retrieved August 18, 2017 from https://golang.org/cmd/go/#hdr-Environment_variables

[7] JSON web tokens — jwt.io. Retrieved August 18, 2017 from https://jwt.io/

Author: Maciej Zdziarstek