Making your Go app configurable

Using a good settings strategy can make local development and deployment a lot easier.

Photo by Digital Buggu from Pexels

Recently, one of my team members wanted to run a service from a Go repository created by another team member. Instinctively, he downloaded the repository and try to fire it up in his favorite editor. But his editor disagreed and all he got was an error message saying that he’s missing an environment variable.

So much for running out-of-the-box.

Unknowingly we created a repository that depends on environment variables and uses validation on them to break whenever something is missing or is invalid.

Although that sounds reasonable, it’s not very developer-friendly. It’s in our nature to always try the fastest option. We don’t want to plow through the README first (which in our case actually contained instructions on which environment variables to set), we just want to git-clone it, build it and run it.

But why does the repo require environment variables to be set in the first place? That is related to our CI/CD infrastructure. We run our applications in Kubernetes (GKE to be precise) and we have several environments at our disposal, each having different settings. For this, Kubernetes allows you to set environment variables, so you can configure different settings for each deployment.

Using environment variables works perfectly for deploying our application to various environments. But what we forgot is that we also have our own machines, which should be regarded as a different environment. One could argue that each developer is responsible for configuring their own laptop, but since most of the settings are the same for everyone, it’s better to configure them in the repository.

Fortunately we’ve faced this scenario before. For our team, Go is not the primary programming language, it’s C#. And for our .Net Core repositories we already have a mechanism to deal with the local environment. First, we use an appsettings.json settings file, which is read during start-up. The settings configured in this file can be regarded as defaults. Next, these settings can be mapped to environment variables using naming conventions. As a result, we can use environment variables to override the default values.

Here’s an example. Suppose we have a settings file like this:

{
  "Log": {
    "MinFilter": "Debug"
  }
}

By convention, environment variable names contain the upper-cased full path of the setting, joined by two underscores. In this case, the env var name for Log.MinFilter is LOG__MINFILTER (note the dual underscore) and the default value is “Debug”. This can be overridden by creating an environment variable LOG__MINFILTER and giving it a different value.

For each .Net Core application that we have, there’s a settings file containing the defaults to use locally. The environment variables are configured in a separate YAML file and are applied by Kubernetes during deployment.

This works neatly in .Net Core, so we would like to have something similar in Go. Luckily, there is a Go package called envconfig that does precisely what we need.

Suppose we have this configuration file:

{
  "Log": {
    "MinFilter": "Debug"
  },
  "Cors": {
    "Origins": ["*"]
  },
  "SomeService": {
    "Url": "https://some-service.com"
  },
  "SomePublisher": {
    "Env": "staging",
    "Project": "staging-project",
    "Topic": "some-pubsub-topic"
  },
  "SomeXml": {
    "Storage": {
      "BucketId": "some-xml-bucket"
    },
    "CacheTtlMin": 10,
    "AuthBasicToken": "LocalUse"
  },
  "SomeName": "Some name"
}

And we have this settings struct:

package main

type AppSettings struct {
   Log struct {
      MinFilter string `envconfig:"optional"`
   }
   Cors struct {
      Origins []string `envconfig:"optional"`
   }
   SomeService struct {
      URL string `json:"Url" envconfig:"optional"`
   }
   SomePublisher struct {
      Env     string `envconfig:"optional"`
      Project string `envconfig:"optional"`
      Topic   string `envconfig:"optional"`
   }
   Google struct {
      Application struct {
         Credentials string
      }
   }
   SomeXML struct {
      Storage struct {
         BucketID string `json:"BucketId" envconfig:"optional"`
      }
      AuthBasicToken string `envconfig:"optional"`
   } `json:"SomeXml"`
   DefaultProvider string `envconfig:"optional"`
}

As you might have noticed, there are specific tags for parsing of JSON and for envconfig. The “optional” tag is used to indicate that there’s no environment variable required for that setting. For example, LOG_MINFILTER is not required here, because we can use the default value. However, GOOGLE_APPLICATION_CREDENTIALS is required here, because it doesn’t contain the “optional” tag. And it’s also not in the settings file, because the credentials should only be stored on your local machine.

We still need some code to read the settings file and make sure envconfig does it’s thing. So let’s create a local package called appsettings and add a read method:

package appsettings

import (
   "encoding/json"
   "io/ioutil"
   "os"

   "github.com/pkg/errors"
   "github.com/vrischmann/envconfig"
)

// ReadFromFileAndEnv reads the settings from a local file and applies any existing environment variables to it.
func ReadFromFileAndEnv(settings interface{}) error {
   file, err := os.Open("appsettings.json")

   if err != nil {
      return err
   }

   defer file.Close()
   data, err := ioutil.ReadAll(file)

   if err != nil {
      return errors.Wrap(err, "Failed to read appsettings")
   }

   err = json.Unmarshal(data, settings)

   if err != nil {
      return errors.Wrap(err, "Failed to unmarshal appsettings")
   }

   err = envconfig.Init(settings)

   if err != nil {
      err = errors.Wrap(err, "Failed to update with env vars")
   }
   return err
}

Now we have settings file and a read method. Let’s see if we can make this work. Here is the main function:

package main

import (
   "fmt"

   "go-configuration-demo/appsettings"
)

func main() {
   // placeholder variable
   settings := AppSettings{}

   // filling the variable with the settings file and env vars
   if err := appsettings.ReadFromFileAndEnv(&settings); err != nil {
      panic(err)
   }

   // do something with the settings
   fmt.Printf("%+v", settings)
}

We’re defining a variable settings as an empty AppSettings struct, which gets filled in by our previously defined appsettings.ReadFromFileAndEnv() method.

When you run this code, chances are that it will fail with the following message:

panic: Failed to update settings with env vars: envconfig: keys GOOGLE_APPLICATION_CREDENTIALS, google_application_credentials not found

Good. The tag in the appsettings file was correctly used and resulted in a panic, because the GOOGLE_APPLICATION_CREDENTIALS environment variable was not set. The reason we want this behavior here is because it’s value is something that you normally don’t want to be part of your repository.

New let’s fill in a value:

export GOOGLE_APPLICATION_CREDENTIALS=/some/path.json

And run the application again. This time we’re getting a successfully merged settings struct:

{Log:{MinFilter:Debug} Cors:{Origins:[*]} SomeService:{URL:https://some-service.com} SomePublisher:{Env:staging Project:staging-project Topic:some-pubsub-topic} Google:{Application:{Credentials:/some/path.json}} SomeXML:{Storage:{BucketID:some-xml-bucket} AuthBasicToken:LocalUse} SomeName:Some name}

Which is exactly what we wanted: settings from the appsettings file merged with environment variables.

To summarize, a solid settings strategy that you could use for Go application development consists of:

• A settings file in your repository for local testing
• Environment variables that can be configured upon deployment
• Some logic in your Go code to combine these two into a settings struct that you can use in your application.