GOLANG: HOW TO GUIDE

Simplify Your App Configurations in Go

Easily read, write, and manage your app configurations from remote sources using Viper (Go open-source)

Gopher art credit to Egon Elbre at www.github.com/egonelbre/gophers

Introduction

Manually copying configurations across different computer systems is a needless risk.

Think about it — how many times has a colleague created a program that required a config but didn’t post it anywhere? Or had something configured differently in different places?

Or forgot to update a system along the way?

Personally, if I can pull up a configuration and see it myself without asking another developer, that’s a big win and saves me time.

Centralizing how your applications can read all their configurations simultaneously is huge, especially when all you have to do is update the configurations once, and boom! You’re done.

Distributed processing, team collaboration and visibility, and accessibility improve almost overnight.

Remote stores, such as databases or specialized config services (e.g., Consul; etcd) can help with that, but the configs inside them still need to be fetched and made available to our application for use.

Instead, let me introduce Viper — an open-source tool that can help you easily manage your Go application's configuration from different sources. It’ll help save you time, money, and future headaches.

GitHub - spf13/viper: Go configuration with fangs

❗️TL;DR:

Viper is an open-source configuration management library for Go that allows you to consume configurations from different sources and use them within your application(s) in a standard way.

Regardless of your configuration source, Viper simplifies your application’s configurations, ensuring the latest ones are used as soon as any changes are made.

If you have a large Go application, or distributed Go applications where consistency is required, Viper becomes a breeze. It decouples your application(s) from its configurations, allowing you to conveniently scale.

Short Preface

What you’ll learn

By the end of this article, you should be able to:

• Read and access configs from remote sources, and use them within your Go application
• Feed them to Viper so that all your configurations are standardized, accessed, and managed in a central location
• Watch config changes from a remote source and provide the latest configs to your application as soon as the changes are made

Prerequisites

To follow along, you’ll likely need:

• A moderate understanding of Go
• Familiarity with Firebase and Cloud Firestore (recommended)

📌 Disclaimer: This is the second of two articles. In the previous article, we focused on Viper’s features and how to use them to manage configs from local sources (and environment variables). To read more, see the article below:

Manage Configurations From Local Sources in Go

Getting Started

Reading remote key-value stores using Viper

The good news is, Viper provides us with tools to read configurations from remote stores, such as Cloud Firestore, Consul, etcd, etc. These are baked into Viper and take just a few lines to set up.

📖 Further Reading: You can learn more here or through the code below👇

viper.AddRemoteProvider("firestore", "google-cloud-project-id", "collection/document")
viper.SetConfigType("json")
err := viper.ReadRemoteConfig()

However, while the inherent support for remote stores is great, it can also be inflexible at times, especially when you want to store configurations in your preferred store. A good example of this is Firestore support (see here for example).

💡 Expert Advice: Just be cognizant of Viper’s possible limitations moving forward, that’s all. It’s not perfect, but if used correctly and in the right circumstances, it can truly be an asset.

Reading configs from non-predefined sources

Viper’s baked-in tools are great, but I want to focus on reading configurations from any remote store we wish. [That’s going to be more helpful, right?]

To do so, we’ll first build our own version of reading configuration sources from Cloud Firestore. Then, we’ll watch changes to our configuration and update our application config whenever these changes happen.

To start, we’ll use the viper.ReadConfigs method provided by Viper, which will allow us to feed it with any data we want.

viper.ReadConfigs accepts io.Reader, and the data that we pass to it can come from any source we want. It just needs to be in a format that Viper can understand (i.e., JSON, YAML, etc.):

We also need to tell Viper the format we are using. We’ll do this through the viper.SetConfigType method, in our case, above JSON ☝️.

Installing dependencies and getting the required credentials

To read configurations from Cloud Firestore, we’ll start by installing the Firebase package for Go and Viper, using go get:

go get firebase.google.com/go
go get github.com/spf13/viper

Next, we’ll need credentials to authorize our Go application to access Firebase Cloud Firestore. We’ll use service accounts to authenticate our Go applications against Firebase, allowing us to query our Cloud Firestore database.

📖 Further Reading: Learn more here about Firebase Service accounts here

We’ll also save the credential file we get from Firebase/Google Cloud Platform within our file system; please take note of the path, as we will need it in the next step.

👉 Note: (1) Do not commit the credentials file you are going to get from Firebase; (2) The service account allows you to access all Firebase services and if you are running this in production, it’s best practice to only allow your credentials to access the necessary services. You can learn more about how to manage credentials on both Firebase and Google Platform here.

If you are using another remote store or specialized config service, follow the relevant, official guidelines on acquiring the credentials for the specific system (and how to use them).

Managing Configurations From Cloud Firestore

Reading configurations from Cloud Firestore

Next, we will need to create a Firestore client that we can use to fetch and watch changes in our Firestore Document.

This is also the place where we need our credentials file by using the option.WithCredentialsFile() method, passing the path to the credential file as the argument. Then we need to pass the resulting variable to the firebase.NewApp() method:

Next, since we now have a client, we can go ahead and read the configuration document from Cloud Firestore, as shown below.

Cloud Firestore is a NoSQL database that is document-oriented. All documents are part of a collection (similar to tables).

To access a document, you need to know the collection it's in and the id of the document.

📖 Further Reading: For more information about the structure of Cloud Firestore, please check the official documentation here

In our case above, we are accessing a collection named configs and a document with id config.

This will allow us to have multiple configs for different applications or users, for instance, if we offer our users the ability to save their app configs remotely and sync across multiple systems.

When we read the data, we are converting the resulting map data to JSON so that we can pass it to Viper using by using the ReadConfig method. ReadConfig accepts an io.Reader, which is what enables us to pass configs from any source to Viper:

At this point, we can read our configurations from Cloud Firestore, just like any other configuration, as if it came from any other source using Viper:

For all methods to Get config values, please refer to Viper docs here

Watching config changes from Cloud Firestore

To recap so far, we can read configs stored in Firestore, but if they changed, we would need to restart our application in order for it to get the latest changes.

It would be great if config changes were automatically detected and updated within our application so we could use them as soon as possible.

Fortunately for us, with Cloud Firestore, we can listen to changes automatically in either a collection or a specific document using the Snapshots method.

👉 Note: Snapshots returns a new snapshot whenever there are changes with the latest snapshot of the data

We will use this method to listen to changes within our config document and use the viper.ReadConfigs to update configs in our application:

What if my config source doesn't have this capability?

In that case, you could opt for a “polling solution,” where you would check in at reasonable intervals to see if configurations have changed:

And that's it; any changes on our Cloud Firestore config document will be automatically reflected within our application. Here is what our code should look like now:

Using Our Configuration within our Application

Let’s set up Viper for our application and use the above configuration. We’ll need to tell Viper that our configuration file format is JSON since we are marshaling our data from Cloud Firestore to JSON.

👉 Note: We are running our function to fetch the remote configs inside a Go routine to enable us to listen to updates from Cloud Firestore asynchronously and update viper configs automatically without blocking the main thread

…and we’re done!

Questions? Comments? Feel free to leave them below👇 and let me know what you think.

Conclusion

In short, we learned how to use Cloud Firestore as a source for our configurations, plus how we can query Cloud Firestore, read the data and pass it to Viper to use it within our application.

On top of that, we also learned how to watch the configuration changes in Firestore and update Viper so that they are made available within our application as soon as that happens.

Key Takeaways

• Viper provides us with baked-in tools to read configurations from remote stores, such as Cloud Firestore, Consul, etcd, etc., which can be set up pretty quickly
• If you want to read configurations from a remote store, you’ll likely need to build your own version of reading configuration sources. Then, watch for configuration changes and update accordingly
• To do so, use viper.ReadConfigs to feed configurations to Viper from sources that Viper doesn't provide direct support for
• We can listen to changes within Cloud Firestore automatically in either a collection or a specific document using the Snapshots method and pass the changes to Viper.
• Lastly, don’t forget to set the format for your configs appropriately. It can drive you nuts if you miss this.

Other Articles To Consider

Manage Configurations From Local Sources in Go

How To Run Go Integration Tests for Third-party Services Using Docker

Disclosure: In accordance with Medium.com’s rules and guidelines, I publicly acknowledge financial compensation from UniDoc for this article. All thoughts, opinions, code, pictures, writing, etc. are my own.