Configuration Management With Consul

Gizem Cifgüvercin
Oct 2 · 4 min read

We use configurations for feature flagging, easily recovering, easier collaboration between teams, and easier data change operations. If we have distributed applications in large-scale, we should deep dive about how to manage them.

Let’s have a look at which alternatives we can choose;

1- Store the data in application settings

2- Store the data in database

3- Store the data in Key/Value store

Problems that we can face with:

If we store config data in application settings, we don’t lose performance but we should commit the changes on the project and restart our application for using new values of config

If we store data in the database, we may lose performance and have more latency and high response time. In this time we can cache data in memory cache for performance but we should restart our application for invalidating cache

Solution

  • dynamic configuration

Using Key/Value store with pub/sub mechanism is a good idea. We can reload changes in the application runtime. So we don’t have to restart or deploy for cache invalidation. When the data change in store, the application gets the new values and bind the data into related configs

We may use RollingUpdate as a deployment strategy. We can think deployment cons doesn’t matter so much for now but with this solution, your application will never be unavailable for config operations

Some popular key-value stores are:

  • Aerospike
  • Apache Cassandra
  • Berkeley DB
  • Couchbase Server
  • Redis
  • Riak
  • Consul

Note

Consul isn’t just a key value store, it provides features about service discovery, secure service communication, multi datacenter etc

Expectations

1- Dynamic configuration

2- Exception Handling

  • On Load
  • On Watch

3- Fallback Scenario

Client Libraries & SDKs

You can use Consul with C#, Go, Java, Python, Php, Swift, Dart, etc

https://www.consul.io/api-docs/libraries-and-sdks

Consul has own endpoints that we can do crud operations https://www.consul.io/api/kv. You can choose either writing your client library or using popular client libraries

I will explain usage with .Net 5 and Winton.Extensions.Configuration.Consul client library

Implementation

First of all, we need to have a consul server which our application will connect to

The command will create consul instance with Docker

Follow http://localhost:8500/ui/dc1/services URL and visit consul UI as below, now we have a key/value store

Create your first configuration data in Key/Value section

Application

Create .Net 5 Web API

dotnet new webapi -n ConsulExApplication

Add consul client library

dotnet add package Winton.Extensions.Configuration.Consul

Configure Consul in your application

- Source will be reloaded if the data in consul changesConsulConfigurationOptions- Allows the default config options for Consul to be overridenPollWaitTime- Maximum amount of time to wait for changes to a key if ReloadOnChange is setOnLoadException- Action that is invoked when an exception is raised during config loadOnWatchException- Func that is invoked when an exception is raised whilst watching

By our consul settings, we will be poll consul data every 5 seconds. If we can’t get config data on the application start, we will catch the exception in OnLoadException func so that our application will not crush and continue with appsettings data in the project.

Additionally, we can improve the log mechanism and monitor exceptions in monitoring tools like Kibana. We can define rule and send alerts for actions

If we can’t get config data while the application is running, the application will handle requests with binded data before from Consul. With the rule that we defined in OnWatchException, the application will try to reconnect and get the latest data from Consul every 2 seconds

Read Config Value

Let’s call the endpoint and see the result :

1- When Consul worked successfully on load

2-When Consul didn’t work successfully on load and consul still gives an error (bc of not running or wrong data format) -Fallback-

3- When Consul didn’t work successfully on load and consul is up again

***** Data will be changed from Consul

From application
From Consul

4- When Consul worked successfully on load but consul is down after (bc of not running or wrong data format)

Last taken data from Consul before consul is down

References

https://www.consul.io/

hepsiburadatech

hepsiburada technology