Dynamic Configuration Management with Consul and .NET Core

Today we will learn, how to create and manage dynamic Configuration in .NET Core applications with Consul.

Pre-requisites,

1. Docker Desktop
2. VS 2022

Let’s get started.

Why Dynamic Configuration is Required?

Now a days all of the applications contains a configuration file that is loaded during runtime and has some settings related to your application, which is used by the code to produce results.

Now let’s say, you are required to make some changes in the settings, you will go to the code repository and make your changes and check in your changes in one of the version control tool such as Git or TFS etc. then build your application again and deploy.

Imagine if you have a large scale distributed system, then managing this configurations will become a pain, this is where Dynamic Configuration comes as a blessing.

Current Available options for Storing Configurations

1. File System: We can store our configurations in file system within the application host, it will have better performance but managing it becomes very difficult, as with a single change you again need to deploy it environment.
2. Databases: We can store the configurations in databases but it will introduce performance loss along with latency and higher response time, to overcome this we can cache the configuration In-Memory from database, but each time a configuration change is required, we need to invalidate the cache by restarting the application, which is again a painful task if you have multiple instances of your application running.
3. Key Value Stores: We can use some specific centralized Key Value store to store our application configuration, and we can watch for any changes are done on the data store side, so our application configuration will be reload on the runtime without any restarts. Such tools are Consul, etcd i.e.

What is Consul?

Consul is full fledged service mesh for microservices architecture, it is used generally used as a traffic manager that manage secure communication between multiple services in multi cloud architecture, but it has a Key/Value store which is normally used for application configuration storing purpose. We are going to leverage the key/value store for our purpose.

HashiCorp Consul(https://www.hashicorp.com/blog/products/consul)

Let’s install Consul on the docker by running below commands,

docker pull hashicorp/consul
docker run  -p 8500:8500 -p 8600:8600/udp --name=consul hashicorp/consul agent -server -bootstrap -ui -client=0.0.0.0

First command will pull the image and next will run the container using the same image with arguments to enable it as a single agent.

Now it created an Container Named “consul”

You can go to “Services — Consul” to check the UI,

Now our Consul installation is done, we will focus on creating a simple API application.

Create an ASP. Net Core Web API Application in Visual Studio 2022 with .NET 5.0 as framework.

VS 2022 .NET 5.0 Web API Application

Add Winton.Extensions.Configuration.Consul package to the project,

dotnet add package Winton.Extensions.Configuration.Consul --version 3.2.0

Let’s create a settings in Key/Value store in Consul via UI,

Create a new folder with path “ConsulIntegration.API/appsettings.Development.json”

It will create a below like folder structure,

As you can see, we have development appsettings as of now, if we want we can add environment wise appsettings as well.

appsettings.Development.json and appsettings.Prod.json

Now our settings are present in Consul KV.

Let’s create a API Endpoint, which will return some configuration value,

Let’s visit our physical appsettings.json file,

key1,key2 and key3 values are different in Consul

Now we will configure our code to read settings from Consul KV, we will configure it in our Program.cs file,

Now we are done with our Consul Configuration.

Key Points to remember,

ConsulConfigurationOptions => It is used to pass connection and authentication information regarding the consul.

PollWaitTime => It is used to configure the interval, to poll changes in Consul Key Value Store, for our configuration.

ReloadOnChange => This is used to reload the configuration if any changes are detected.

OnLoadException => This is used to handle if any exception occurred during initial load of the configurations.

OnWatchException => It is used handle if there is any exception occurred during polling

Now we are passing below, environment variables, which are used to construct the key for Consul and also Consul connection details( we have not setup any authentication, it is not recommended for Production environment)

{
        "ASPNETCORE_ENVIRONMENT": "Development",
        "ApplicationName": "ConsulIntegration.API",
        "ConsulURL": "http://localhost:8500/"
}

Let’s run the application and invoke API endpoint via Swagger , see the results,

Values are Coming from Consul

Let’s change the values in Consul and check if it is getting applied in the application without any manual interventions,

key1 value has been changed in Consul KV

Again invoke the API endpoint via swagger and check,

key1 value has been changed in application

As you can see, with out any manual intervention new configuration values have been applied.

Now let’s test this for resiliency against failures. Let’s say, our First Time load is successful but afterwards for any reason, application is not able to connect to Consul for detecting any changes.

In our code we have handled the situation only by logging the error, but application runtime will hold the earlier configuration thus ensuring application does not go into any unwanted state.

Application Logs

Configuration is still present

Now if we want to load different environment configuration, we can do that by changing the “ASPNETCORE_ENVIRONMENT” environment variable in launch settings,

{
        "ASPNETCORE_ENVIRONMENT": "Prod",
        "ApplicationName": "ConsulIntegration.API",
        "ConsulURL": "http://localhost:8500/"
}

launchSettings.json

Below are settings available, for appsettings.Prod.json,

Now again run the application and invoke the API endpoint directly, you will get a response like below,

Our Prod Environment variables are visible

Now we know, how Consul Key/Value stores makes our life little easier by keeping this configuration safe in a centralized location and providing the changes seamlessly.

For reference,

1. Consul: https://developer.hashicorp.com/consul
2. GitHub Repository: https://github.com/arkapravasinha/ConsulIntegration
3. Open Source Library: https://www.nuget.org/packages/Winton.Extensions.Configuration.Consul/

Kudos to https://www.nuget.org/packages/Winton.Extensions.Configuration.Consul/