Configuration Management in Ballerina

Published in

Ballerina Swan Lake Tech Blog

7 min readJan 1, 2019

This post is a looong overdue follow-up of a sort to my previous post on the config API: https://medium.com/@pubuduf/ballerina-config-api-f6a9c455267b. The config API went through a major refactoring for the Ballerina 0.970.0 release. With the new type system which was introduced for the 0.970.0 version, retrieving configurations using the then API became cumbersome. There were instances where 5–10 line functions had to be written to read a single config value in the required data type (e.g., retrieving a port number as an integer). To address this, a new comprehensive API was introduced. In addition to the API change, the following changes were also made:

Adopting TOML as the configuration file format.
Addition of a mechanism for securely storing/reading sensitive data in/from a configuration file.
Improvements to the configuration resolving.

TOML for Configurations

The Ballerina package management tool uses TOML for its configurations. To keep things consistent across all aspects of Ballerina and since the config file format was already pretty close to TOML, we opted to use TOML for the config API as well. However currently some features are not supported (e.g., arrays, inline tables).

Let’s take a look at the changes by using the config API to configure a simple echo service.

The New Configuration File Format

We’ll use the following config file to configure the echo service.

As it can be seen, there isn’t much of a difference between the old format and the newly adopted TOML format. The most notable difference would be in how the config values are treated in the TOML based configs. Previously, all config values were treated as strings. Hence, there was no need to enclose string values within quotes. However, in TOML, values can be of the following types: string, integer, float, boolean, datetime, array, or inline table. The config API however only supports the following value types: string, integer, float and boolean. If the value is a string, now you have to enclose the value within quotes.

Configuration Resolving

Configurations are resolved and stored in a registry at the start of the Ballerina runtime. The configuration source priority hierarchy remains unchanged, and is as follows, in descending order of priority:

Runtime parameters (provided through the CLI to the run command using the -e flag)
e.g., $ ballerina run -e b7a.log.level=DEBUG foo.bal
Environment variables
Since periods are not allowed in environment variables, periods in config keys are replaced with underscores when looking up environment variables. Hence if you are setting an environment variable to override a config given using a config file or a default configuration, you should use underscores in place of periods (if there are any) in the config key.
Config file
Default configurations (hard coded configurations. e.g., log level)

Configurations for a given program can be a mix of the above 4 sources of configurations. If the same configuration key is present in multiple sources, the precedence goes to the source which has the highest priority.

The New API

The following APIs are available for the user:

public function contains(string key) returns booleanThis can be used to check if a particular configuration is present in the config registry. Useful if there is a need to distinguish between the default value returned from a getAsXXX() function and the absence of a configuration since the default value is returned in the absence of a configuration as well.
public function getAsBoolean(string key, boolean default = false) returns boolean
Returns the config value for the specified key as a boolean. If the config is present in the registry, it is returned. In case the value you are attempting to retrieve is a string, it will attempt to parse it as a boolean. If the value is “true” (ignoring case), it will return true.
public function getAsFloat(string key, float default = 0.0) returns floatReturns the config value for the specified key as a float. If the actual value is not a float nor an int nor a string which is parse-able to a float, it will result in a panic.
public function getAsInt(string key, int default = 0) returns intReturns the config value for the specified key as a int. If the actual value is not an int nor a string which is parse-able to an int, it will result in a panic.
public function getAsMap(string key) returns map<any>Returns a related set of configurations (a table in TOML terms) as a map. For example, calling config:getAsMap("echo") would return a map with all the configurations in the table “echo”. If there is no such table or if it is an empty table, an empty map will be returned.
public function getAsString(string key, string default = "") returns string
Returns the config value for the specified key as a string.
public function setConfig(string key, string|int|float|boolean value)Can be used to add configs to the registry in Ballerina code.

Note the optional parameter default in the getAsXXX() functions (except in getAsMap()). This is the value returned if the specified config key cannot be found in the registry. This is useful if a required config is missing during run time, but you still want to attempt to run it using a default value.
e.g., config:getAsString("host", default = "localhost")

The following simple echo service demonstrates the use of this new API.

As it can be seen, this approach is much more user friendly than the previous approach where the conversion of the values to their appropriate type was left for the user to do.

The above service can be run with the config file as:
$ ballerina run -c echo.conf echo_service.bal or
$ ballerina run --config echo.conf echo_service.bal

Securing Sensitive Configuration Values

As you may have noticed, in the above config file we used, the key store password is stored in plain text. With Ballerina 0.970.0, the config API was further enhanced with an encryption tool for encrypting sensitive values.

To encrypt a value, one can use the ballerina encrypt command. Here’s what it looks like in the terminal, when the key store password is encrypted using this tool. The value “ballerina” was encrypted using the secret “1234”.

$ ballerina encrypt
Enter value:Enter secret:Re-enter secret to verify:Add the following to the runtime config:
@encrypted:{92XujbVRx+rXspbI/8sbdpdrmBmMF1PBDnuVUJKdK/0=}Or add to the runtime command line:
-e<param>=@encrypted:{92XujbVRx+rXspbI/8sbdpdrmBmMF1PBDnuVUJKdK/0=}

The following is the updated config file, with the plain text password replaced with its encrypted version.

Now let’s try running our echo service using this updated config file.

$ ballerina run -c secure-echo.conf echo_service.bal 
ballerina: enter secret for config value decryption:Initiating service(s) in 'echo_service.bal'
[ballerina/http] started HTTPS/WSS endpoint 0.0.0.0:9095
[ballerina/http] started HTTP/WS endpoint 0.0.0.0:9090

If the runtime detects any encrypted values in the configurations provided to it, it will prompt the user to enter the secret (the one used when encrypting the values). Providing an incorrect secret will result in a panic.

All the encrypted config values used when running a program should be encrypted using the same secret.

Alternatively, you can place the secret in a file and point to it using the -e b7a.config.secret flag. When this flag is set, the runtime will read the secret from the file and delete the file. The user is not prompted to enter the secret in this case. This is the more practical way of providing the secret to the config API.

Let’s try this out. Create a file (say, secret.txt) and place the secret in it (1234 in this case).

$ ls
echo_service.bal  secret.txt  secure-echo.conf

Now run the program with the b7a.config.secret flag set.

$ ballerina run -c secure-echo.conf -e b7a.config.secret=secret.txt echo_service.bal
Initiating service(s) in 'echo_service.bal'
[ballerina/http] started HTTPS/WSS endpoint 0.0.0.0:9095
[ballerina/http] started HTTP/WS endpoint 0.0.0.0:9090

As it can be seen, the user is not prompted. Now if we run ls again, we can see that the secret.txt file is deleted.

$ ls
echo_service.bal  secure-echo.conf

From point of view of the code, there’s no distinction between how normal configurations and encrypted configurations are accessed: you retrieve it using the key. Internally, encrypted values are stored in the registry in its encrypted form while plain text values are stored in the registry as-is. The encrypted configs are decrypted on-demand using the secret provided at the start of the runtime.

Reading Configurations from the Environment

A common requirement is for to provide the configurations to the program through environment variables. Therefore, let’s take a look at that as well. There are two cases to consider:

Reading a configuration from the environment
Overriding a configuration in the config file using an environment variable

In both cases, we just have to set the environment variable and can then simply retrieve it using the key. However in the current implementation there’s a slight difference in behaviour between case 1 and 2 . In the first case, it will simply look up the environment variable and return its current value. In the second case, when resolving the configurations, it will look up the key in environment variables and if there is such a variable, will store its value in the config registry. So such overridden configs will always have the values of the respective environment variables at the start up of the runtime.

Let’s take an example using a modified version of the config file we considered earlier. We will be overriding the httpPort config and set the httpsPort config using environment variables.

To set the environment variables in Linux:

$ export echo_httpPort=8080
$ export echo_httpsPort=8085

Note how the fully qualified key of the configuration is used when setting the variable with the period replaced by _. Now let’s run this and verify that the configurations are correctly applied.

$ ballerina run -c echo2.conf echo_service.bal 
Initiating service(s) in 'echo_service.bal'
[ballerina/http] started HTTPS/WSS endpoint 0.0.0.0:8085
[ballerina/http] started HTTP/WS endpoint 0.0.0.0:8080

The listeners are now on ports 8080 and 8085 respectively, instead of 9090 and 9095.

So that’s about it on managing configurations in Ballerina. Give it a try and do let us know about your experience with it. Were there any pain points? Got any new ideas on how and what we can improve? Drop in an email to our dev group: https://groups.google.com/forum/#!forum/ballerina-dev or open an issue in our Github repo: https://github.com/ballerina-platform/ballerina-lang :)