Version Control of Configuration Files Using Kubernetes

The Billing API SDK team in Netcracker develops tools for the Cloud Billing product. We deal with version control, API lifecycle, REST/gRPC server configuration and start, authentication and authorization, audit, tracing, and many other things in Go language.

Let’s talk about the configuration files!

If your applications have configuration files, following situation must be familiar to you: you develop an application, and then you create a configuration file and document it. After a while, you need to add some settings as the old ones do not meet all the requirements and, in general, it is better to change the structure.

What to do? If you do not change the configuration format, over time, it will turn into a bunch of things that “we need to deal with for historical reasons”. And if you change it… In this case, you always need to check if the configuration files are compatible with the product version you are installing for the customer. The operation team, customers, and many others will not really like this.

These problems can be solved by multi-version configurations. Borrowing them from the Kubernetes, we have developed and applied them. Now let’s discuss how it works.

A few words about the hub-spoke model

The hub-spoke model was initially an architectural pattern of the network topology, which referred not only to a computer network but also, for example, to a transport one. However, the pattern has been also put to good use in objects versioning. Thus, Kubernetes uses it for versioning of its APIs and resources.

The hub-spoke model assumes that there is a “hub” version of some object, which cannot be directly used by users, but it is related to other spoke versions. The hub version is the current version used in our code. When the configuration structure is changed, a new “spoke” version is separated from the hub. This version is identical to the hub version. To ensure the compatibility of the versions, we must be able to convert any Spoke version to the Hub version and vice versa.

Thus, there is no more worrying about configuration version that users of our service utilize. All you need to do is:

· define config version;

· convert config from “spoke” to “hub” version;

· use “hub” version of config in our code.

Thus, our Cloud Billing components can seamlessly upgrade config versions inside without updating all customer configuration files to new versions.

A nice bonus: all fixes made in new version are automatically added for users of the old config versions.

Adding a touch of versioning to config

Let’s say that our application has a configuration file describing roles and resources available for each of them:

roles:
- name: "account_ro"
rules:
- verbs: [ "Get", "List" ]
apiGroups: [ "billing.netcracker.com" ]
resources: [ "customer", "account", "address" ]
- name: "account_rw"
rules:
- verbs: [ "Get", "List", "Create", "Update" ]
apiGroups: [ "billing.netcracker.com" ]
resources: [ "customer", "account", "address" ]
- name: "dev"
rules:
- verbs: [ "Get", "List" ]
apiGroups: [ "billing.netcracker.com" ]
resources: [ "*" ]

To generate conversion functions between versions of this config, we will use the Kubernetes apimachinery library and conversion-gen, deepcopy-gen and defaulter-gen Go code generators from kubernetes/code-generator.

In order for the Kubernetes tools to identify our config, convert it to the Kubernetes-like view:

apiVersion: config.billing.netcracker.com/v1alpha1
kind: RolesConfig
spec:
roles:
- name: "account_ro"
rules:
- verbs: [ "Get", "List" ]
apiGroups: [ "billing.netcracker.com" ]
resources: [ "customer", "account", "address" ]
- name: "account_rw"
rules:
- verbs: [ "Get", "List", "Create", "Update" ]
apiGroups: [ "billing.netcracker.com" ]
resources: [ "customer", "account", "address" ]
- name: "dev"
rules:
- verbs: [ "Get", "List" ]
apiGroups: [ "billing.netcracker.com" ]
resources: [ "*" ]

Where

· apiVersion identifies the config version. It consists of the API group (`config.billing.netcracker.com`) and spoke version name (`v1alpha1`);

· kind is a config identifier;

· spec contains config content itself.

To work with this configuration in code, let’s add Go structures that will store config data. For this, we’ll create two packages for hub and spoke versions in our project. It looks like this:

├── hub

├── v1alpha1

Note: some files will be identical (except for the package name) in both hub and spoke packages. In this case, I will give example of only one file.

Add types.go file to the hub package. The file should contain:

package hub
import meta "k8s.io/apimachinery/pkg/apis/meta/v1"
// RolesConfig contains a list of roles for authorization
type RolesConfig struct {
// TypeMeta describes an individual object in an API response or request
// with strings representing the type of the object and its API schema version.
// Structures that are versioned or persisted should inline TypeMeta.
meta.TypeMeta `json:",inline"`
// ObjectMeta is metadata that all persisted resources must have,
meta.ObjectMeta `json:"metadata,omitempty"`
// RolesSpec is a list of existing RBAC Roles.
RolesSpec RolesSpec `json:"spec,omitempty"`
)
type RolesSpec struct {
Roles []Role `json:"roles,omitempty"`
)
// Role contains rules that represent a set of permissions.
type Role struct {
// Name is unique role name.
// +kubebuilder:validation:Required
Name string `json:"name,omitempty"`
// IsAnonymous identify that role is allowed for anonymous user - user.
// +kubebuilder:validation:Optional
IsAnonymous bool `json:"anonymous"`
// Rules is set of rules available for the role.
// +kubebuilder:validation:Required
Rules RuleSet `json:"rules"`
)
// RuleSet contains set of rules for a role
type RuleSet []Rule
// Rule is the list of actions the subject is allowed to perform on resources.
type Rule struct {
// +kubebuilder:validation:Required
Verbs []string `json:"verbs,omitempty"`
// +kubebuilder:validation:Required
Groups []string `json:"apiGroups,omitempty"`
// +kubebuilder:validation:Required
Kinds []string `json:"resources,omitempty"`
)

Similarly, add types.go to the v1alpha1 package. Since we have only one configuration version so far, hub and v1alpha1 are identical.

Add doc.go with config metadata to the hub package:

// +k8s:deepcopy-gen=package,register
// +groupName=config.billing.netcracker.com
package hub
const (
Version = "hub"
Group   = "config.billing.netcracker.com"
Kind    = "RolesConfig"
)

The `+k8s:deepcopy-gen` comment specifies settings for the deepcopy-gen plugin.

· package — this plugin generates the DeepCopy() method for all types in types.go

· register — registers generated methods in the scheme (more on that below).

Same file is required in the v1alpha1 package:

// +k8s:deepcopy-gen=package,register
// +k8s:conversion-gen=nrm.netcracker.cloud//billing-api-sdk/pkg/sdk/security/authorization/hub
// +groupName=config.billing.netcracker.com
package v1alpha1
const (
Version = "v1alpha1"
Group   = "config.billing.netcracker.com"
Kind    = "RolesConfig"
)

In `+k8s:conversion-gen`, specify path to the hub package.

Kubernetes uses concept of a scheme to generate conversion functions between different versions. Go structures of each version as well as the SetDefault and DeepCopy features are registered in it. Add register.go file to the hub and v1alpha1 to register newly added versions:

// +k8s:deepcopy-gen=package
// +groupName=config.billing.netcracker.com
package v1alpha1
import (
meta "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/schema"
"sigs.k8s.io/controller-runtime/pkg/scheme"
)
var (
// SchemeGroupVersion is group version used to register these objects
SchemeGroupVersion = schema.GroupVersion{Group: Group, Version: Version}
// SchemeBuilder is used to add go types to the GroupVersionKind scheme
SchemeBuilder = &scheme.Builder{GroupVersion: SchemeGroupVersion}
localSchemeBuilder = runtime.NewSchemeBuilder()
)
func init() {
SchemeBuilder.SchemeBuilder.Register(func(s *runtime.Scheme) error {
s.AddKnownTypeWithName(SchemeBuilder.GroupVersion.WithKind(Kind), &RolesConfig{})
meta.AddToGroupVersion(s, SchemeBuilder.GroupVersion)
return RegisterConversions(s)
})
)

After all previous steps, you must get following file structure:

├── hub
│   ├── doc.go
│   ├── register.go
│   ├── types.go
├── v1alpha1
│   ├── doc.go
│   ├── register.go
│   ├── types.go

Now, let’s use Kubernetes plugins and generate the conversion functions between the versions. First, install the following plugins:

· “k8s.io/code-generator/cmd/conversion-gen”

· “k8s.io/code-generator/cmd/deepcopy-gen”

· “k8s.io/code-generator/cmd/defaulter-gen”

You can install each plugin using the go install command. Then, call each plugin passing paths to the hub and v1alpha1 versions at the input:

conversion-gen --input-dirs $(PATH_TO_SPOKE_CONFIG) --output-package $(ROOT_PKG) --output-file-base zz_generated.conversion --output-base $(CURDIR) --go-header-file header.go.txt -v 1
deepcopy-gen --input-dirs $(PATH_TO_SPOKE_CONFIGS),$(PATH_TO_HUB_CONFIG) --output-package $(ROOT_PKG) --output-file-base zz_generated.deepcopy --output-base $(CURDIR)       --go-header-file header.go.txt -v 1
defaulter-gen --input-dirs $(PATH_TO_SPOKE_CONFIGS) --output-package $(ROOT_PKG) --output-file-base zz_generated.default --output-base $(CURDIR) --go-header-file  header.go.txt -v 1

Plugin work should result in following file structure:

├── hub
│   ├── doc.go
│   ├── register.go
│   ├── types.go
│   ├── zz_generated.deepcopy.go
│   ├── zz_generated.default.go
├── v1alpha1
│   ├── doc.go
│   ├── register.go
│   ├── types.go
│   ├── zz_generated.conversion.go
│   ├── zz_generated.deepcopy.go
│   ├── zz_generated.default.go

Great! Now we can convert our configuration from v1alpha1 to the hub version.

Providing an API to load configurations from file

Let’s provide our users with the possibility to load configurations from files into the RolesConfig object from hub. Name the file config.go and place it next to hub and v1alpha1 packages. It is not required to do a separate loading function for each type. Since all configuration types are registered in the global scheme and implement runtime.Object interface, you can write only one loading function that will accept a config object and fill it from a file for all config types.

// getConfigScheme returns a new instance of runtime.Schema with registered authorization config.
func getConfigScheme() (*runtime.Scheme, error) {
scheme := runtime.NewScheme()
err := hub.SchemeBuilder.AddToScheme(scheme)
if err != nil {
return nil, err
)
err = v1alpha1.SchemeBuilder.AddToScheme(scheme)
if err != nil {
return nil, err
)
return scheme, nil
)
// LoadFromFile reads roles configuration from provided config file and converts it to hub representation.
func LoadFromFile(filePath string) (*hub.RolesConfig, error) {
// get conversion schema
scheme, err := getConfigScheme()
if err != nil {
return nil, err
)
// read file content
data, err := os.ReadFile(filePath)
if err != nil {
return nil, fmt.Errorf("unable to read roles configuration: %w", err)
)
// unmarshal it to meta.TypeMeta to get metadata
typeMeta := &meta.TypeMeta{}
err = yaml.Unmarshal(data, typeMeta)
if err != nil {
return nil, err
)
// validate that configuration in file has the same kind
if typeMeta.Kind != hub.Kind {
return nil, fmt.Errorf("unable to read roles configuration: invalid config kind '%s', only '%s' supported", typeMeta.Kind, hub.Kind)
)
// create new spoke with same version as in config
storedObject, err := scheme.New(typeMeta.GroupVersionKind())
if err != nil {
return nil, err
)
// unmarshall config to spoke version
err = yaml.Unmarshal(data, storedObject)
if err != nil {
return nil, err
)
// set defaults for spoke. If you added defaults.
scheme.Default(storedObject)
// convert spoke to hub
hubObj := &hub.RolesConfig{}
err = scheme.Convert(storedObject, hubObj, nil)
if err != nil {
return nil, err
)
// set default for hub, if required
scheme.Default(hubObj)
return hubObj, nil
)

We need getConfigScheme() method to get Kubernetes scheme, in which all configuration versions are registered. Having received this scheme, we can determine version from the file passed to LoadFromFile(). Then, create an instance of the RolesConfig object of necessary version and unmarshal the file content to this object. Using the scheme, convert this object to the object hub version, set default values and return as an output.

It is not required to create a new scheme to load one type of config. You can have a global scheme where you will register all application configs, or you can pass the scheme directly to the loading method.

Adding new spoke version

If you need to change the configuration structure, do it in the hub version. After making changes, you need to:

· Create a package for the new spoke version (for example, v1alpha2) and copy updated types.go from the hub to it;

· Add doc.go, register.go and default.go, similarly to v1alpha1;

Call the conversion-gen, deepcopy-gen, and defaulter-gen plugins for all three versions (hub, v1alpha1, v1alpha2).

Major changes: what to do?

What if some major changes were made to the configuration and the conversion-gen plugin cannot automatically convert that configuration between the versions? You may actually face a situation like this… In this case, you need to implement the conversion function and place it in package of the spoke version, with which you are having problems.

Name the file zz_generated.manual.go to distinguish it from the rest of the generated files. The code will be as follows:

package v1alpha1
import (
hub "billing-api-sdk.git/pkg/sdk/db/hub"
conversion "k8s.io/apimachinery/pkg/conversion"
)
func Convert_hub_RolesSpec_To_v1alpha1_RolesSpec(in *hub.RolesSpec, out *RolesSpec, s conversion.Scope) error {
// some custom conversion logic here...
)

Other possible improvements

To simplify our work with versioned configs, we can do the following:

Adding ability to simultaneously load several configurations from a directory and merge them into one

Configurations often have different sources of settings. Some of them come with the product, others are customized, some are always defaulted and others need to be changed for every environment. If you load several configuration fragments at once and merge them, configuration parts can be divided into default, environment-specific, and other types, depending on the selected criteria.

For this, we need to clearly define the criteria based on which we can merge the configuration parts. Kubernetes uses this merge strategy. It is important to have same merge rules for all configs, as people who fill the configuration often do not know all merge aspects. Therefore, we simply recommend merging object lists by the Name field and returning an error if there are conflicts.

Next, you can find all files with the yaml/yml extension in the directory, find all files of the appropriate kind among them, unmarshal them to their spoke versions, merge them, and then convert to hub version.

Adding ability to load multiple configurations with conditions

Kubernetes provides us with ability to use selectors for working with objects, which improves loading from a directory. Using a selector, you can add different labels to configuration files and load all configurations with provided label. Thus, you can store multiple configurations for different environments in one directory and quickly change the profiles.

Adding configuration validation through CRD

Since structure of versioned configurations corresponds to CustomResource from Kubernetes, we can generate CustomResourceDefinition and validate configuration content using it. See the detailed information about the validations using CRD here.

Making a centralized service for configuration

Since a configuration is in fact a public API, it is reasonable to make it centralized. By registering all configuration types and versions in one scheme, you can also generate documentation as well as find delta.

Conclusions

It is impossible to build an ideal system that requires no improvements. And configs are no exception.

Using the hub-spoke versioning model allows you to seamlessly make changes to configuration files, ensuring backward and forward compatibility. This promotes quick adaptation to environment requirements without confusing different versions and making the configuration body too complicated.

Backward and forward compatibility of configuration makes life easier!