CodeX
Published in

CodeX

Using Kustomize and Confd Notes

This post was written to collect notes from different sources.

“Kustomize[1] is a standalone tool to customize Kubernetes objects through a kustomization file. Since 1.14, Kubectl also supports the management of Kubernetes objects using a kustomization file. To view Resources found in a directory containing a kustomization file, run the following command”[0]:

To apply those Resources, run kubectl apply with — kustomize or -k flag:

Kustomize is used for declarative object configuration.

What are imperative and declarative object configurations?

Imperative object configuration

In imperative object configuration, the kubectl command specifies the operation (create, replace, etc.), optional flags, and at least one file name. The file specified must contain a full definition of the object in YAML or JSON format.[2]

Examples

Create the objects defined in a configuration file:

Delete the objects defined in two configuration files:

Trade-offs

Advantages compared to imperative commands:

Object configuration can be stored in a source control system such as Git.

Object configuration can integrate with processes such as reviewing changes before push and audit trails.

Object configuration provides a template for creating new objects.

Disadvantages compared to imperative commands:

Object configuration requires a basic understanding of the object schema.

Object configuration requires the additional step of writing a YAML file.

Advantages compared to declarative object configuration:

Imperative object configuration behavior is simpler and easier to understand.

As of Kubernetes version 1.5, imperative object configuration is more mature.

Disadvantages compared to declarative object configuration:

Imperative object configuration works best on files, not directories.

Updates to live objects must be reflected in configuration files, or they will be lost during the next replacement.

Declarative object configuration

When using declarative object configuration, a user operates on object configuration files stored locally, however, the user does not define the operations to be taken on the files. Create, update, and delete operations are automatically detected per object by kubectl. This enables working on directories, where different operations might be needed for different objects.

Trade-offs

Advantages compared to imperative object configuration:

Changes made directly to live objects are retained, even if they are not merged back into the configuration files.

Declarative object configuration has better support for operating on directories and automatically detecting operation types (create, patch, delete) per-object.

Disadvantages compared to imperative object configuration:

Declarative object configuration is harder to debug and understand results when they are unexpected.

Partial updates using diffs create complex merge and patch operations.

Kustomize Usage

  • Generating resources from other sources
  • Setting cross-cutting fields for resources
  • Composing and customizing collections of resources

Generating Resources

“ConfigMaps and Secrets hold configuration or sensitive data that are used by other Kubernetes objects, such as Pods. The source of truth of ConfigMaps or Secrets are usually external to a cluster, such as a .properties file or an SSH keyfile. Kustomize has secretGenerator and configMapGenerator, which generate Secret and ConfigMap from files or literals.[0]”

configMapGenerator

“To generate a ConfigMap from a file, add an entry to the files list in configMapGenerator[0]”

This is an example deployment that uses a generated ConfigMap:

Kustomize generates ConfigMap and ConfigMap is mapped to a volume.

secretGenerator can be found at [0].

Setting cross-cutting fields

It is quite common to set cross-cutting fields for all Kubernetes resources in a project. Some use cases for setting cross-cutting fields:

  • adding the same name prefix or suffix
  • Adding the same set of labels
  • Adding the same set of annotations
  • Setting the same namespace for all Resources

Here is an example:

Composing and Customizing Resources

Composing

Kustomize supports composition of different resources. The resources field, in the kustomization.yaml file, defines the list of resources to include in a configuration. Set the path to a resource’s configuration file in the resources list. Here is an example of an NGINX application comprised of a Deployment and a Service:

The Resources from kubectl kustomize ./ contain both the Deployment and the Service objects.

Customizing

Patches can be used to apply different customizations to Resources. Kustomize supports different patching mechanisms through patchesStrategicMerge and patchesJson6902. patchesStrategicMerge is a list of file paths. Each file should be resolved to a strategic merge patch. The names inside the patches must match Resource names that are already loaded. Small patches that do one thing are recommended. For example, create one patch for increasing the deployment replica number and another patch for setting the memory limit.

Run kubectl kustomize ./ to view the Deployment:

CONFD

confd is a lightweight configuration management tool focused on:

keeping local configuration files up-to-date using data stored in etcd, consul, dynamodb, redis, vault, zookeeper, aws ssm parameter store or env vars and processing template resources. reloading applications to pick up new config file changes[3].

AWS SSM

AWS Systems Manager allows you to centralize operational data from multiple AWS services and automate tasks across your AWS resources. You can create logical groups of resources such as applications, different layers of an application stack, or production versus development environments. With Systems Manager, you can select a resource group and view its recent API activity, resource configuration changes, related notifications, operational alerts, software inventory, and patch compliance status. You can also take action on each resource group depending on your operational needs. Systems Manager provides a central place to view and manage your AWS resources, so you can have complete visibility and control over your operations.[4]

Confd is related with configuration management. That’s why I am interested in SSM parameter store.

SSM Parameter Store

AWS Systems Manager provides a centralized store to manage your configuration data, whether plain-text data such as database strings or secrets such as passwords. This allows you to separate your secrets and configuration data from your code. Parameters can be tagged and organized into hierarchies, helping you manage parameters more easily. For example, you can use the same parameter name, “db-string”, with a different hierarchical path, “dev/db-string” or “prod/db-string”, to store different values. Systems Manager is integrated with AWS Key Management Service (KMS), allowing you to automatically encrypt the data you store. You can also control user and resource access to parameters using AWS Identity and Access Management (IAM). Parameters can be referenced through other AWS services, such as Amazon Elastic Container Service, AWS Lambda, and AWS CloudFormation.[4]

One such service is SSM Parameter Store which is a secured and managed key/value store perfect for storing parameters, secrets, and configuration information. However, in April of 2018, AWS also introduced another service called AWS Secrets Manager that offers similar functionality. Given that both services kind of do the same thing, which to choose isn’t clear. With that in mind, let us take a look at the similarities and differences of these two services to better understand which service will best fit your architectural needs.[5]

SSM Parameter Store is free, that’s why I prefer it instead of AWS Secrets Manager.

Above commands put two parameter to SSM Parameter Store

The confdir is where template resource configs and source templates are stored.

Template resources are defined in TOML config files under the confdir.

/etc/confd/conf.d/myconfig.toml

Source templates are Golang text templates.

/etc/confd/templates/myconfig.conf.tmpl

0-) https://github.com/kelseyhightower/confd

1-) https://aws.amazon.com/systems-manager/features/

2-) https://www.1strategy.com/blog/2019/02/28/aws-parameter-store-vs-aws-secrets-manager/

3-) https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/

4-) https://github.com/kubernetes-sigs/kustomize

5-) https://kubernetes.io/docs/concepts/overview/working-with-objects/object-management/

--

--

Everything connected with Tech & Code. Follow to join our 1M+ monthly readers

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Yunus Kılıç

I have 9 years of experience in high-quality software application development, implementation, and integration.