Kubernetes configuration management using Kustomize

Published in

DevBulls

5 min readJul 31, 2024

This tutorial guides you through deploying a Apache web server to a Kubernetes cluster, showcasing how Kustomize overlays can be utilized to apply distinct configurations across development, staging, and production environments.

Prerequisites

A Kubernetes cluster
Kubectl installed and configured
Basic understanding of Kubernetes objects (Deployments, Services, ConfigMaps)

Overview

We will deploy an Apache web server that displays a message. This message will vary depending on the environment (development, staging, production) to demonstrate how Kustomize manages configurations.

Basic Concepts

Base

A base is a set of Kubernetes manifests that define the common configuration shared across environments. Typically located in a directory named base or similar.

Overlay

An overlay is a set of configurations that modify the base to suit a specific environment. Overlays can be found in directories like overlays/development, overlays/staging, and overlays/production.

To understand those concepts imagine you have a plain white cake, which represents your base Kubernetes configuration. This cake is the foundation that has all essential ingredients and structure. Now, you want to customize this cake for different occasions:

For a casual family gathering, you add a light layer of frosting and some simple sprinkles. This is like applying a development overlay with minor tweaks to your configuration.

For a friend’s birthday, you add more elaborate decorations, maybe some smooth icing and colorful decorations. This represents your staging overlay, with more substantial changes to test before going live.

For a grand wedding, you add intricate designs, multiple tiers, and elegant decorations. This is your production overlay, with the final, polished configurations ready for the big event.

Now knowing the basics lets jump into code.

Step 1: Base Configuration

Namespace

Create a Namespace object to isolate base environment. Later we will overwrite its name using overlay.

# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: base

ConfigMap

The ConfigMap stores the HTML content served by the Apache server.

# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  index.html: |
    <html>
    <head><title>App Message</title></head>
    <body><h1>Base message</h1></body>
    </html>

Deployment

Define the Apache web server Deployment.

# web-server-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web-server
          image: httpd
          ports:
            - containerPort: 80
          resources:
            limits:
              cpu: "0.5"
              memory: "512Mi"
          volumeMounts:
            - name: config-volume
              mountPath: /usr/local/apache2/htdocs
      volumes:
        - name: config-volume
          configMap:
            name: app-config

Service

Expose the web server within the cluster via a ClusterIP service.

# web-server-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: web-server
spec:
  type: ClusterIP
  selector:
    app: web
  ports:
    - protocol: TCP
      port: 80

Step 2: Creating Overlays for Different Environments

For each environment (development, staging, production), we will create a Kustomize overlay that customizes the index.html file to demonstrate deploying with environment-specific configurations. We will also patch the namespace name, to create new namespace for given environment.

Directory Structure

Arrange your files as follows, for clear separation of base and environment-specific configurations:

k8s/
├── base/
│   ├── configmap.yaml
│   ├── deployment.yaml
│   ├── kustomization.yaml
│   ├── namespace.yaml
│   └── service.yaml
└── overlays/
    ├── development/
    │   ├── kustomization.yaml
    │   └── index.html
    ├── staging/
    │   ├── kustomization.yaml
    │   └── index.html
    └── production/
        ├── kustomization.yaml
        └── index.html

Development Environment Example

Customize the development environment by specifying a unique kustomization.yaml and replacing the ConfigMap content, using configMapGenerator.

kustomization.yaml for Development:

# overlays/development/kustomization.yaml
namespace: development

resources:
  - ../../baseconfigMapGenerator:
  - name: app-config
    files:
      - index.html
    behavior: replacepatches:
  - target:
      kind: Namespace
      name: base
    patch: |-
      - op: replace
        path: /metadata/name
        value: development

Development index.html content:

<html>
  <body>
    <h1>Development Environment Message</h1>
  </body>
</html>

Step 3: Deploying to Your Cluster

Deploy the configuration using Kustomize:

kubectl apply -k overlays/development/

Repeat for staging and production by adapting their respective kustomization.yaml and configmap.yaml contents.

You can see how flexible Kustomize is. Once the base configuration is established, deploying to a new environment is a breeze. Just patch the values you want to be different for the specific environment, and that’s it.

Advanced Kustomize Features

Now that you’ve grasped the basics of Kustomize overlays, let’s explore some advanced features that can further enhance your Kubernetes configuration management. These advanced features will help you handle more complex scenarios and optimize your workflow.

Using Common Labels

Kustomize allows you to apply common labels across all resources, helping you organize and manage your resources more effectively.

Example: Adding common labels

# base/kustomization.yaml
commonLabels:
  app: web-server
  environment: base

resources:
  - namespace.yaml
  - configmap.yaml
  - web-server-deployment.yaml
  - web-server-service.yaml

ConfigMap and Secret Generators

Kustomize can generate ConfigMaps and Secrets from files, literals, or environment variables, ensuring sensitive information is managed securely.

Example: Generating a Secret from literals:

# overlays/production/kustomization.yaml
namespace: production

resources:
  - ../../basesecretGenerator:
  - name: web-server-secret
    literals:
      - username=admin
      - password=supersecretconfigMapGenerator:
  - name: app-config
    files:
      - index.html
    behavior: replacepatches:
  - target:
      kind: Namespace
      name: base
    patch: |-
      - op: replace
        path: /metadata/name
        value: production

Resource Order and Dependencies

Kustomize ensures resources are applied in the correct order by respecting dependencies, which is crucial for resources like CRDs and their instances.

Example: Ordering resources with dependencies:

# base/kustomization.yaml
resources:
  - crd.yaml
  - crd-instance.yaml
  - configmap.yaml
  - web-server-deployment.yaml
  - web-server-service.yaml

Replacements

Kustomize allows you to dynamically replace values across different fields within your configurations. This feature ensures consistency and maintainability by sourcing values from one resource and applying them to specified targets. By using replacements, you can manage and update your configurations more efficiently, reducing redundancy and the risk of errors.

Example:

In a Kustomize setup, you can replace placeholder values in a deployment with actual values from a ConfigMap, ensuring the environment variables are consistently updated across your Kubernetes resources. This approach streamlines configuration management across different environments.

# database.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: db-config
data:
  db-host: "database.local"
  db-name: "mydatabase"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-deployment
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: my-app
          image: my-app:latest
          env:
            - name: DB_HOST
              value: "PLACEHOLDER_DB_HOST"
            - name: DB_NAME
              value: "PLACEHOLDER_DB_NAME"

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - database.yaml
replacements:
  - source:
      kind: ConfigMap
      name: db-config
      fieldPath: data.db-host
    targets:
      - select:
          kind: Deployment
          name: app-deployment
        fieldPaths:
          - spec.template.spec.containers.[name=my-app].env.[name=DB_HOST].value
  - source:
      kind: ConfigMap
      name: db-config
      fieldPath: data.db-name
    targets:
      - select:
          kind: Deployment
          name: app-deployment
        fieldPaths:
          - spec.template.spec.containers.[name=my-app].env.[name=DB_NAME].value

Conclusion

This tutorial showcased Kustomize’s effectiveness in managing configurations for Kubernetes applications across multiple environments. By utilizing overlays, we demonstrated how deployments can be easily customized, highlighting Kustomize’s versatility in simplifying application management across development, staging, and production environments. With additional advanced techniques, you’re well-equipped to handle a variety of scenarios, making your Kubernetes configuration management more robust and maintainable.