Kubernetes manifests with Kustomize in Multiple Environments

Photo Credits — Bruno Croci https://unsplash.com/photos/jwkOaqUZtuM

Let’s learn how to compose and deploy a Kubernetes application to Staging and Production environments with differing configuration using Kustomize.

Kustomize is a standalone tool to customize Kubernetes objects through a kustomization file.

We’ll start with a simple application deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

What if we want our stage and prod deployments to have different configurations? Such as:

• 1 vs 3 replicas
• unique ingress
• unique labels
• unique namespace

This is where we will leverage Kustomize to compose our multiple environment manifests.

Let’s setup our directory as such for my-web-app :

# https://github.com/dangreenlee/k8s-multi-env-kustomize-demo
└── apps
    └── my-web-app
        ├── base
        │   ├── deployment.yaml
        │   ├── kustomization.yaml
        │   └── service.yaml
        └── overlays
            └── environments
                ├── prod
                │   ├── deployment.yaml
                │   ├── ingress.yaml
                │   └── kustomization.yaml
                └── stage
                    ├── deployment.yaml
                    ├── ingress.yaml
                    └── kustomization.yaml

A base is a directory with a kustomization.yaml, which contains a set of resources and associated customization. An overlay is a directory with a kustomization.yaml that refers to other kustomization directories as its bases. A base has no knowledge of an overlay and can be used in multiple overlays. An overlay may have multiple bases and it composes all resources from bases and may also have customization on top of them. — K8s Docs

Our base layer consists of:

A deployment manifest.

# ~/apps/my-web-app/base/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-deployment
spec:
  template:
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

A service manifest.

# ~/apps/my-web-app/base/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: my-web-app-service
spec:
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

and Kustomization manifest.

# ~/apps/my-web-app/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml
  - service.yaml

Next, let’s take a look at the stage overlay manifests which consists of:

A unique replica count is specified in this deployment.

# ~/apps/my-web-app/stage/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-deployment
spec:
  replicas: 1

A unique ingress.

# ~/apps/my-web-app/stage/ingress.yaml
kind: Ingress
apiVersion: networking.k8s.io/v1
metadata:
  name: my-app
spec:
  rules:
    - host: stage-my-app.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-app
                port:
                  number: 80

And Kustomization file to compose the manifests, set unique labels, and namespace.

# ~/apps/my-web-app/stage/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namePrefix: stage-

commonLabels:
    app: my-app
    env: stage

bases:
  - ../../../base

resources:
  - ingress.yaml

patchesStrategicMerge:
  - deployment.yaml
  - ingress.yaml

namespace: stage

To build the stage manifests you would run:

kustomize build apps/my-web-app/overlays/environments/stage > stage.yaml 

Can you guess how the prod overlay will look?

Build it with:

kustomize build apps/my-web-app/overlays/environments/stage > stage.yaml

Check out the full demo GitHub repository here.

As I hope you have seen through this demonstration, Kustomize is a powerful tool that you can use to compose Kubernetes manifests and leverage the layers of control.