DB Operator — Manage Databases for Kubernetes Like a Pro

Photo by Christian Holzinger on Unsplash

DB Operator is a Kubernetes Operator developed by Kloeckner-i for the management of cloud databases, primarily Google Cloud SQL(GCSQL). DB Operator allows for the creation of GCSQL databases using Kubernetes Custom Resources(CR). It enables us to deploy databases as a Kubernetes resources, while allowing us to avoid the pain of running our own stateful databases on Kubernetes.

Background

At Kloeckner-i we utilise an advanced CI/CD strategy, that involves creating an on demand environment for each feature branch. The Kubernetes resource manifests for these environments are stored in the applications source repository.

But running the database as a pod is not highly available, or production ready due to failing on the following points.

• Kubernetes persistent volumes cannot be easily resized.
• When a pod is rescheduled to another node, persistent storage has to be detached and reattached, this is sometimes a slow process.
• The pod has to be completely terminated in order to detach and reattach the persistent volume to another node otherwise pod creation will fail with a multi-attach error due to database volumes being ReadWriteOnce.
• It’s possible for a pod to end up stuck in pending mode due to disk being unavailable in a specific zone.

googleapi: Error 503: ZONE_RESOURCE_POOL_EXHAUSTED - The zone ‘projects/xxx/zones/us-east1-d’ does not have enough resources available to fulfill the request.

Given these failings we asked the question:

Can we just use cloud managed database instance but somehow works with life cycle of on demand environment?

From that question, Kloeckner-i’s DB Operator was born.

How it works

DB Operator can be installed via helm.

$ helm repo add kloeckneri https://kloeckner-i.github.io/db-operator/helm/
$ helm install --name my-release kloeckneri/db-operator 

This will deploy the DB Operator and the following Custom Resource Definitions(CRDs):

• DbInstance: defines target database server where the Operator creates database in. DbInstance resource in Kubernetes represents the actual database server.
• Database: defines a database that must exist in the DbInstance. For database creation in a DbInstance, the DbInstance must firstly have status true; The name of the DbInstance previously created is referred in the spec.instance of Database resource.

Once deployed, the DB Operator will watch for Database, and DbInstance Custom Resources.

db operator workflow

DbInstance

When a new DbInstance CR is created, and it is configured to use GCSQL, the DB Operator will create a new GCSQL instance via the Google Cloud Platform (GCP) API.

Here is an example of a GCSQL backed DbInstance:

apiVersion: kci.rocks/v1alpha1
kind: DbInstance
metadata:
  name: example-gsql
spec:
  adminSecretRef:
    Name: example-admin-secret
    Namespace: example
  configmap: example-gsql-config # contains GCSQL configuration
  engine: <postgres or mysql>
  google:
    instance: dboperator-example # GCSQL Instance name in GCP
    accessSecret: cloudsql-client-serviceaccount

With the provided example DbInstance, it will create a new GCSQL instance with the name dboperator-example in GCP.

$ gcloud sql instances list
NAME                 DATABASE_VERSION   TIER                STATUS
dboperator-example   POSTGRES_11        db-g1-small         RUNNABLE

Database

When a Database CR is created, the operator creates the actual database in the instance.

Here is an example of a Database:

apiVersion: "kci.rocks/v1alpha1"
kind: "Database"
metadata:
  name: "example-db"
spec:
  secretName: example-db-credentials 
  instance: example-gsql
  deletionProtected: false
  backup:
    enable: true
    cron: "0 0 * * *"

With the example Database, it should create a database called [namespace]-example-db in the dboperator-example database instance. If the Database was created in default namespace:

$ gcloud sql databases list --instance dboperator-example
NAME                     CHARSET  COLLATION
postgres                 UTF8     en_US.UTF8
default-example-db       UTF8     en_US.UTF8

Connecting to the Database

After database is successfully created, the operator creates a ConfigMap and a Secret containing connection information such as host, user and password for accessing the database. Application Pods can connect to the database by using the ConfigMap and the Secret through envFrom or volumeMounts.

The following deployment is an example of how application Pods can connect to the database.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: example
  template:
    metadata:
      labels:
        app: example
    spec:
      containers:
      - name: example-app
        image: "app:latest"
        imagePullPolicy: Always
        env:
        - name: PGPASSFILE
          value: /run/secrets/postgres/POSTGRES_PASSWORD
        - name: PGUSER
          valueFrom:
            secretKeyRef:
              key: POSTGRES_USER
              name: example-db-credentials
        - name: PGDATABASE
          valueFrom:
            secretKeyRef:
              key: POSTGRES_DB
              name: example-db-credentials
        - name: PGHOST
          valueFrom:
            configMapKeyRef:
              key: DB_CONN
              name: example-db-credentials
        volumeMounts:
        - mountPath: /run/secrets/postgres/
          name: db-secret
          readOnly: true
      volumes:
      - name: db-secret
        secret:
          defaultMode: 420
          secretName: example-db-credentials

For further documentation on how the DB Operator works, and how to use the Custom Resources please refer to our Github project.

Writing Kubernetes Operator using Go Operator SDK

Back in 2018, when the DB Operator was conceived, not many companies were developing operators themselves. Although, our DevOps team had already experience of developing Stanley Operator which is our deployment tool and it was written in Python from the scratch. But writing operator from the scratch from my side was hard without knowledge. I didn’t know where to start. This time we decided to try something new which was Go Operator Framework SDK.

Benefits

We found the operator SDK was the right choice for our team. It allowed us to get off to an easy start, within approximately ten minutes you can get a simple operator running on Minikube without any prior knowledge of the Go language.

Challenges

We started out using a very early version of the operator SDK (v0.0.1). When it came to upgrading the operator SDK, before open sourcing, we found that the updated version was not backwards compatible and there were many large changes. There was big change which was that instead of Handle() function in newer code reconcile code was defined in the Reconcile() method of a controller’s Reconciler. Simply upgrading didn’t work because it broke all existing code structure, we had to follow migration guide and it was more or less same effort of re-writing operator completely.

Final words

1. If you can’t find a Kubernetes tool which is suitable for your needs, don’t dismiss the possibility of writing your own. Existing tools such as Operator Framework, KUDO, kubebuilder, and Metacontroller significantly lower the barrier to entry.
2. If you want to be able to create databases using Kubernetes Custom Resource’s, use the DB Operator. Currently the DB Operator supports GCSQL or generic ip and port accessible instances for the backend server. If you wish to use the DB Operator with other managed database providers, the DB Operator can easily be extended.