Multi-Region CockroachDB Cluster on Kubernetes

Happy devSecOps

(λx.x)eranga

Published in

Effectz.AI

4 min readNov 3, 2021

Background

CockroachDB is a distributed SQL database built on a transactional and strongly-consistent key-value store. Main motivation of CockroachDB is to facilitate strong consistent and highly scalable distributed SQL database. Normally relational databases(e.g MySQL) offer strong consistency, but they don’t really scale horizontally. NoSQL databases(e.g Cassandra) can scale horizontally, but users need to deal with the issue of eventual consistency. CockroachDB sits on both sides. It scales horizontally and supports strongly-consistent ACID transactions(it’s CP based system on CAP theorem). Behind the scene CockroachDB using Raft consensus. Data replication(e.g storage consistency) and automatic repairs handles with Raft consensus. You can read more information about these theoretical concepts from CockroachDB paper.

CockroachDB was designed to be highly compatible with PostgreSQL(CockroachDB provides a familiar SQL API for structuring, manipulating, and querying data). This is an important point, and not just because it means that users don’t have to learn as many new things just to work with CockroachDB. This means that some applications written to work with PostgreSQL may be able to work with CockroachDB with no changes necessary. It also means that you can use a wide variety of existing PostgreSQL client drivers to talk to CockroachDB. Following are some main feature of CockroachDB.

In this post I’m gonna discuss about simulating multi-region CockroachDB cluster on local Kubernets environment(e.g Minikube). All the deployments which related to this post available in gitlab. Please clone the repo and continue the post.

Cluster Setup

Mainly there are three regions(us-west, us-east, eu-west) in this simulation. Each region has three CockroachDB node.

Minikube Setup

First I need to setup Minikube Kubernets cluster. I have discussed detailed information about setting-up Minikube-based local Kubernets cluster in my previous post. Please follow the post if you need more information about setting-up Minikube.

CockroachDB Services

I have defined three NodePort services for three different regions, so different nodes can communicate each other. Then defined ClusterIP service to expose internal services to outside the cluster. For an example we can access CockroachDB web dashboard etc from outside the cluster via this service. Following is the structure of the NodePort service(show only single service here) and ClusterIP service.

CockroachDB PVCs

I have created Kubernets PersistentVolumeClaim objects for different CockroachDB nodes(Pods) in the cluster. Altogether there are 9 PVCs. Following is the structure of a single PVC object.

CockroachDB Deployment

Then I have defined the Kubernets Pods of the CocoroachDB nodes. There are 9 CocoroachDB Pods in three different clusters. Each Pod linked with a PVC. Following is the structure of a single Pod.

CockroachDB Jobs

Next I have defined two Kubernets Jobs. First Job initiate the cluster. The second Job runs the SQL statements to set the geo coordinates of the regions, so the CockroachDB dashboard Map knows where to place them. Please note that, to deal with Map UI in the dashboard, you need to have CockroachDB enterprise license.

Distributed SQL Queries

I can connect to different nodes in the cluster and execute distributed SQL queries. I have connected to the nodes in different regions and executed following queries and tested the results.

CockroachDB Dashboard

CockroachDB provides a dashboard UI. We can view the information about the cluster and various matrixes via the CockroachDB dashboard(the dashboard runs on port 8080 by default). I can to connect to the dashboard via Minikube host in different cluster regions.