Open in app

Sign in

Write

Sign in

Setting up Cassandra Multi-Site on Google Kubernetes Engine with Casskop

Alexandre Guitton
6 min readJan 15, 2020

--

Casskop is a Kubernetes operator that makes it easy to run Apache Cassandra on Kubernetes. It can create, configure, and manage Cassandra clusters running in Kubernetes.

Multi-Casskop is a controller that will be in charge of creating CassandraClusters CRD objects in several different Kubernetes clusters and in a manner that all Cassandra nodes will be part of the same ring.

In Previous posts, John Sanda, Cyril Scetbon and Sebastien Allamand already talked about Casskop and Multi-Casskop:

Pre-requisites

User should need :

  • terraform version v0.12.7+
  • kubectl version v1.13.3+
  • kubectx & kubens
  • Helm version v2.15.1+
  • gcloud sdk version 272.0.0+
  • A service account with enough rights (for this example : editor)
  • Having a DNS zone in google cloud dns.

Setup GCP environment

Multi site infrastructure

To setup the GCP environment we will use terraform provisionning, to instantiate the following infrastructure :

  • 2 GKE clusters :
  1. First on europe-west1-b which will be the master
  2. Second on europe-west1-c which will be the remote
  • Firewall rules to allow clusters to communicate
  • External DNS on each cluster to expose cassandra nodes
  • Casskop operator on each cluster to focus on multi-casskop usage

Environment setup

Start to set variables needed for the instantiation :

$ export CASSKOP_WORKSPACE=<path to cassandra-k8s-operateur project>
$ export PROJECT=<gcp project>
$ export SERVICE_ACCOUNT_KEY_PATH=<path to service account key>
$ export NAMESPACE=cassandra-demo
$ export DNS_ZONE_NAME=external-dns-test-gcp-trycatchlearn-fr     # -> change with your own one
$ export DNS_NAME=external-dns-test.gcp.trycatchlearn.fr          # -> change with your own one
$ export MANAGED_ZONE=tracking-pdb                                # -> change with your own one

Setup base infrastructure

$ cd ${CASSKOP_WORKSPACE}/multi-casskop/samples/gke/terraform
$ terraform init

Master provisionning

With the master provisionning, we will deploy firewall and Cloud dns configuration :

$ terraform workspace new master
$ terraform workspace select master
$ terraform apply \
    -var-file="env/master.tfvars" \
    -var="service_account_json_file=${SERVICE_ACCOUNT_KEY_PATH}" \
    -var="namespace=${NAMESPACE}" \
    -var="project=${PROJECT}" \
    -var="dns_zone_name=${DNS_ZONE_NAME}" \
    -var="dns_name=${DNS_NAME}" \
    -var="managed_zone=${MANAGED_ZONE}"

Remote provisionning

$ terraform workspace new remote
$ terraform workspace select remote
$ terraform apply \
    -var-file="env/slave.tfvars" \
    -var="service_account_json_file=${SERVICE_ACCOUNT_KEY_PATH}" \
    -var="namespace=${NAMESPACE}" \
    -var="project=${PROJECT}" \
    -var="dns_zone_name=${DNS_ZONE_NAME}" \
    -var="dns_name=${DNS_NAME}" \
    -var="managed_zone=${MANAGED_ZONE}"

Check installation

Check master configuration

Now we will check that everything is well deployed in the GKE master cluster :

$ gcloud container clusters get-credentials cassandra-europe-west1-b-master --zone europe-west1-b --project ${PROJECT}
$ kubectl get pods -n ${NAMESPACE}
NAME                                          READY   STATUS    RESTARTS   AGE
casskop-cassandra-operator-54c4cfcbcb-b4qxq   1/1     Running   0          4h9m
external-dns-6dd96c985-h76gh                  1/1     Running   0          4h16m

Check remote configuration

Now we will check that everything is well deployed in the GKE remote cluster :

$ gcloud container clusters get-credentials cassandra-europe-west1-c-slave --zone europe-west1-c --project ${PROJECT}
$ kubectl get pods -n ${NAMESPACE}
NAME                                          READY   STATUS    RESTARTS   AGE
casskop-cassandra-operator-54c4cfcbcb-sxjz7   1/1     Running   0          4m56s
external-dns-7f947c5b5b-mq7kg                 1/1     Running   0          5m46s

Check DNS zone configuration

Make a note of the nameservers that were assigned to your new zone :

$ gcloud dns record-sets list \
    --zone "${DNS_ZONE_NAME}" \
    --name "${DNS_NAME}." \
    --type NS
NAME                                     TYPE  TTL    DATA
external-dns-test.gcp.trycatchlearn.fr.  NS    21600  ns-cloud-e1.googledomains.com.,ns-cloud-e2.googledomains.com.,ns-cloud-e3.googledomains.com.,ns-cloud-e4.googledomains.com.

Check Firewall configuration

$ gcloud compute firewall-rules describe gke-cassandra-cluster
allowed:
- IPProtocol: udp
- IPProtocol: tcp
creationTimestamp: '2019-12-05T13:31:01.233-08:00'
description: ''
direction: INGRESS
disabled: false
id: '8270840333953452538'
kind: compute#firewall
logConfig:
  enable: false
name: gke-cassandra-cluster
network: https://www.googleapis.com/compute/v1/projects/poc-rtc/global/networks/default
priority: 1000
selfLink: https://www.googleapis.com/compute/v1/projects/poc-rtc/global/firewalls/gke-cassandra-cluster
sourceRanges:
- 0.0.0.0/0
targetTags:
- cassandra-cluster

Check Storage Class

$ kubectl get storageclasses.storage.k8s.io 
NAME                 PROVISIONER            AGE
standard (default)   kubernetes.io/gce-pd   28m
standard-wait        kubernetes.io/gce-pd   24m

Multi casskop deployment

Bootstrap API access to Remote from Master

Multi-Casskop will be deployed in master cluster, change your kubectl context to point this cluster.

In order to allow Multi-CassKop controller to have access to slave from master, we are going to use kubemcsa from admiralty to be able to export secret from remote to master.

Install kubemcsa :

$ export RELEASE_VERSION=v0.6.1
$ wget https://github.com/admiraltyio/multicluster-service-account/releases/download/${RELEASE_VERSION}/kubemcsa-linux-amd64
$ mkdir -p ~/tools/kubemcsa/${RELEASE_VERSION} && mv kubemcsa-linux-amd64 tools/kubemcsa/${RELEASE_VERSION}/kubemcsa
$ chmod +x ~/tools/kubemcsa/${RELEASE_VERSION}/kubemcsa
$ sudo ln -sfn  ~/tools/kubemcsa/${RELEASE_VERSION}/kubemcsa /usr/local/bin/kubemcsa

Generate secret for master :

$ kubectx # Switch context on master cluster
Switched to context "gke_<Project name>_europe-west1-b_cassandra-europe-west1-b-master".
$ kubens # Switch context on correct namespace
Context "gke_<Project name>_europe-west1-b_cassandra-europe-west1-b-master" modified.
Active namespace is "<Namespace>".
$ kubemcsa export --context=gke_poc-rtc_europe-west1-c_cassandra-europe-west1-c-slave --namespace ${NAMESPACE} cassandra-operator --as gke-slave-west1-c | kubectl apply -f -
secret/gke-slave-west1-c created

Check that the secret is correctly created

$ kubectl get secrets -n ${NAMESPACE}
...
gke-slave-west1-c                Opaque                                5      28s

Install Multi-CassKop

Add MultiCasskop crd on the remote cluster :

# Will be remove in version > 0.5.0$ kubectx # Switch context on remote cluster
Switched to context "gke_<Project name>_europe-west1-c_cassandra-europe-west1-c-slave".
$ kubectl apply -f https://raw.githubusercontent.com/Orange-OpenSource/casskop/master/multi-casskop/deploy/crds/multicluster_v1alpha1_cassandramulticluster_crd.yaml

Deployment with Helm :

$ kubectx # Switch context on master cluster
Switched to context "gke_<Project name>_europe-west1-b_cassandra-europe-west1-b-master".
$ helm init --client-only
$ helm repo add orange-incubator https://orange-charts-incubator.storage.googleapis.com
$ helm repo update
$ cd ${CASSKOP_WORKSPACE}
$ helm install --name multi-casskop orange-incubator/multi-casskop --set k8s.local=gke-master-west1-b --set k8s.remote={gke-slave-west1-c} --set image.tag=0.5.0-multi-cluster #--no-hooks if crd already install

Create the MultiCasskop CRD

Now we are ready to deploy a MultiCassKop CRD instance. We will use the example in multi-casskop/samples/gke/multi-casskop-gke.yaml :

$ kubectl apply -f multi-casskop/samples/gke/multi-casskop-gke.yaml

Check multi cluster installation

We can see that each cluster has the required pods :

$ kubectx # Switch context on master cluster
Switched to context "gke_<Project name>_europe-west1-b_cassandra-europe-west1-b-master".
$ kubectl get pods -n ${NAMESPACE}
NAME                                          READY   STATUS    RESTARTS   AGE
cassandra-demo-dc1-rack1-0                    1/1     Running   0          8m30s
casskop-cassandra-operator-54c4cfcbcb-8qncr   1/1     Running   0          34m
external-dns-6dd96c985-7jf6w                  1/1     Running   0          35m
multi-casskop-67dc74dff7-z4642                1/1     Running   0          11m
$ kubectx # Switch context on slave cluster
Switched to context "gke_<Project name>_europe-west1-c_cassandra-europe-west1-c-slave".
$ kubectl get pods -n ${NAMESPACE}
NAME                                          READY   STATUS    RESTARTS   AGE
cassandra-demo-dc3-rack3-0                    1/1     Running   0          6m55s
cassandra-demo-dc4-rack4-0                    1/1     Running   0          4m59s
cassandra-demo-dc4-rack4-1                    1/1     Running   0          3m20s
casskop-cassandra-operator-54c4cfcbcb-sxjz7   1/1     Running   0          71m
external-dns-7f947c5b5b-mq7kg                 1/1     Running   0          72m

If we go in one of the created pods, we can see that nodetool see pods of both clusters :

$ kubectx # Switch context on master cluster
Switched to context "gke_<Project name>_europe-west1-b_cassandra-europe-west1-b-master".
$ kubectl exec -ti cassandra-demo-dc1-rack1-0 nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.52.2.3  108.62 KiB  256          49.2%             a0958905-e1fa-4410-baca-fc86f4457f1a  rack1
Datacenter: dc3
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.8.3.2   74.95 KiB  256          51.5%             03f8eede-4b69-43be-a0c1-73f73470398b  rack3
Datacenter: dc4
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.8.4.3   107.87 KiB  256          47.8%             1a7432e2-4ca8-4767-acdb-3b40e6ff4a57  rack4
UN  10.8.2.5   107.85 KiB  256          51.6%             272037ce-4146-42c1-9079-ef4561249254  rack4

Clean up everything

If you have set the deleteCassandraCluster to true, then when deleting the MultiCassKop object, it will cascade the deletion of the CassandraCluster object in the targeted k8s clusters. Then each local CassKop will delete their Cassandra clusters (else skip this step)

$ kubectl delete multicasskops.db.orange.com multi-casskop-demo
$ helm del --purge multi-casskop

Cleaning remote cluster

$ cd ${CASSKOP_WORKSPACE}/multi-casskop/samples/gke/terraform
$ terraform workspace select remote
$ terraform destroy \
    -var-file="env/slave.tfvars" \
    -var="service_account_json_file=${SERVICE_ACCOUNT_KEY_PATH}" \
    -var="namespace=${NAMESPACE}" \
    -var="project=${PROJECT}" \
    -var="dns_zone_name=${DNS_ZONE_NAME}" \
    -var="dns_name=${DNS_NAME}" \
    -var="managed_zone=${MANAGED_ZONE}"

Cleaning master cluster

Before running the following command, you need to clean dns records set.

$ terraform workspace select master
$ terraform destroy \
    -var-file="env/master.tfvars" \
    -var="service_account_json_file=${SERVICE_ACCOUNT_KEY_PATH}" \
    -var="namespace=${NAMESPACE}" \
    -var="project=${PROJECT}" \
    -var="dns_zone_name=${DNS_ZONE_NAME}" \
    -var="dns_name=${DNS_NAME}" \
    -var="managed_zone=${MANAGED_ZONE}"
Docker
Cassandra
Kubernetes
Google Cloud Platform
Open Source

--

--

Alexandre Guitton

Written by Alexandre Guitton

140 Followers

Lead Data Ops @Doctolib

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams