Cassandra in Kubernetes

Headless Services, KubeDNS, Init Containers, Lifecycle hooks and other K8s concepts on the way

There is pretty good and standard documentation regarding deploying Cassandra in K8s as a Stateful Set. This is almost sufficient for us. Let’s take a little dive when we do the setup.

Example: Deploying Cassandra with Stateful Sets

Why StatefulSet and not a Deployments or others?

1. We want each pod to have its own Persistent Volume. Though Deployments are for generally stateless applications, it can also have a PV; but that is then shared across all the replicas/pods. That is one PV for all the pods. Not what we need.

2. We need each pod to have a unique Network IP. This is because each Cassandra server sets up a ring topology network connection with other servers. That is each pod needs to know the unique network IP of the other pods, as each server is a pod in Kubernetes. This way even if a pod is restarted, the other nodes can react to the predictable topology change.

These are the two key features that a Stateful Set offers

From the blog https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/

StatefulSets are valuable for applications that require one or more of the following. Stable, unique network identifiers. Stable, persistent storage.

One requirement that is needed by a StatefulSet is to create a Headless Service

StatefulSets currently require a Headless Service to be responsible for the network identity of the Pods. You are responsible for creating this Service.

Please follow these gist for the below tutorial https://gist.github.com/alexcpn/6a919b500211b5c253a4e28010325a1d#file-cassandra_green_statefulset-yaml

https://gist.github.com/alexcpn/6a919b500211b5c253a4e28010325a1d#file-cassandra_headless_service-yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: cassandra
  name: cassandra
  namespace: green
spec:
  clusterIP: None
  ports:
  - port: 9042
  selector:
    app: cassandra
https://gist.github.com/alexcpn/6a919b500211b5c253a4e28010325a1d#file-cassandra_green_statefulset-yaml
kind: StatefulSet
metadata:
  name: cassandra
  namespace: green
  labels:
    app: cassandra
spec:
  serviceName: cassandra
  replicas: 3
  selector:
    matchLabels:
      app: cassandra

Note the two highlighted parts above. ClusterIP is given as None for this service.

From official site — https://kubernetes.io/docs/concepts/services-networking/service/#headless-services

For headless Services, a cluster IP is not allocated, kube-proxy does not handle these Services, and there is no load balancing or proxying done by the platform for them. How DNS is automatically configured depends on whether the Service has selectors defined

We have used the selector field to use the DNS allocation per pod.

From the official docs https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/

For headless Services that define selectors, the endpoints controller creates Endpoints records in the API, and modifies the DNS configuration to return records (addresses) that point directly to the Pods backing the Service.

For a headless service, this resolves to multiple answers, one for each pod that is backing the service, and contains the port number and the domain name of the pod of the form auto-generated-name.my-svc.my-namespace.svc.cluster-domain.example.

Let us see the DNS records of the pods created by the StatefulSet cassandra-0, cassandra-1, cassandra-2.

It will be of form <podname>.headless service name.namespace.cluster domain. Example cassandra-0/1/2.cassandra.green.svc.cluster.local

If we exec into a pod say cassandra-0 and ping, it will reach the IP of the pod. Basically each pod can contact the other nodes through KubeDNS.

Let us use the DNS query utility dig to check this. First, let's find the Service IP of KubeDNS (coredns) in our cluster

[root@green--1 ~]# kubectl get svc  --namespace=kube-system
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   21d

and use that DNS server to query ‘cassandra-0.cassandra.green.svc.cluster.local’ using the KubeDNS.

[root@green--1 ~]# dig  @10.96.0.10   cassandra-0.cassandra.green.svc.cluster.local
; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> @10.96.0.10 cassandra-0.cassandra.green.svc.cluster.local
; (1 server found)
;; global options: +cmd
;; Got answer:
...
;; QUESTION SECTION:
;cassandra-0.cassandra.green.svc.cluster.local. IN A
;; ANSWER SECTION:
cassandra-0.cassandra.green.svc.cluster.local. 30 IN A 10.244.4.70
....

This gives the pod IP.

root@green--1 ~]# k get pods -o wide -n green
NAME          READY   STATUS    RESTARTS   AGE   IP    NODE       
cassandra-0   1/1     Running   1  7d18h   10.244.4.70   green--5   
cassandra-1   1/1     Running   1  7d18h   10.244.3.29   green--4   
cassandra-2   1/1     Running   3  7d18h   10.244.5.25   green--6

In each Cassandra pod, the KubeDNS IP will be configured as default in /etc/resolv.conf.

[root@green--1 ~]# kubectl -n green exec -it cassandra-0 -- /bin/bash
root@cassandra-0:/# cat /etc/resolv.conf 
nameserver 10.96.0.10

And so each Cassandra pod will use KubeDNS to interface to the other and form a ring. Note that the Cassandra Docker Image takes the initial seed as an environment variable. All pods are able to reach this and connect and then discover each other through Cassandra’s discovery mechanism.

https://gist.github.com/alexcpn/6a919b500211b5c253a4e28010325a1d#file-cassandra_green_statefulset-yaml
....
env:
  - name: MAX_HEAP_SIZE
 value: 1G
  - name: HEAP_NEWSIZE
 value: 100M
  - name: CASSANDRA_SEEDS
 value: "cassandra-0.cassandra.green.svc.cluster.local"
  - name: CASSANDRA_CLUSTER_NAME
 value: "green"

In regular Cassandra server, the seeds are configured in cassandra.yaml. But in Dockerised version, it is taken as an environment variable and the cassandra.yaml is updated at pod startup with the environment variable values.

Now we come to another requirement. Say that we need to make the Cassandra server have password-based authentication. This is just one line in cassandra.yaml.

Lets do this one step at a time. Let’s ignore for the time that this cassandra.yaml gets modified during pod startup. So we have a configuration file that needs some change and how do we load it to a pod. We can create a k8s configmap with the file contents and copy from the configmap to a file inside the pod in a directory.

kubectl create configmap cassandraconfig --from-file cassandra.yaml -ns green
-------and in out stateful set -----------------
https://gist.github.com/alexcpn/6a919b500211b5c253a4e28010325a1d#file-cassandra_green_statefulset-yaml
.....
volumeMounts:
 
 - name: cassandraconfig
   mountPath: /etc/cassandra/cassandra.yaml
   subPath: cassandra.yaml
volumes: 
  - name: cassandraconfig
 configMap:
   name: cassandraconfig

There is one problem with this; the destination directory becomes read-only and it contains only the file that is copied. We don’t want this.

So we can copy to another folder and use an IntiContainer to copy from that folder the file to our destination folder; without overwriting the destination folder.

What is an Inti container?

Init containers are started before the pod is started and run to completion and can contain utilities or setup scripts not present in an app image. https://kubernetes.io/docs/concepts/workloads/pods/init-containers/

Let’s use this. We mount the configmap contents into a temporary directory and use the init container to copy the particular file from there to the target directory. This way the target directory contents do not get over-written.

initContainers:
       - name: copy
         image: busybox:1.28
         command: ["/bin/sh", "-c", "cp /config/cassandra.yaml  /etc/cassandra/"]
         volumeMounts:
         - name: cassandraconfig
           mountPath: /config/
      containers:
      - name: cassandra
        image: cassandra:3.0
        imagePullPolicy: Alway
volumeMounts:
        - name: cassandradata
          mountPath: /cassandra_data
- name: tmp-config
          mountPath: /config/cassandra/
volumes: 
      - name: cassandraconfig
        configMap:
          name: cassandraconfig
      - name: tmp-config
        emptyDir: {}

However, since our Cassandra docker image is regenerating the cassandra.yaml at pod startup, the above, which may work for other scenarios, don’t work here.

However we can use the Pod lifecycle hook PostStart to manipulate the settings what we want. I am just doing a crude append to end of the cassandra.yaml, but we could do this more elegantly via sed.

lifecycle:
  postStart:
 exec:
   command: 
   - /bin/sh
   - -c
   - "echo \"authenticator: PasswordAuthenticator\" >> /etc/cassandra/cassandra.yaml"
   
  preStop:
 exec:
   command: 
   - /bin/sh
   - -c
   - nodetool drain

Note — Since we are isolating each Cassandra instance by a namespace, it is trivial to spin up more test clusters, just by changing the namespace in the Stateful Set

------------------------------------------------
[root@green--1 ~]# k get pods -o wide -n green
NAME          READY   STATUS    RESTARTS   AGE     IP            NODE       
cassandra-0   1/1     Running   1          7d18h   10.244.4.70   green--5   
cassandra-1   1/1     Running   1          7d18h   10.244.3.29   green--4   
cassandra-2   1/1     Running   3          7d18h   10.244.5.25   green--6   
[root@green--1 ~]# k get pods -o wide -n blue
NAME          READY   STATUS    RESTARTS   AGE     IP            NODE       
cassandra-0   1/1     Running   0          4m40s   10.244.1.49   green--2   
cassandra-1   1/1     Running   0          3m54s   10.244.2.56   green--3   
cassandra-2   1/1     Running   1          3m22s   10.244.3.59   green--4
-----------------------------------------------
[root@green--1 ~]# kubectl -n green exec -it cassandra-0 -- /bin/bash
root@cassandra-0:/# nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address      Load       Tokens       Owns (effective)  
UN  10.244.4.70  202.93 KB  256          66.1%             
UN  10.244.3.29  260.3 KB   256          68.1%             
UN  10.244.5.25  276.92 KB  256          65.8%             
root@cassandra-0:/# exit

How to connect

User a docker container which has the latest CQLsh and use one of the IP from the above nodetool status command to connect to the CQL of Cassandra

Use the below to connect
docker run -it --net=host cassandra  /bin/bash -
[root@green--1 ]#  docker run -it --net=host cassandra  /bin/bash
root@green--1:/# cqlsh -u cassandra -p cassandra 10.244.1.49
Connected to blue at 10.244.1.49:9042.
[cqlsh 5.0.1 | Cassandra 3.0.20 | CQL spec 3.4.0 | Native protocol v4]
Use HELP for help.
cassandra@cqlsh>

You can use the following examples to play around with https://docs.datastax.com/en/dse/6.7/cql/cql/examples/cyclist_id-rename.html

Reference

https://kubernetes.io/docs/tutorials/stateful-application/cassandra/#creating-a-cassandra-headless-service