Part One: Deploy a Multi-Datacenter Apache Cassandra Cluster in Kubernetes
Author: Jeff Carpenter
--
This is the first post in a series that details how to create Cassandra clusters with different deployment topologies using K8ssandra.
If you’re looking to spin up a single Apache Cassandra® datacenter in a single Kubernetes (K8s) cluster, you’ll find plenty of examples on K8ssandra. However, there are many situations where you’d benefit from deploying a multi-center Cassandra cluster in K8s. For example, if you had a read-heavy analytics workload supporting your main application, you could isolate it by creating a secondary Cassandra datacenter.
In this post, we take a look at some of these situations and walk you through creating a Cassandra cluster with two data centers in a single cloud region, using separate Kubernetes namespaces to isolate workloads. But first, let’s start with a bit of background on how Cassandra enables different deployment patterns.
Flexible topologies with Cassandra
From its earliest days, Cassandra has included the ability to assign nodes to datacenters and racks.
- Rack: Originally conceived as mapping to a single rack of servers connected to shared resources, like power, network, and cooling.
- Datacenter: This could consist of multiple racks with physical separation.
These constructs allowed developers to create high-availability deployments by replicating data across different fault domains. It also ensured that Cassandra clusters stayed operational amid failures ranging from a single physical server, rack, to an entire data center facility.
The concepts of datacenters and racks are flexible in Cassandra, allowing a variety of deployment patterns. In public cloud deployments, racks are often mapped to cloud provider zones and datacenters to regions. For example, in deployment on Google Cloud Platform (GCP), you might have Cassandra dc1deployed in region us-west4, with racks in zone us-west4-a, us-west4-b, and us-west4-c. Another datacenter dc2might be deployed in us-east1.
Another common pattern takes advantage of the fact that data centers and racks don’t have to strictly map to physical topology. You can create a separate logical data center to isolate workloads. For example, when running heavy analytics workloads it may make sense to spin up a separate logical data center for those queries. Transactional workloads won’t be affected by the surge in queries as Spark jobs are submitted and run against the dedicated analytics datacenter.
How to deploy a separate K8ssandra install per Cassandra datacenter
Let’s look at how you can use Kubernetes namespaces to perform separate K8ssandra installations in the same cloud region.
1. Create a Kubernetes cluster for testing
To try this out, you’ll need a Kubernetes cluster to test on. For the example in this post, I followed the instructions on the K8ssandra Google Kubernetes Engine (GKE) installation docs to create a GKE cluster in the us-west4 region. Note that I followed the instructions on this page up to the “Install K8ssandra” section and stopped, given my installation is slightly different.
The GKE install instructions reference scripts that are provided as part of the K8ssandra GCP Terraform Example.
2. Create administrator credentials
According to K8ssandra’s default options, Cassandra nodes are created with authentication enabled. If no secret is provided containing authentication credentials, K8ssandra creates a secret containing an administrator username and password that will be applied to all the nodes.
In our deployment, you’ll create administrator credentials that are applied to both K8ssandra installs. This ensures that both datacenters will use the same credentials. Since Kubernetes secrets are namespaced, you’ll need to create a copy in each namespace where our data centers are deployed.
You can create the namespaces and secrets as follows:
For the purpose of this blog, let’s keep things simple and let kubectl encode the credentials. You may want to use a more complex password for production deployments. See the Security documentation page and the cassandra.authsection of the K8ssandra helm chart documentation for more information on configuring administrator credentials.
3. Create the first datacenter
Now you’re ready to start creating a multi-datacenter deployment. First, create the configuration for the first data center in a file called dc1.yaml:
In addition to requesting three nodes in the data center, this configuration points to a specific storage class, which uses volumeBindingMode: WaitForFirstConsumer, and uses affinity to specify how the racks are mapped to GCP zones. This configuration also uses the secret you created previously for the Cassandra superuser account.
Now, create a release for the data center using this command:
This installs the K8ssandra release in the txndcnamespace, which is just a name picked to indicate that this is a Cassandra datacenter intended for transactional work.
It’s possible to check the allocation of the Cassandra nodes to workers in different zones using the output of the kubectl describe nodes command, which includes information about the zone of the node as well as its executing pods (I’ve shortened the output slightly for readability):
As you can see:
rack1maps to zoneus-west4-arack2to zoneus-west4-brack3to zoneus-west4-c
As would be the case for any Cassandra cluster deployment, you’ll want to wait for the first data center to be completely up before adding a second data center. A simple way to do this is to watch until the Stargate pod shows as initialized since it depends on Cassandra being ready:
4. Create a second datacenter
Now let’s create a configuration to deploy the second data center. For the nodes in dc2 to join the cluster, a couple of things are required.
- Use the same Cassandra cluster name as for the first datacenter.
- Provide some seed nodes so that the nodes in the second data center know how to contact nodes in the first data center to join the cluster.
In a traditional Cassandra installation, you might grab the IP addresses of a couple of nodes dc1 to provide to dc2 as seed nodes. In K8s, we expect pods to be replaced on a fairly regular basis so there’s no guarantee that a Cassandra node will be available at a particular IP address should it get recreated in a new pod. Fortunately, K8ssandra takes care of this by providing a seed service.
The seed service provides a stable DNS address that resolves IP addresses during the bootstrap process for other Cassandra nodes looking to join a cluster. K8ssandra manages the details of mapping of the pod IP addresses of a couple of underlying Cassandra nodes as the actual seeds. You can find the seed service using a command like this:
Which produces output such as this:
As you can see, the name for the service is mixed-workload-seed-service. To reference it as a seed provider in the other namespace, you’ll need to include the namespace as part of the DNS name: mixed-workload-seed-service.txndc. Now, create a configuration with those values in a file called dc2.yaml:
Like the configuration for dc1, this configuration also uses affinity. A similar allocation of racks can be used to make sure Cassandra nodes are evenly spread across the remaining workers. You can deploy the release using this command:
This installs the K8ssandra release in the analyticsdc namespace. If you look at the resources in this namespace using a command like kubectl get services,pods -n analyticsdc, you’ll note that there are a similar set of pods and services as for txndc, including Stargate, Prometheus, Grafana, and Reaper. Depending on how you wish to manage your application, you’re free to tailor the configuration to disable any components you don’t need.
5. Configure keyspaces
Once the second data center comes online, you’ll want to configure Cassandra keyspaces to replicate across both clusters. To do this, connect to a node in the original data center and execute cqlsh:
Use the DESCRIBE KEYSPACES command to list the keyspaces and DESCRIBE KEYSPACE <name> to identify those using the NetworkTopologyStrategy. For example:
Typically, you’ll find that the system_auth, system_traces, and system_distributed keyspaces use NetworkTopologyStrategy. You can then update the replication strategy to ensure data is replicated to the new data center. You’ll execute something like the following for each of these keyspaces:
Important: Remember to create or alter the replication strategy for any keyspaces you need for your application. This is to make sure you have the desired number of replicas in each datacenter. If you’ve enabled Stargate, make sure to alter the
data_endpoint_authkeyspace to useNetworkTopologyStrategyas well.
After exiting cqlsh, make sure existing data is properly replicated to the new data center with the nodetool rebuild command. It needs to be run on each node in the new data center, for example:
Repeat for the other nodes mixed-workload-dc2-rack2-sts-0 and mixed-workload-dc2-rack3-sts-0.
Testing the configuration
How do you know the configuration worked? It’s time to test it out. The most straightforward way is using the nodetool status command. So, you’ll need to pick a Cassandra node to execute the nodetool command against.
Since K8ssandra configures Cassandra nodes to require login by default, you’ll need credentials. Execute the nodetool the command against the node, making sure to swap in any alternate credentials you may have used:
This will produce output similar to the following:
If everything has been configured correctly, you’ll see both datacenters in the cluster output. Here’s a diagram of what you’ve just deployed, focusing on the Cassandra nodes:
What about multiple Cassandra datacenters in a single K8ssandra install?
You might be wondering: why not do a single K8ssandra installation with multiple datacenters? For example, you might try a configuration like this:
However, if you attempt an installation like this and inspect the results using kubectl get cassdc, you’ll only see a single data center in the resulting deployment:
What’s the issue? It turns out that K8ssandra 1.x releases only support the deployment of a single datacenter. While cass-operator does support multiple datacenters, releases of K8ssandra prior to 2.0 rely heavily on Helm’s templating feature to provide additional infrastructure around Cassandra.
Configuring this additional infrastructure to add or remove datacenters will require significant effort to implement in Helm templates, as discussed in issue #566. Happily enough, the 2.0 release will provide a K8ssandra operator to greatly simplify multi-datacenter deployments. You can read more about these exciting plans in issue #485.
What’s next?
In the second post in this series, we’ll guide you through your first steps in multi-datacenter topologies that span Kubernetes clusters. If you’d like to tell us about alternative configurations you build, or ask us any questions about this topic, jump into the K8ssandra forum or the K8ssandra Community Discord channel.
Follow DataStax on Medium for exclusive posts on all things Cassandra, streaming, Kubernetes, and more. To join the conversation with developers from around the world, follow DataStaxDevs on Twitter.
This post was originally published on K8ssandra.
