Introducing the K8ssandra Cloud-Readiness Framework
Author: Jeff Banks
This post introduces a cloud-readiness testing guide for K8ssandra version 0.1 to help you easily discover cloud-specific issues for new versions of K8ssandra. Using decoupled and customized modules, you can mix and match your testing APIs and frameworks for better cloud testing. For this solution we used Terraform, GoLang, Kubernetes and Terratest.
The K8ssandra team recently released K8ssandra-operator v1.1. Even though K8ssandra is still evolving, a large number of features have been added, ranging from multi-cluster support to restoring a Medusa backup from any Apache Cassandra®cluster. With all of these capabilities, we need to ensure quality checks are executed at various levels — from unit to cloud verification.
We included an upfront test focus in the construction of K8ssandra-operator v1.0, with numerous test scenarios primarily scoped to unit, integration, and e2e. Currently, these tests can run locally or as part of the CI/CD process.
Automated cloud smoke tests were particularly needed while building K8ssandra features to help discover cloud-specific issues early in the development process. As such, we built a K8ssandra cloud-readiness framework to support the development of “in cloud” tests for new versions of K8ssandra.
Architecture and technologies
In a nutshell, the cloud-readiness framework is a GoLang module that plugs-in reusable cloud infrastructure provisioning, creates Kubernetes resources, and optionally deploys a K8ssandra ecosystem in a single or multi-cluster configuration.
Version 0.1 for cloud-readiness leverages Terraform modules for infrastructure provisioning, as this was something the K8ssandra team already had available. The technologies we used for this solution include the following:
A conscious decision was made to not only reuse existing provisioning, but also include simulation and verbose logging in this solution. This combination makes troubleshooting and test failure diagnostics easier given optional settings available for Kubernetes and cloud deployments.
For example, to enable a simulation for provisioning, it is as simple as assigning the value true to the “Simulate” flag and running your specific scenario using the command:
go test -v <your-test>.
Cloud-readiness models are what inform the test scenario to provision and install for each cloud environment.
As you construct scenario models, you will notice that K8ssandra tests (i.e. smoke) reference their own reusable named packages. The idea is to allow for reuse of test scenario definitions involving cloud specific setup and resource creation.
Currently, there is a cloud context model and readiness model used to separate areas of concern, such as: cloud resource/infrastructure setup, K8ssandra installation, and pre-installation activities.
Testing the tests
At the top-level of the cloud-readiness project, there is another test folder. This folder is not specific to K8ssandra verifications, rather it is used to verify the overall automation provided by the framework.
This enables “testing the tester,” so to speak. It’s essentially able to verify the framework components to ensure future changes don’t break existing and new scenarios.
Kicking the tires
A cloud-readiness user guide for v0.1 is available and walks you through the details of getting started with your initial scenario construction. What is even better is that all of this is part of the K8ssandra open-source projects.
The decoupled modules and components allow for mixing and matching your own set of test frameworks, APIs, deployments, etc. For example, if you want to use Pulumi instead of Terraform, simply fork the cloud-readiness repository and customize your Go-based test pre-conditions to reference those bits. Then, only reference the installation activities for K8ssandra (pre-install Cert-manager and K8ssandra ecosystem install).
Also, if you would like to run a set of system tests and smoke tests, create a custom GoLang test to plugin infrastructure provisioning, the deployment of K8ssandra, and the deployment of your own Kubernetes resources.
The cloud-readiness framework is still in its infancy with v0.1. Stay tuned for more updates as we roll out additional features. As always, feel free to open enhancement requests, fork the repository, and contribute — we love hearing from you!