Stress-testing Cassandra with JMeter and Groovy

About testing a Cassandra cluster with JMeter and Groovy scripts.

The JMeter test presented will simulate an application acquiring measures from sensors and inserting them in a Cassandra table.

The versions used here are JMeter 3.0 and Cassandra 3.10 (trunk as of today).

This howto post is divided in 4 parts:

1. The prerequisites: JMeter+Groovy, the Cassandra driver, the Cassandra cluster with the application data model
2. Test plan overview
3. Creating the test plan elements
4. Executing the test and results

1. Prerequisites

JMeter 3.0

You will need the standard Apache JMeter 3.0 distribution.
In our test JMeter will be used in non distributed mode in a dedicated VM beside the Cassandra VMs cluster . This is not the best configuration for tests but simpler for the story.

Groovy scripting support

Groovy scripting is supported by JMeter with the groovy-all.jar file in your JMeter lib directory, and Groovy version 2.4.6 is included in the JMeter 3.0 distribution.
JMeter has good support for Groovy, however, debugging or testing a script in JMeter is tedious, I recommend you to download the Groovy SDK and use the Groovy console to test your scripts, or much better, download and use the IntelliJ Community Edition (free) who has an outstanding Groovy support if you want to modify the scripts or go further.
This is not needed if you only want to reproduce this test.

The Cassandra driver

In the scripts we will develop in this test, we will use the Datastax Cassandra java driver. You can downlad the driver’s jar and it’s dependencies from maven here and put them in the lib folder of your JMeter installation or you can pick them from any java project build or your own who use it, but versions must match together.

The jar file I use is is cassandra-driver-core-3.1.2.jar and I have the following dependencies with versions : guava-16.0.1.jar , HdrHistogram-2.1.4.jar , lz4–1.2.0.jar , metrics-core-3.1.2.jar , netty-buffer-4.0.33.Final.jar , netty-codec-4.0.33.Final.jar , netty-common-4.0.33.Final.jar , netty-handler-4.0.33.Final.jar , netty-transport-4.0.33.Final.jar , snappy-java-1.0.5.jar

The Cassandra cluster

In the test, we will use a Cassandra cluster of 3 nodes named cstar1, cstar2 and cstar3, each one running on a small virtual machine of the same name.
You can use whatever configuration of your own for the cassandra cluster, the only thing to change will be the clusterNodes JMeter variable in the test plan.

The application Data Model

The test will simulate an application acquiring measures from sensors and inserting them into a “measures” Cassandra table in a “sensors” keyspace.
Each measure comes from a location, has an event time, a metric id and a value.
The Cassandra data model is given by the following CQL script :

DROP KEYSPACE if exists sensors;
CREATE KEYSPACE IF NOT EXISTS sensors
WITH REPLICATION = { ‘class’ : ‘SimpleStrategy’, ‘replication_factor’ : 2 };
drop table if exists sensors.measures;
CREATE TABLE sensors.measures
(
  bucket int,
  event_time timeuuid,
  location_id text,
  metric_id text,
  value double,
  PRIMARY KEY ((bucket, location_id), metric_id, event_time)
) WITH CLUSTERING ORDER BY ( metric_id ASC, event_time DESC)
and DEFAULT_TIME_TO_LIVE = 604800;
 — one day bucket time : on insert 
 — eventTime.getTime() / (24*60*60*1000L); => day bucket
 — one day ttl : 86400 seconds
 — one week ttl : 604800 seconds

You must execute this script in DevCenter or Cqlsh to create the keyspace for the test on your cassandra cluster.
Note that the data will be bucketed by day to avoid too large partitions and also that in order to spread data on the cluster, the location of the sensor has been added to the partition key.
The bucket will be set on insert by the application (the groovy script).
The order of the metric_id and event_time columns in the primary key and clustering columns section could have been set the other way round, depending on the requirements of the application.

2. Test plan overview

The test plan is made of the following JMeter elements:

• a User Defined Variables element: those variables are parameters to the plan like the cassandra node names, the keyspace, the number of locations simulated or the number of metrics per location and the duration of test in seconds.
• a setUp Thread Group: a group of scripts that will be executed at test startup. It’s main purpose is to open the cassandra connection and prepare the statement used in the other scripts (the Datastax driver requires that there is only one Cluster and one Session instance shared by all threads of a java application).
  Note: there is a trick here to share the Session and PreparedStatement objects between threads, explained below.
• the main test Thread Group: a runtime controller that insert measures for “duration” seconds (60 seconds here)
• a tearDown Test Group : closes the connection
• an Aggregate Report : gather events from other script elements and build a final report.

Before going into elements details, let’s see how to share variables in groovy scripts between JMeter test threads.

Sharing objects between JMeter threads in Groovy scripts

JMeter User defined variables are local to a JMeter thread context and cannot be shared between threads and thread groups, however JMeter has a class named JMeterUtils that gives access to the static java.util.Properties hashtable of JMeter with the static method getJMeterProperties(), so we will use the put method of this hashtable to store our variables as objects (and get method in other scripts):

public static Properties getJMeterProperties() {
    return appProperties;
}

In order to acces them from a groovy script , we can simply import the JMeterUtils class and call this method like in the cluster-connection script:

import org.apache.jmeter.util.JMeterUtils;
...
cluster = ... assign cluster local variable here
session = ...
Properties properties = JMeterUtils.getJMeterProperties();
properties.put("cluster",cluster); // store cluster local variable in properties
properties.put("session",session);

This is not thread safe and assignment should be done only in a single ‘setUp’ thread, not in user threads. Reading properties in other threads is not a problem if the hashtable remains unmodified.

Note: this trick should work most of the times but is not guaranteed. Use at you own risk … as always :)

3. Creating the test plan

Adding User Defined Variables

Right click on Test Plan icon and choose ‘Add/Config Element/UserDefined variables’ element.
Add the following variables and change clusterNodes to match your configuration or nlocations, nmetrics or duration.

Adding setUp Test Group

right click on Test Plan icon and choose ‘Add/Threads(Users)/setUp Thread Group’ element.
Select ‘Stop Test’ to instruct JMeter to abort test in case of error.
Be sure there is only one thread.

Adding cluster-connection script to the setUp Thread Group

To add the connection-script element to the test right-click on the ‘setUp Thread Group’ choose ‘Add/Sampler/JSR223 Sampler’, then change the Name to ‘cluster-connection’, select ‘groovy’ language in the Combo and ‘cache compiled script’ (if not already selected)

In the Script field, put the following script (change datacenter name or other options to suit your needs):

import org.apache.jmeter.util.JMeterUtils;
import com.datastax.driver.core.Cluster;
import com.datastax.driver.core.Session;
import com.datastax.driver.core.policies.DCAwareRoundRobinPolicy;
String nodes = vars.get("clusterNodes");
String keyspace = vars.get("keyspace");
String [] addresses = nodes.split(",");
Cluster cluster = Cluster.builder()
   .addContactPoints(addresses)
   .withLoadBalancingPolicy(
     DCAwareRoundRobinPolicy.builder()
      .withLocalDc("datacenter1")
      .withUsedHostsPerRemoteDc(2)
      .allowRemoteDCsForLocalConsistencyLevel()
      .build())
   .withCredentials("cassandra","cassandra")
   .build();
Session session = cluster.connect(keyspace);
// global JMeter variables are shared in the same ThreadGroup but cluster and session
// must be shared by all ThreadGroups
// => use JMeter properties hashtable in order to reuse them in other scripts
Properties properties = JMeterUtils.getJMeterProperties();
properties.put("cluster",cluster);
properties.put("session",session);

Adding the prepare-statement script to the setUp Thread Group

Follow the same procedure as for cluster-connection, with the following script:

import org.apache.jmeter.util.JMeterUtils;
import com.datastax.driver.core.PreparedStatement;
import com.datastax.driver.core.Session;
def insertMeasureCqlString = ' insert into sensors.measures ( ' 
 +     ' bucket , event_time, location_id, metric_id, value  ) ' 
 +     ' values (  ?, ?, ?, ?, ? ) ;'
Properties properties = JMeterUtils.getJMeterProperties();
Session session = properties.get("session");
PreparedStatement preparedInsertStatement = session.prepare(insertMeasureCqlString);
properties.put("preparedInsertStatement",preparedInsertStatement);

Adding Main Thread Group

Right click on Test Plan icon and choose ‘Add/Threads/Thread Group’ element.
Select ‘Stop Test’ to instruct JMeter to abort test in case of error.
Change the number of threads to ${nlocations} variable

Adding a Run time Controller to the main thread Group

Right click on Thread Group icon and choose ‘Add/Logic Controller/Runtime Controller’ element.
Select ‘Stop Test’ to instruct JMeter to abort test in case of error.
Change the ‘Runtime (seconds)’ variable to ${duration}

Adding the ‘insert-measures’ script to the previous Runtime Controller

Right click on Run Time Controller icon you just added, and choose ‘Add/Logic Sampler/JSR223 Sampler’ element.
Change name to ‘insert-measures’, select ‘Stop Test’ to instruct JMeter to abort test in case of error.

change the script to the following

import org.apache.jmeter.util.JMeterUtils;
import com.datastax.driver.core.ConsistencyLevel;
import com.datastax.driver.core.PreparedStatement;
import com.datastax.driver.core.Session;
import com.datastax.driver.core.utils.UUIDs;
//  get the shared Session and Prepared statement 
// from JMeter properties
Properties properties = JMeterUtils.getJMeterProperties();
Session session = properties.get("session");
PreparedStatement preparedInsertStatement = properties.get("preparedInsertStatement");
Date eventTime = new Date(); // in ms since 1/1/70
int bucket = eventTime.getTime() / (24*60*60*1000L); // day bucket
int nMetrics = Integer.parseInt(vars.get("nmetrics"));
Random random = new Random((long)bucket);
// The "location" will be the thread number
String location = "location-T${__threadNum}";
for (int metric = 1; metric <= nMetrics; metric++) {
    String metricId = "metric-"+metric.toString();
    double value = random.nextDouble() * 100.0;
    // insert into sensors.measures( bucket, event_date,     
    // location_id, metric_id, value ) ' 
    session.execute(
            preparedInsertStatement.bind(bucket,     
                     UUIDs.timeBased(),
                     location, metricId, value )
                   .setConsistencyLevel(ConsistencyLevel.ANY)
    );
}

Adding tearDown Test Group

Right click on Test Plan icon and choose 'Add/Threads(Users)/tearDown Thread Group' element.
Select 'Stop Test' to instruct JMeter to abort test in case of error.

Adding close-connection script to the tearDown Thread Group

To add the connection-script element to the test, right-click on the ‘tearDown Thread Group’, choose ‘Add/Sampler/JSR223 Sampler’, change the Name to ‘close-connection’, select ‘groovy’ language in the Combo and ‘cache compiled script’ (if not already selected)
and set the following script:

import org.apache.jmeter.util.JMeterUtils;
import com.datastax.driver.core.Cluster;
import com.datastax.driver.core.Session;
Properties properties = JMeterUtils.getJMeterProperties();
Session session = properties.get("session");
Cluster cluster = properties.get("cluster");
session.close();
cluster.close();

And finally the last element we will add is an

Aggregate Report

Right click on Test Plan icon, choose Add/Listener/Aggregate Report

And that’s it !

4. Running the Test

With the Cassandra cluster up and running, the keyspace created with the given CQL script, you can run the test we have created by clicking on the ‘Start No Pause’ icon .

If you have a VM management tool you can see your cluster is working

And after the end of the test, the Agreggate Report shows the numbers

100 locations of 10 metrics each running 60 seconds gives 1,703 insert-measures/sec, this makes 17 K inserts/sec on Cassandra.

You may notice that the numbers are not very high but given that we have with a replication factor of 2, very small VMs used (2 VCPU, 2G RAM each) for cassandra, the Xen virtual disks of all VMs all going to the same SSD, with a 6VCPU for the testloader and an hypervisor a bit short on VCPUs, this is not too bad.

Note: A quick cpu profiling of one cassandra node with jvisualvm showed lot of time spent in network (netty/epollWait), the Xen frontend/backend networking is clearly a bottleneck in this configuration, and that’s another reason to use real hardware for real tests.