Apache Pulsar Performance Testing with NoSQLBench
Author: Yabin Meng
This is an overview of NoSQLBench and how NoSQLBench Pulsar Driver is an improved performance testing tool capable of simulating real production workloads for applications and systems based on Apache Pulsar for DevOps.
Traditionally, there are two ways of doing performance testing with Apache Pulsar. One is with the pulsar-perf command-line tool that’s part of the Pulsar distribution. Another is through the OpenMessaging Benchmark (OMB) tool, which is a generic testing framework for a variety of messaging platforms including Apache Pulsar, and Kafka.
From a workload generation and performance testing perspective, both tools can be used to simulate high-throughput message publishing or consuming against one or multiple Pulsar topics. Pulsar-perf is also able to simulate other types of Pulsar workloads, such as Pulsar reader, WebSocket producer, and managed ledger.
Both tools, however, have limitations when it comes to simulating realistic production workloads. This is because:
- The support for message keys is limited. In OMB, there’s no easy way to specify message keys as part of the workload generation; on the other hand, pulsar-perf only supports key generation either as auto-increment or purely random
- Both tools can only test Pulsar topics within a specific tenant and namespace
- Neither tool supports message schema. They can only use the default binary format as the message payload type
- The message payload is either randomly generated or statically assigned for each message
- The OMB tool is only available on AWS public cloud platform, which may not represent the actual Pulsar production infrastructure environment
- Not all tuning parameters are available to fine control the Pulsar client execution behavior
Because of this, there’s a need for a new performance testing tool that’s suitable for simulating real production workload. Here is a way to test it: NoSQLBench Pulsar Driver.
In this post, you’ll get a quick introduction to the NoSQLBench tool and then explore how the NoSQLBench Pulsar Driver can simulate a realistic production Pulsar workload.
Introducing NoSQLBench
NoSQLBench is a generic workload generation and performance testing tool for a distributed system. It started as a DataStax internal performance testing tool for Apache Cassandra® and then gradually expanded to other distributed systems, including Apache Pulsar. NoSQLBench was open-sourced in 2020 under Apache 2.0 license (APLv2).
Compared with other performance testing tools for a distributed system, NoSQLBench has some unique features and benefits, such as:
- Recipe-oriented procedural data generation through built-in function flows. Data can be both statically and dynamically bound
- Deterministic and repeatable workload behavior
- Modular protocol support via driver extensions for different types of distributed systems.
- Configuration (YAML) based testing scenario design and access pattern modeling using the target system’s native approach
NoSQLBench workloads are defined in YAML configuration files. At its core, a NoSQLBench workload is composed of statements. Each statement can have its own parameters. A set of statement blocks can be grouped together under a common name and selected by tags. The data used in each statement can be either statically or dynamically bound by bindings, with each representing how one particular piece of data is generated via a series of built-in functions.
The example YAML file below (e.g. hello_world.yaml) shows the basic structure of a NoSQLBench workload for Cassandra. In this workload, there are two statements (one for Cassandra write and another for Cassandra read) that can be grouped together under a common phase, main. The Cassandra write/read ratio (4 vs. 1) is specified by the statement parameter ratio. The data for Cassandra write and read is dynamically bound according to the binding function flows.
To execute the above workload, simply run the following NoSQLBench command to trigger one million Cassandra operations, including 800K writes and 200K reads.
NoSQLBench tool has more features and capabilities than I can cover in one blog post, so make sure to check out NoSQLBench documentation for more information. Another good resource to get started is this NoSQLBench tutorial by Jonathan Shook.
NoSQLBench Pulsar Driver in detail
While NoSQLBench provides a common framework for workload generation and performance testing, the actual workload execution against a particular distributed system is achieved via custom-built drivers.
For performance testing against Apache Pulsar, this is NoSQLBench Pulsar Driver.
At a high level, we can achieve the following testing objectives with the NoSQLBench Pulsar Driver that are hard to implement (or not possible at all) with other existing Apache Pulsar performance testing tools:
- Concurrent message publishing and consuming for topics under multiple tenants and/or namespaces
- Realistic production workload simulation with fine-controls of message keys, message properties, and message payloads
- Support for complex Pulsar message schema types, such as Apache Avro
- Able to track end-to-end message processing latency, from a message being published by a producer to it being received by a consumer
- Capable of detecting abnormal message processing errors such as message loss, message duplication, and message out-of-order
- Easy to integrate the rich client-side metrics with Pulsar cluster metrics and get a holistic view of the system performance behaviors
Deterministic workload definition
Following the generic NoSQLBench workload definition pattern, the Pulsar Driver defines Pulsar workloads in YAML files as well. At a high level, a Pulsar workload YAML file has three main sections: bindings, params, and blocks. A template YAML file is as below:
- Bindings section defines how message data is actually generated. For this part, the Pulsar driver follows the generic NoSQLBench data binding principle.
- Params section defines the document-level configuration settings that may impact multiple or all Pulsar workload statements. We’ll cover this in more detail in the next section.
- Blocks section defines the actual Pulsar workload details as NoSQLBench statement blocks. Depending on the Pulsar workload type, the statement block content may be different.
Currently, the NoSQLBench Pulsar Driver supports the following types of Pulsar workload:
- Create or delete Pulsar tenants
- Create or delete Pulsar namespaces
- Create or delete Pulsar topics
- Message publishing
- Message batch publishing (only relevant when using Pulsar sync API)
- Message consuming
- Message reading
- Message consuming using topic list or pattern
For a detailed description of how these Pulsar workload types are defined as NoSQLBench statement blocks, take a look at the NoSQLBench Pulsar Driver documentation.
Complete workload behavior configuration
The behavior of the Pulsar workload execution is impacted by configuration settings that can be set at different levels:
- Global
- Document
- Statement
Let’s look at each level a little more closely.
Global Level configuration
Under the hood, the NoSQLBench Pulsar Driver is built upon Pulsar’s Java client API. With this API, there are various configuration settings that can be used to fine-tune the behaviors for the following:
The NoSQLBench Pulsar Driver supports all these configuration settings. You can define the desired configuration settings in a Java property file (e.g. config.properties) with the right prefixes. For example, for all client connection-related configuration settings, the prefix is “client.” For Pulsar producer-related configuration settings, the prefix is “producer.”, and so on. To give you a better idea, here’s an example file of global level configuration parameters.
The settings under global level configuration will impact all NoSQLBench Pulsar workload definition YAML files, and they’re provided as a NoSQLBench runtime execution parameter, config, as shown below:
Document and Statement Level configuration
Another set of configuration settings can only impact one particular Pulsar workload definition YAML file. Depending on the scope of their impacts within the file, these settings are further categorized as Document Level settings and Statement Level settings.
The document level configuration settings are placed under the “params:” section of the NoSQLBench workload definition YAML file. These settings will impact multiple or all Pulsar workload types defined in the YAML file. At the moment, there are four Document Level configuration settings for the NoSQLBench Pulsar Driver.
On the other hand, one Pulsar workload type may include one or more statements (grouped under different statement blocks) and each statement may have its unique configuration settings. The settings under one statement have no impact on other statements.
You can find a detailed description of the Document Level and the Statement Level configuration settings in the NoSQLBench Pulsar Driver documentation.
Realistic message simulation
When we use the NoSQLBench Pulsar Driver for Pulsar message processing testing, we can achieve a realistic message workload simulation that’s as close as possible to a production environment. Let’s use a simple oil and gas sensor IoT example to see how it works.
In this example, we’re assuming that each message corresponds to a piece of sensor data containing the following information:
- Message Key: The drilling bit identifier
- Message Property: Various static properties of a drilling bit
- Message Payload: Actual data collected by one sensor on a drilling bit
Simulating a message format in such details isn’t possible with any existing Pulsar performance testing tools. But with NoSQLBench Pulsar Driver, it’s simple. Below is an example of a NoSQLBench workload definition YAML file.
In the above example, the message key is a UUID string representing a specific drilling bit ID. The message property is defined by a JSON string that contains a list of key/value pairs representing various drilling bit properties, and the message payload is also defined by a JSON string that follows an Avro schema definition. See below:
Note that the Avro schema definition is made available to the above Pulsar workload by the following global level configuration settings:
Authentic multi-tenancy message processing testing
Another big advantage of using NoSQLBench Pulsar Driver over Pulsar performance testing tools is that we can achieve real multi-tenancy message processing testing. We can do this by dynamical data binding to Pulsar topic names. For a bit more context:
- Statically binding means that the Pulsar topic name will be the same for all NoSQLBench execution cycles (each cycle representing one particular operation, such as publishing a message). This is the single-topic testing as we’ve seen from other Pulsar testing tools.
- Dynamically binding means that the Pulsar topic name will be different for each NoSQLBench execution cycle. The actual Pulsar topic name per cycle will follow the NoSQLBench data binding principle.
In the example below, we create 100 Pulsar topics under 10 tenants, with two namespaces per tenant and five topics per namespace. The Pulsar topic names that are generated follow the pattern of persistent://tnt[0~9]/ns[0~1]/t[0~4].
When running a NoSQLBench Pulsar message publishing workload with one million execution cycles (publishing one million messages), NoSQLBench will make sure that the one million messages are evenly distributed among 100 topics under 20 namespaces of 10 tenants.
Conclusion
In this post, we explored a powerful performance testing utility for Apache Pulsar: NoSQLBench Pulsar Driver. Compared with existing Pulsar performance testing utilities, this one has distinct advantages that enable deterministic, realistic, and production-like Pulsar workload simulation and execution in an easy and flexible way.
This utility also has some other useful features that are simply too much to cover in one article. One other distinguishing feature, for example, is to detect abnormal Pulsar message processing errors such as message loss, message duplication, and message out-of-order. The deterministic nature of this utility’s workload generation and execution makes it easy to capture, analyze and debug these errors. Actually, this feature has already helped our Pulsar engineers uncover a few Apache Pulsar bugs in some rare, racing-related situations.
Despite its powerfulness, this utility is completely open-source. We hope this post invites the Apache Pulsar community to start benefiting from NoSQLBench Pulsar Driver in performance testing, messaging model design, sizing analysis, and cluster deployment evaluation for applications and systems based on Apache Pulsar.
Follow DataStax on Medium for exclusive posts on all things Pulsar, Cassandra, streaming, Kubernetes, and more. To join a buzzing community of developers from around the world and stay in the data loop, follow DataStaxDevs on Twitter and LinkedIn.
