Yandex Pandora Performance Testing: Unleashing Versatility Like a Swiss Army Knife
Hi colleagues!
This is a continuation of my first article on how to use Yandex Tank for stress testing your services. The documentation mentions that the main engine for load is Phantom, but it can be replaced with Pandora if you need to create unique test scenarios or work with other types of requests (such as grpc or any other binaries). In this article, we will explore how to integrate Yandex Pandora for two cases:
– grpc
– flatbuffer (over http)
As I mentioned before, our main goal is to use a simple tool with straightforward documentation that doesn’t require hiring a bunch of people to set up the test. Pandora, like Tank, has very simple and understandable documentation, does exactly what we need, and has a low entry threshold. In my case, the primary stack is Golang, and I choose solutions around it.
My previous article about load testing:
Let’s start by referring to the documentation and see what Pandora is all about.
“Pandora is a high-performance load generator in the Go language. It has built-in HTTP(S) and HTTP/2 support, and you can write your own load scenarios in Go, compiling them just before your test.”
Since we are interested in scenarios, let’s take a look at the documentation:
As we can see from the documentation, there is an excellent example of how to write our “gun” to create our load engine. Great, let’s take this example and copy-paste it into our repository. Next, let’s look at our service contracts:
syntax = "proto3";
option go_package = "/docs";
service DocumentService {
rpc GetAllByLimitAndOffset(GetAllRequest) returns (GetAllResponse) {}
rpc Save(SaveRequest) returns (SaveResponse) {}
rpc Validate(ValidateRequest) returns (ValidateResponse) {}
}Our task involves covering two methods:
- Save
- Validate
Create a Yandex-tank folder, create a grpc folder inside it, move our Proto file there, and generate contracts with the command:
protoc - go_out=./docs - go_opt=paths=source_relative \
- go-grpc_out=./docs - go-grpc_opt=paths=source_relative docs.protoAfter that, we will get files as shown in the screenshot.
And now, let’s start to write own gun. We have to add a huge number of imports and create struct for Custom params:
package main
import (
"context"
"log"
"strconv"
"time"
_ "github.com/golang/protobuf/ptypes/timestamp"
"github.com/spf13/afero"
"google.golang.org/grpc"
pb "grpc-gun/docs"
"github.com/yandex/pandora/cli"
"github.com/yandex/pandora/components/phttp/import"
"github.com/yandex/pandora/core"
"github.com/yandex/pandora/core/aggregator/netsample"
"github.com/yandex/pandora/core/import"
"github.com/yandex/pandora/core/register"
)
type Ammo struct {
Tag string
Param1 string
Param2 string
Param3 string
}Now, we should add GunConfig and Gun where we can add our grpc client (just copy and paste from official docs)
type GunConfig struct {
Target string `validate:"required"` // Configuration will fail, without target defined
}
type Gun struct {
// Configured on construction.
client grpc.ClientConn
conf GunConfig
// Configured on Bind, before shooting
aggr core.Aggregator // Maybe your custom Aggregator.
core.GunDeps
}And add new func for Gun, almost done!
func NewGun(conf GunConfig) *Gun {
return &Gun{conf: conf}
}We can see, that we should add a little bit more for running:
– Bind method. We just bind our client and args:
– override Shoot method
func (g *Gun) Bind(aggr core.Aggregator, deps core.GunDeps) error {
// create gRPC stub at gun initialization
conn, err := grpc.Dial(
g.conf.Target,
grpc.WithInsecure(),
grpc.WithTimeout(time.Second),
grpc.WithUserAgent("load test, pandora custom shooter"))
if err != nil {
log.Fatalf("FATAL: %s", err)
}
g.client = *conn
g.aggr = aggr
g.GunDeps = deps
return nil
}
func (g *Gun) Shoot(ammo core.Ammo) {
customAmmo := ammo.(*Ammo)
g.shoot(customAmmo)
}It’s time to add switch method for SaveCase and ValidateCase requests and to add methods, as example for ValidateCase:
func (g *Gun) caseValidate(client pb.DocumentServiceClient, ammo *Ammo) int {
code := 0
docName := ammo.Param1
out, err := client.Validate(
context.TODO(), &pb.ValidateRequest{Document: createDoc(docName)},
)
if err != nil {
log.Printf("FATAL: %s", err)
code = 500
}
if out != nil {
code = 200
}
return code
}Full code here:
Alright, pay close attention!
func main() {
// Standard imports.
fs := afero.NewOsFs()
coreimport.Import(fs)
// May not be imported, if you don't need http guns, etc.
phttp.Import(fs)
// Custom imports. Integrate your custom types into configuration system.
coreimport.RegisterCustomJSONProvider("grpc_provider", func() core.Ammo { return &Ammo{} })
register.Gun("grpc_gun", NewGun, func() GunConfig {
return GunConfig{
Target: "localhost:84",
}
})
cli.Run()
}When creating our gun, we must specify everything to compile and run it.
We create a provider, specifying the name “grpc_provider”
We create a gun and specify the name “grpc_gun”
In the target, we specify “localhost:84” the location where our service is accessible.
Okay, let’s continue. We have successfully copied and adjusted the code for the gun. Now, how do we run it, and what do we need? First, let’s create ammo again. This time, we’ll simply describe them like this:
{"tag": "/ValidateCase", "Param1": "validate_doc_grpc_pandora"}
This is necessary so that, in the method, we can retrieve the parameters needed for the test. In our case, we will retrieve the document name specified in Param1. Our code will automatically pass this parameter to the ValidateCase method, and we’ll be able to retrieve it.Great, the ammo is ready. Now, let’s configure the test itself:
pools:
- id: HTTP pool
gun:
type: grpc_gun # custom gun name (specified at `register.Gun("my_custom_gun_name", ...`)
target: "localhost:84"
ammo:
type: grpc_provider
source:
type: file
path: validate-json.ammo
result:
type: phout
destination: ./phout.log
rps: { duration: 60s, type: const, ops: 1000 }
startup:
type: once
times: 10Here, we specify our ammo as “validate-json.ammo”
Their type is “grpc_provider”
Our gun is “grpc_gun”
The load is 1000 rps for 60 seconds:
rps: { duration: 60s, type: const, ops: 1000 }And the output goes to the file “./phout.log” — Yandex Tank understands this.
The finishing straight — let’s now perform three simple actions:
1) Build our gun.
2) Assemble a new configurator for the tank.
3) Run tests and check the results.
Building it is very straightforward — just like a simple Go application:
GOOS=linux GOARCH=amd64 go buildIt’s important to specify `GOOS=linux GOARCH=amd64`, otherwise, you might encounter errors when launching the tank. That’s it, the binary file is ready. Let’s update another configurator:
phantom:
enabled: false
pandora:
enabled: true
package: yandextank.plugins.Pandora
pandora_cmd: ./grpc-gun # Pandora executable path
config_file: ./validate-load.yml # Pandora config path
overload:
enabled: true
package: yandextank.plugins.DataUploader
job_name: "grpc validate report"
token_file: "env/token.txt"
telegraf:
enabled: false
autostop:
autostop:
- time(1s,10s) # if request average > 1s
- http(5xx,100%,1s) # if 500 errors > 1s
- http(4xx,25%,10s) # if 400 > 25%
- net(xx,25,10) # if amount of non-zero net-codes in every second of last 10s period is more than 25Specify where our gun and its settings are located. Leave everything else as before and run it all with this script:
echo "run tank"
docker run \
-v $(pwd):/var/loadtest \
- net="host" \
-it direvius/yandex-tank -c validate-tank-load.ymlAfter that, we will see our results.
For flatbuffer, the story repeats. All you need to do is adjust your client code from grpc to http. Here’s an example:
type Gun struct {
// Configured on construction.
client http.Client
conf GunConfig
// Configured on Bind, before shooting.
aggr core.Aggregator // Maybe your custom Aggregator.
core.GunDeps
files [][]byte
}
func NewGun(conf GunConfig) *Gun {
return &Gun{conf: conf}
}
func (g *Gun) Bind(aggr core.Aggregator, deps core.GunDeps) error {
c := http.Client{Timeout: time.Duration(1) * time.Second}
g.client = c
g.GunDeps = deps
g.aggr = aggr
}And then create similar methods and describe their behavior:
func (g *Gun) caseValidate(client *http.Client) int {
host := g.conf.Target
response, err := client.Post(host+"/report/validate", "application/octet-stream", bytes.NewBuffer(g.files[2]))
if err != nil {
log.Printf("FATAL: %s", err)
return 500
}
return response.StatusCode
}That’s it!
In conclusion:
We’ve explored a new tool for performance testing called Yandex Pandora. You can run it as a standalone load engine if you don’t need fancy metrics. Alternatively, you can run it based on Yandex Tank, and in this case, you’ll get beautiful graphs and all the features of this cool tool!
That’s all, hope it was helpful! Thank you.
Full code here:
