Introducing WSO2 Testgrid
WSO2 Testgrid…. mmh, first thing that comes to mind when someone see this is did WSO2 release another (cool!) product? Well what does WSO2, as an integration company, has to do with testing? But, there’s this github repo that says its about increasing enterprise customer confidence on WSO2 products. That sounds like it’s some kind of test tool to test and validate WSO2’s existing product stack. Is that what this is?
Yes, that’s quite right. Testgrid is written with the goal of automated testing and validation of WSO2 product releases, weekly updates/patches across multiple infrastructures. As WSO2 products can be run on customers’ on-premise data centers, we needed to test all WSO2 deployment-patterns against each possible combinations of infrastructure WSO2 claim to support.
It further has the goals of automated execution of performance tests, long running tests, and chaos tests. These need to be run against all four products of WSO2, namely — API-Manager, Enterprise Integrator (EI), Identity Server (IS), and Stream Processor (SP).
To summarize, testgrid:
- Certify WSO2 product releases
- Certify WSO2 product updates
- Validate WSO2 installation scripts,
And, it provides a standard platform to run,
- Integration tests
- Performance tests
- long running tests, and
- Chaos tests,
for all products at wso2 with focus on three-dimensional automation — ie. infrastructure provisioning, deployment creation, and test execution.
As a WSO2 user, why do I need to know about Testgrid?
Testgrid is WSO2’s way of providing quality guarantee that its products can be run on any enterprise environments. As we are moving away from manual QA to automated testing, this will be the central hub where all automation tests will get executed. We run the same test across loads of different combinations of enterprise environments. See further below to learn about our current offering.
Suppose you like to prioritize the testing on a particular infrastructure, or if your enterprise has a different infrastructure requirements (say, you want to run WSO2 on an exotic cloud provider called XYZ). Or suppose you want WSO2 to validate the WUM updates it releases on your own production deployment’s replicate, with support engineers available 24/7, 365 to monitor and validate the WSO2 scenarios you use? Then, testgrid can fulfill your needs.
We further can provide automation engineers to write automated scenario test-scripts (jmeter-based?) for your own artifacts in order to validate your WSO2 product deployment. This facility is available for WSO2 premium subscription customers.
Like to engage with us? Subscribed customers can use the WSO2 support portal, or use our usual communication channels at wso2.com/mail. I’m available at kasung@wso2.com. Mentioning “Testgrid Enablement” in the subject line makes it easy find.
How we run Testgrid
Testgrid build jobs does the above by following this three-step pipeline:
- Provision an infrastructure (infra.sh) — say in AWS via Cloudformation/Terraform Infrastructure-as-Code
- Create product deployment (deploy.sh) — say via puppet or just plain shell scripts
- Execute tests (test.sh) — we support jmeter, maven/testng/junit, or any other test via plain shell scripts.
Take extra note that we see the deployment and tests as orthogonal concepts. That is, irrespective of the deployment pattern (ie. one node, clustered two node, distributed, container) does not affect how the test is written. What tests expect from the deployment is simply a set of endpoints of the deployment. For API-M for example, the deployment scripts just need to output the gateway http/s endpoints, store/publisher endpoints etc. This makes it really straightforward to write tests because test writer does not need to worry about the intrinsic details of the deployment.
Following diagram shows how testgrid helps to improve the products and deployment scripts of wso2:
Testgrid currently has OOTB plugins to provide specialized support to run Cloudformation scripts, Kubernetes via Amazon EKS, JMeter tests, and TestNG/JUnit based tests via Maven. Testgrid is aware of how to handle AWS credentials, parse result outputs of JMeter (.jtl files) and maven surefire report, and generate alerts in case of test failure.
WSO2 has infrastructure and deployment provision scripts for almost all deployment patterns of wso2 products. We are currently heavily testing the AWS CloudFormation and Puppet scripts via TestNG based scenario test scripts. WSO2 customers can extend these test scripts to validate their own deployments and artifacts as well. For example, you can write your own tests to validate EI/ESB proxy services, or validate your own managed API functionality deployed in API-Manager, or validate that the OAuth2.0/OIDC based authorization you implemented for your web application via WSO2 IAM product works as expected after a product migration/wum updates.
While Testgrid can run tests against the wso2 deployments, it is not limited to that. The abstract architecture makes it possible to run any infra, deployment, and test scripts. Essentially, these three are just three shell scripts for us. Reusable templates and samples are available such that you can start directly jump into start writing scenario tests. This abstract architecture is the real power of testgrid in my opinion. I write more on that in the coming weeks.
We are still on private beta. You can email me on kasung@wso2.com or use contact channels mentioned above if you are interested in getting your product deployment validated via WSO2 Testgrid. I would love to know what you think, good or bad. I’m looking forward to it!
