Building an efficient CI/CD pipeline for Siddhi
What’s CI/CD for Siddhi?
CI/CD contains two separate but complementary parts that enable users to automate continuously integrate of Siddhi Applications that they have developed, and continuously deliver the integrated apps by verifying they are ready to run on production.
If we look at these in detail; the Continuous Integration (CI) is a process of automatically testing and building software after new bits of application code is integrated into a shared repository. This yields “builds” of the application that is in a working state at all times. Unit tests and integration tests are an integral part of the continuous integration process, where they validate the functionality of the software. This identifies bugs up-front and prevents wasted cycles down the line.
Whereas Continuous Delivery (CD) is a process of delivering applications created in the CI process to the production environment. Here, the apps get deployed into production-like environments where they go through additional automated tests (black-box testing) to ensure that they function and interact with other software systems as expected. Successful CD means that the builds are always ready for deployment to production, either via an automated or manual process.
It’s debatable whether “unit tests”, “integration tests” and “black-box tests” are the right terms. For simplicity’s sake we’ll call tests that run “within one process” without any external dependencies as “unit tests”, the ones that run with a mock or scaled-down version of the external systems as “integration tests”, and the ones run in a production-like environment as “black-box tests”.
Siddhi Test Framework
Implementing an effective CI/CD pipeline can require lots of effort and writing plenty of boilerplate code. This process has been simplified using the Siddhi Test Framework which is a TestNG based implementation that has enough prewritten components to help developers implement unit, integration and back-box tests for their Siddhi Apps.
The test framework when needed dynamically spawning dependent containers and allows apps to integrate with external systems to test and validate their functionality and integration against the whole ecosystem.
The following diagram depicts a reference implementation of the CI/CD pipeline of Siddhi Apps.
As a prerequisite, developers need to create a Siddhi Runtime Docker image and publish them to the Docker Registry if they use extensions that are not packed by default in the Siddhi Runner or is they use any other custom jar. Further, the developed Siddhi Apps (.siddhi script files) should be maintained in a code repository such as GitHub.
The CI/CD process is triggered when the developer pushes a commit with changes to the Siddhi application in the Git repository. The CI server notices the change and starts running through its pipeline. The pipeline would be configured to perform both Continuous Integration and Continuous delivery tasks.
As part of Continuous Integration, the pipeline would run unit tests by running Siddhi Application in isolation, and performs integration tests by running the application and dependent services as Docker containers. This ensures that the updated Siddhi application functions as expected.
Then, in the Continuous delivery phase, the CI server requests the Kubernetes cluster to deploy a production-like environment and runs black-box tests against the updated Siddhi application by letting it communicate with other dependent services in the environment. Upon successful tests, it requests the Kubernetes cluster to move the updated Siddhi Application to the production environment.
Let’s see how these can be implemented in a CI/CD pipeline for a sample Siddhi Application.
Sample Scenario
Below is a sample Siddhi application which performs temperature monitoring and anomaly detection. Let’s see how we can configure CI/CD for this application.
Continuous Integration
Continuous Integration can be validated through unit tests and integration tests. The process of writing these tests for Siddhi is discussed below.
Unit Testing
During unit testing, Siddhi is used in an embedded sandbox mode to make sure no network connections or external dependencies are required for testing. This requires the developer to write a simple Java code to test the system.
Here, the Siddhi Application is passed to the Siddhi Core Java library, and a sandboxed runtime will be created. This starts the application without connecting to any external sources or sinks, and all external data stores (tables) will be converted to in-memory tables. Developers can now publish events to any stream of the application and consume events produced in any stream or query using callbacks and assert them.
Below is a sample unit test that performs functional testing on the above-mentioned sample Siddhi Application.
Here, in line 7 we load the Siddhi Application which resides within the resources directory and gets its content as a String. Then, the application content is passed to SiddhiManager by calling createSandboxAppRuntime() to create a Sandbox SiddhiAppRuntime for testing, without any network endpoints such as sources, sinks, and stores.
The output of the monitered-filter query is captured by registering a QueryCallBack at line 10, which asserts the deviceID of the first and second event occurrences. Further, a counter is also maintained, such that the test will wait until all the expected events are outputted from the query before exiting.
To publish events into the DeviceTemperatureStream stream an InputHandler is retrieved at line 23, and after starting the SiddhiAppRuntime, three events to the InputHandler of DeviceTemperatureStream adhering to the definition of the stream. For example, the events sent to DeviceTemperatureStream should have attributes in with types String, String, Double, and String in the given order.
Apart from publishing data and asserting the output for the streaming queries, users can also inspect the internal state of the system and assert them. The query() is used at line 30 to retrieve all the events stored at the InternalDevicesTempTable at the end of the test for the assertion.
Please refer to the official documentation for more information on using Siddhi as an Embedded Java library.
Integration Testing
As the purpose of the integration test is to verify how the application behaves with other systems within the whole solution stack, such as with databases, message brokers, backend services, and so on. The interaction between these systems can only be tested by actually running all of these next to each other.
In the integration tests we will try to achieve the following goals:
- Start a production-ready package of the Siddhi app.
- Start an instance of each dependent system.
- Run tests by interacting only via the public service endpoints with the Siddhi app.
- Not preserving the data between tests, such that we don’t have to worry about restoring to the initial state before each test.
- Allocating resources only at test execution and decommissioning them when the tests are done.
The Siddhi Test Framework provides these capabilities through its containerized platform built using TestContainers and TestNG. Here, the build job will start an isolated execution environment for every integration test or for a set of tests. During this phase, the framework allows bring up third party dependent systems as Docker containers for testing, which lives only for the lifetime of that job. As this framework is built using TestContainers which in turn depends on Docker, see general docker requirements for more information.
A sample implementation of the test suite for the Temperature Alert App that we discussed earlier, with necessary dependencies can be found here. This can be used as a reference when you are trying to build your own Test Suite for your Siddhi Application.
The folder structure of the Temperature Alert App test suite is as follows,
This maintains the Siddhi Applications, configuration files, and client jars that are used from the test cases within src/main/resources/TemperatureAlertApp/apps, src/test/resources/TemperatureAlertApp/config, and src/test/resources/TemperatureAlertApp/jars directories respectively. These resources are imported into the Siddhi Runner Docker container during test startup.
If the client JARs not locally available, users can also use the Maven POM to import the dependent jars to be included in the Siddhi Runner container.
Within the test cases, @BeforeClass, @Test, and @AfterClass annotations are used. The method annotated with @BeforeClass runs before the first test invocation in the current class, which is followed by a series of test methods being invoked that are annotated with @Test, and finally, the method annotated with @AfterClass will get invoked.
In the following sections, we will look at what operations are performed in each of these sections.
Method annotated with@BeforeClass
As the sample use case depends on database and messaging system modules; those modules are configured in the setUpCluster() method which is annotated with @BeforeClass. This will configure the necessary infrastructure to run the test cases. Here the setUpCluster() method performs the following tasks.
- Creates an instance of a virtual network.
- Spins up a MySQL container with a specific database name, a network alias, and registers it to the network.
- Spins up a NATS messaging system container with a specific NATS cluster ID, a network alias, and registers it to the network.
- Spins up a Siddhi Runner container with the SiddhiApps, jars, configs, environment variables, and registers it in the network. The NATS URL and database URL that are used by Siddhi to communicate with the said modules are exposed as environment variables in the Siddhi Runner container which will then be fetched by the Siddhi Application at runtime.
Methods annotated with @Test
The testing logic is written under testAppOutput() method annotated with @Test. This will be picked up by the framework as a test during the testing phase.
The Siddhi Test Framework provides NATS client as a utility, using this the above test publishes messages to the topic associated with the DeviceTemperatureStream and captures the output produced via topic associated with the AlertStream to asserts them.
Method annotated with @AfterClass
The shutdownCluster() method annotated with @AfterClass, runs after all the @Test methods in the current class have been run. It cleans the infrastructure which is created by the setUpCluster() method during test suite startup, by stoping all the dependent modules which were spawned by the framework.
Continuous Deployment
Continuous Deployment can be validated through the black-box tests. The process of writing these tests for Siddhi is discussed below.
Black-box Testing
Siddhi Test Framework can also be used to run black-box testing by configuring the Siddhi Runner instances to communicate with the actual external systems. This can be achieved by updating the environment variables of the Siddhi Runner to point the test/production systems.
Following is a sample cluster setup configuration to run a black-box test with Siddhi Test Framework for the aforementioned Temperature Alerting Application.
As before, the setUpCluster() method configures the infrastructure for the tests to run, and in this case, instead of spawning dependent modules, it configuring the Siddhi Runner instance to communicate with the actual systems in the environment. Here, the URL will be pointing to the test/production systems that are passed via environment variables to the Siddhi Runner.
This can be followed by the same testing logic that we used during the integration test to run the black-box test. The code for the test case can be written and managed in a single file for both integration and black-box test as in the abstract test class implementation of Temp-Alert-App in the sample test suite.
Once the Siddhi Application passes the unit test, integration test, and black-box test phases, it can be deployed to the production environment.
Deploying Siddhi CI/CD pipelines in a Kubernetes cluster
We have created a CI/CD infrastructure setup consisting of Jenkins and Spinnaker to set up your Siddhi CI/CD pipeline. It can be deployed on Kubernetes via HELM, allowing you to easily configure, install, scale and upgrade. In addition to the tools, the necessary jenkins jobs, and spinnaker pipelines are also preconfigured, for a hassle-free start.
Follow the instructions in the ReadMe to set up and run your own CI/CD infrastructure.
