Introducing Siddhi YAML Config Manager

Siddhi can run as an embedded Java or Python library, microservice on bare-metal, VM, or Docker, and natively in Kubernetes. With Siddhi 5.1 release a common YAML based configuration can be used to pass system and extension configurations across all types of deployment. In this article, we’ll see how the new common YAML based configuration can be applied to Siddhi running as an embedded Java library.

These system and extension configurations that are relevant to the execution environment can be passed at runtime allowing the same Siddhi App to work on various environments without any changes. These configurations can be passed to SiddhiManager via SiddhiConfigManagers at initialization.

In the next sections, we’ll see the configurations that can be passed to Siddhi Apps at runtime, and how we can write Java API based on the configurations with the new YAML based approach.

Siddhi Configuration Types

Siddhi Manager allows setting of three types of siddhi configurations through Config Manager as follows,

Extension System Parameters

Some extension properties that are common across all instances of the extension can be set as extension system parameters. See the below siddhi definition as an example,

@Store(type="rdbms", jdbc.url="jdbc:mysql://localhost:3306/foo",  
       username="root", password="root",  
       jdbc.driver.name="com.mysql.jdbc.Driver")
define table FooTable(name string, foo string);

This defines a FooTable table that uses MySQL as the backend store. Here, the RDBMS store uses some default properties such as connection timeout which is set to 60000 sec. Let’s say we need to override this to 80000 sec, due to the network condition of the environment, we can simply set this as an extension system parameter such that the property does not need to be defined in each Siddhi store RDBMS annotation.

References

The references allow some sections of source, sink, and store configuration to be passed at runtime. This helps to pass different configuration on various environments such as using MySQL database on one environment and using Oracle on another, without changing the Siddhi Application logic.

Consider the same table example, this specific store configuration can be defined as a reference and used in Siddhi Apps as follows,

@Store(ref='fooDBStore')
define table FooTable(name string, foo string);

Siddhi Properties

Lastly, Siddhi Named aggregation expects some properties related to Distributed Aggregations such as shardId and partitionById that should be passed at runtime.

Let’s say that we need to set 2 extension system parameters for RDBMS store, 1 reference, and 2 Siddhi properties. In the following sections, we’ll see how we can pass them via InMemoryConfigManager and YAMLConfigManager.

Usage of existing InMemoryConfigManager

Up to Siddhi 5.1, the only way of passing configurations is via the InMemoryConfigManger. As it only accepts Maps, we can initialize 3 maps and pass these three types of configurations as follows,

Map<String, String> extensionConfigs = new HashMap<>();
extensionConfigs.put("store.rdbms.mysql.batchEnable", "true");
extensionConfigs.put("store.rdbms.mysql.batchSize", "1000");
Map<String, String> references = new HashMap<>();
references.put("fooDBStore.jdbc.url",     
                                 "jdbc:mysql://localhost:3306/foo");
references.put("fooDBStore.username", "root");
references.put("fooDBStore.password", "root");
references.put("fooDBStore.jdbc.driver.name", 
                                  "com.mysql.jdbc.Driver");
Map<String, String> properties = new HashMap<>();
properties.put("partitionById", "true");
properties.put("shardId", "shard1");
InMemoryConfigManager inMemoryConfigManager = new 
    InMemoryConfigManager(extensionConfigs, references, properties);
SiddhiManager siddhiManager = new SiddhiManager();
siddhiManager.setConfigManager(inMemoryConfigManager);

As seen above, this needs repetitive coding to initiate the Config Manager, and this also needs code change when deploying in different environments.

Now let’s check how the same properties can be passed via YAML Config Manager.

Usage of YAMLConfigManager

From Siddhi 5.1 onwards, the YAMLConfigManager can be used to pass the same configuration via YAML as follows,

# Siddhi Extensions system Parameters
extensions:
  - extension:
      name: rdbms
      namespace: store
      properties:
        mysql.batchEnable: true
        mysql.batchSize: 1000
# Siddhi references
refs:
  - ref:
      name: fooDBStore
      type: rdbms
      properties:
        jdbc.url: jdbc:mysql://localhost:3306/foo
        username: root
        password: root
        jdbc.driver.name: com.mysql.jdbc.Driver
properties:
  partitionById: TRUE
  shardId: shard1

Let’s say this file is saved as siddhi.yaml.

This can be initialized, configured and passed to SiddhiManager via YAMLConfigManager with only 4 lines of code.

In conclusion, YAML Config Manager comes as a convenient and efficient way to set Siddhi Configurations to Siddhi Java library and enables Siddhi Apps to efficiently migrate among various environments.