New Configuration model of WSO2 products

https://upload.wikimedia.org/wikipedia/commons/1/12/Logo-toml.svg

Hello people,

Today topic is the new configuration model of WSO2 products, and I am using the WSO2 APIM 3.x product to explain this topic.

WSO2 is implementing 3 main products. They are WSO2 Enterprise Integrator, WSO2 API Manager and WSO2 Identity Server. The latest versions of each product are using this new configuration model.

The new configuration model is based on the toml format, and it has merged all configurations to a single file. That file is “deployment. toml”, which resides in the <Product_Home>/repository/conf directory. This model has introduced for products from the below versions.

• From the WSO2 EI 7.1.0 version
• From the WSO2 APIM 3.0.0 version
• From the WSO2 IS 5.9.0 version

In the previous version of WSO2 products, we need to change multiple XML files to configure a product like master-datasource.xml file, registry.xml file, use-mgt.xml file ..etc. However, due to this centralized toml configuration model, we need to edit one file and make it easy. This is a great advantage when setting up a new deployment.

How to configure the “deployment. toml” file?

If you want to migrate from an old to a new version of a WSO2 product, then you need to set up a deployment. In that case, you need to migrate those old configurations to the new set-up to enable the features you used in the old environment. This is calling configuration migration, and for that, you need to learn how this new configuration model is working.

Identifying configuration parameters

To configure the “deployment. toml” file, you need to aware of the “.j2 template files” and “default.json” file.

.j2 template files

You can find the “.j2 template” files in the <Product_home>repository/resources/conf/templates/repository/conf directory, and it has the same file structure as the files in the directory <product_home>/repository/conf. The above paths may change in the EI product, but APIM and IS have a similar structure.

These “.j2 template” files contain the parameterizations which require configuring the toml file.

e.g., If you want to change the formatter and builder based on the content type, you need to refer to the axis2.xml.j2. For example, try to change the formatter of “multipart/form-data”,

axis2.xml.j2:

<messageFormatter contentType=”multipart/form-data”
 class=”{{message_formatter.multipart_form_data}}”/>

Let’s check the default value of this message formatter. This default value can find in the “default .json” values located in the <product_home>/repository/resources/conf directory. You can search this “message_formatter.multipart_form_data” parameter that has mentioned in the j2 file.

default.json:

"message_formatter.multipart_form_data": "org.apache.axis2.transport.http.MultipartFormDataFormatter",

Now you need to change this to “org.wso2.carbon.relay.ExpandingMessageFormatter”. Then you can change this through the “deployment. toml” file by configuring the j2 parameters as follows.

deployment. toml:

[message_formatter]
multipart_form_data = "org.wso2.carbon.relay.ExpandingMessageFormatter" 

As same, you can change the builder also by checking the parameter of the j2 template file.

axis2.xml.j2:

<messageBuilder contentType="multipart/form-data"
                            class="{{message_builder.multipart_form_data}}"/>

default.json:

"message_builder.multipart_form_data": "org.apache.axis2.builder.MultipartFormDataBuilder",

deployment. toml:

[message_builder]
multipart_form_data = "org.wso2.carbon.relay.BinaryRelayBuilder"

Even you can add custom formatter and builder too.

axis2.xml.j2:

{% for message_formatter in custom_message_formatters %}
        <messageFormatter contentType="{{message_formatter.content_type}}"
                          class="{{message_formatter.class}}"/>
     {% endfor %}

Then you don’t need to check the defult.json file because custom things are unique for you.

deployment. toml: (please note below are examples only)

[[custom_message_formatters]]
content_type = ".*"
class = "org.wso2.carbon.relay.ExpandingMessageFormatter"
[[custom_message_formatters]]
content_type = "application/body"
class = "org.wso2.carbon.relay.ExpandingMessageFormatter"

As similar, you can configure other configurations by checking the corresponding j2 files.

Configuration types in the .j2 files

1. For loops- If you look at the .j2 template files, you may notice “for loops” in some places. Those for loops are there to add multiple/repetitive configurations. In the above example also, you see, we can add multiple custom formatters to the axis2.xml.

axis2.xml.j2:

{% for message_formatter in custom_message_formatters %}
        <messageFormatter contentType="{{message_formatter.content_type}}"
                          class="{{message_formatter.class}}"/>
     {% endfor %}

As per above, you can see that there is a for loop to iterate through each section called custom_message_formatters. Therefore, when configuring the “deployment. toml” file, you need to add grouped sections. We need to configure the particular parameter(e.g., custom_message_formatter) inside the double square brackets.

deployment. toml:

[[custom_message_formatters]]
content_type = ".*"
class = "org.wso2.carbon.relay.ExpandingMessageFormatter"
[[custom_message_formatters]]
content_type = "application/body"
class = "org.wso2.carbon.relay.ExpandingMessageFormatter"

Another example with custom message builders,

axis2.xml.j2

{% for message_builder in custom_message_builders %}
    <messageBuilder contentType="{{message_builder.content_type}}"
                          class="{{message_builder.class}}"/>
     {% endfor %}
    </messageBuilders>

deployment. toml:

[[custom_message_builders]]
content_type = ".*"
class = "org.wso2.carbon.relay.BinaryRelayBuilder"
[[custom_message_builders]]
content_type = "application/body"
class = "org.wso2.carbon.relay.ExpandingMessageBuilder"

2. Key-Value Pairs:

If you look at the “.j2 template” files, you may notice another set of for loops for adding key-value pairs in some places. Most of the time, this is there for adding multiple properties.

For example, let’s look at how we can set the properties in the http transport sender configuration in axis2_client.xml. This is how the section for http transport sender in axis2_client.xml.j2 file configuration looks like.

E.g., Let’s assume that you need to set the HTTP transport sender configuration properties in the axis2_client.xml file. The following is how the configuration for the HTTP transport sender appears in the axis2_client.xml.j2 file.

axis2_client.xml.j2:

<transportSender name="https" class="org.apache.axis2.transport.http.CommonsHTTPTransportSender">{% for name,value in transport.client.https.properties.items()%}<parameter name="{{name}}">{{value}}</parameter> {% endfor %} </transportSender>

In the above section, you can see a for loop to iterate through “transport.client.https.properties”.

As suggested early, First, let’s look at the default.json file to see the default properties getting added to this transport sender.

default.json:

"transport.client.http.properties.PROTOCOL": "HTTP/1.1", "transport.client.http.properties.Transfer-Encoding": "chunked", "transport.client.http.properties.SO_TIMEOUT": "60000", "transport.client.http.properties.CONNECTION_TIMEOUT": "60000",

Let’s assume that you need to increase the timeout values, which means you have to override these values via the “deployment. toml” file. In this case, you need to add those values as key-value pairs.

deployment. toml:

[transport.client.https.properties] 
SO_TIMEOUT = 120000 
CONNECTION_TIMEOUT = 120000

Common Errors

When starting the server, the product will assign the values in the “default.json” file to its respective parameters inside the multiple configuration files like axis2.xml, master-datasource.xml, registry.xml files …etc.

If we have set any values for any of those parameters(inside deployment. toml), the server will assign the value from “deployment. toml” instead of getting the values from the default.json file.

• Error 01: Wrong config

Therefore if you configured put parameters the wrong way on the toml file, then startup will fail and throw the errors below.

e.g.,

[2021-06-26 15:47:32,592] ERROR {org.wso2.config.mapper.TomlParser} - content_type previously defined at line 293, column 1 (line 295, column 1)
[2021-06-26 15:47:32,592] ERROR {org.wso2.config.mapper.TomlParser} - class previously defined at line 294, column 1 (line 296, column 1)
[2021-06-26 15:47:32,593]  SEVERE {org.wso2.carbon.server.Main handleConfiguration} - Error while performing configuration changes
org.wso2.config.mapper.ConfigParserException: Error parsing deployment configuration
at org.wso2.config.mapper.TomlParser.parse(TomlParser.java:140)
at org.wso2.config.mapper.ConfigParser.parse(ConfigParser.java:257)
at org.wso2.config.mapper.ConfigParser.deploy(ConfigParser.java:225)
at org.wso2.config.mapper.ConfigParser.deployAndStoreMetadata(ConfigParser.java:188)
at org.wso2.config.mapper.ConfigParser.parse(ConfigParser.java:135)
at org.wso2.carbon.server.Main.handleConfiguration(Main.java:236)
at org.wso2.carbon.server.Main.main(Main.java:107)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.wso2.carbon.bootstrap.Bootstrap.loadClass(Bootstrap.java:69)
at org.wso2.carbon.bootstrap.Bootstrap.main(Bootstrap.java:51)

The above error is telling; there is an issue at lines 295 & 296, then we have to check those lines and correct them.

example:

[custom_message_formatters]
content_type = ".*"
class = "org.wso2.carbon.relay.ExpandingMessageFormatter"
[custom_message_formatters]
content_type = "application/body"
class = "org.wso2.carbon.relay.ExpandingMessageFormatter"

The above config should correct by putting the parameters inside the double square brackets. Due to such missings, the above error(Error 01) can be thrown.

• Error 02: Configure child parameter before the Parent parameter

[2021-06-26 18:40:41,329]  WARN {org.wso2.config.mapper.ConfigParser} - Configurations Changed in :repository/conf/deployment.toml
[2021-06-26 18:40:41,331]  WARN {org.wso2.config.mapper.ConfigParser} - Overriding files in configuration directory /Users/user/Documents/wso2am-3.2.0
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.wso2.carbon.bootstrap.Bootstrap.loadClass(Bootstrap.java:69)
at org.wso2.carbon.bootstrap.Bootstrap.main(Bootstrap.java:51)
Caused by: java.lang.StackOverflowError
at net.consensys.cava.toml.MutableTomlTable.lambda$keyPathSet$1(MutableTomlTable.java:72)
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267)
at java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1696)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at net.consensys.cava.toml.MutableTomlTable.keyPathSet(MutableTomlTable.java:92)
at net.consensys.cava.toml.MutableTomlTable.lambda$keyPathSet$1(MutableTomlTable.java:80)
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267)
at java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1696)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)

e.g., If you configure the throttling configs as follows, you may get the above error.

• Wrong order:

[apim.throttling.blacklist_condition]
start_delay = "5m"
period = "1h"
[apim.throttling]
#enable_data_publishing = true
#enable_policy_deploy = true
#enable_blacklist_condition = true
#enable_persistence = true
throttle_decision_endpoints = ["tcp://localhost:5672","tcp://localhost:5672"]

The reason is that the “apim.throttling.blacklist_condition” parameter is a child parameter of “apim.throttling” parameter. Therefore when putting those config sections to the toml file, you need to put that parent parameter first, and after that, you need to put the child parameters. Otherwise, you will get the errors as above(error 02) when starting the server.

• Correct order:

[apim.throttling]
#enable_data_publishing = true
#enable_policy_deploy = true
#enable_blacklist_condition = true
#enable_persistence = true
throttle_decision_endpoints = ["tcp://localhost:5672","tcp://localhost:5672"]
[apim.throttling.blacklist_condition]
start_delay = "5m"
period = "1h"

You can get more details here[1] that is describing more regarding this model.

I hope this content will help you to understand the new configuration model of the WSO2 product. Appreciate your claps and feedback :).

Stay safe and Healthy !!!

References

[1]. https://apim.docs.wso2.com/en/latest/reference/understanding-the-new-configuration-model/