[WSO2] Clustering EI Analytics Profile

Setup

  • Enterprise Integrator 6.1.0
  • Analytics profile configure with Oracle 12 c database
  • Analytics two nodes in separate instances

Data source Configuration

  1. Install Oracle database server.
  2. Create databases in Oracle.

Create

  • REGISTRY database (ei610regana),
  • USER MANAGEMENT database (),
  • WSO2_ANALYTICS_EVENT_STORE_DB (anaevesto61),
  • WSO2_ANALYTICS_PROCESSED_DATA_STORE_DB (anaprodstor61) and
  • Two carbon databases (wso2carbonana1, wso2carbonana2) as we need 2 local databases for analytics nodes.
drop user ei610regana cascade;
Create user ei610regana identified by ei610regana account unlock;
grant connect to ei610regana;
grant resource, create session, dba to ei610regana;
grant unlimited tablespace to ei610regana;
grant create table to ei610regana;
commit;
drop user ei610user cascade;
Create user ei610user identified by ei610user account unlock;
grant connect to ei610user;
grant resource, create session, dba to ei610user;
grant unlimited tablespace to ei610user;
grant create table to ei610user;
commit;
drop user anaevesto61 cascade;
Create user anaevesto61 identified by anaevesto61 account unlock;
grant connect to anaevesto61;
grant resource, create session, dba to anaevesto61;
grant unlimited tablespace to anaevesto61;
grant create table to anaevesto61;
commit;
drop user anaprodstor61 cascade;
Create user anaprodstor61 identified by anaprodstor61 account unlock;
grant connect to anaprodstor61;
grant resource, create session, dba to anaprodstor61;
grant unlimited tablespace to anaprodstor61;
grant create table to anaprodstor61;
commit;
drop user wso2carbonana1 cascade;
Create user wso2carbonana1 identified by wso2carbonana1 account unlock;
grant connect to wso2carbonana1;
grant resource, create session, dba to wso2carbonana1;
grant unlimited tablespace to wso2carbonana1;
grant create table to wso2carbonana1;
commit;
drop user wso2carbonana2 cascade;
Create user wso2carbonana2 identified by wso2carbonana2 account unlock;
grant connect to wso2carbonana2;
grant resource, create session, dba to wso2carbonana2;
grant unlimited tablespace to wso2carbonana2;
grant create table to wso2carbonana2;
commit;

Configure data source related configs in EI Analytic profile

3. Download Oracle JDBC driver. Add driver jar in to lib folder ($EI_HOME/lib)

4. Add carbon data source details in to master-datasources.xml ($EI_HOME/wso2/analytics/conf/datasources)

Replace <ip >with instance IP of which installed oracle server.

<datasources>
<datasource>
<name>WSO2_CARBON_DB</name>
<description>The datasource used for registry and user manager</description>
<jndiConfig>
<name>jdbc/WSO2CarbonDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:oracle:thin:@<ip>:1521/pdbora12c</url>
<username>wso2carbonana1</username>
<password>wso2carbonana1</password>
<driverClassName>oracle.jdbc.driver.OracleDriver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
<defaultAutoCommit>false</defaultAutoCommit>
</configuration>
</definition>
</datasource>
<datasource>
<name>WSO2_REG_DB</name>
<description>The datasource used for registry and user manager</description>
<jndiConfig>
<name>jdbc/WSO2RegDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:oracle:thin:@<ip>:1521/pdbora12c</url>
<username>ei610regana</username>
<password>ei610regana</password>
<driverClassName>oracle.jdbc.driver.OracleDriver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
<defaultAutoCommit>false</defaultAutoCommit>
</configuration>
</definition>
</datasource>
<datasource>
<name>WSO2_UM_DB</name>
<description>The datasource used for registry and user manager</description>
<jndiConfig>
<name>jdbc/WSO2UMDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:oracle:thin:@<ip>:1521/pdbora12c</url>
<username>ei610user</username>
<password>ei610user</password>
<driverClassName>oracle.jdbc.driver.OracleDriver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1 FROM DUAL</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
</datasources>

5. Add WSO2_ANALYTICS_EVENT_STORE_DB and WSO2_ANALYTICS_PROCESSED_DATA_STORE_DB data source details in to analytics-datasources.xml ($EI_HOME/wso2/analytics/conf/datasources).

Replace <ip >with instance IP of which installed oracle server.

<datasource>
<name>WSO2_ANALYTICS_EVENT_STORE_DB</name>
<description>The datasource used for analytics record store</description>
<definition type="RDBMS">
<configuration>
<url>jdbc:oracle:thin:@<ip>:1521/pdbora12c</url>
<username>anaevesto61</username>
<password>anaevesto61</password>
<driverClassName>oracle.jdbc.driver.OracleDriver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<validationQuery>SELECT 1 FROM DUAL</validationQuery>
<defaultAutoCommit>false</defaultAutoCommit>
<initialSize>0</initialSize>
<testWhileIdle>true</testWhileIdle>
<minEvictableIdleTimeMillis>4000</minEvictableIdleTimeMillis>
<defaultTransactionIsolation>READ_COMMITTED</defaultTransactionIsolation>
</configuration>
</definition>
</datasource>
<datasource>
<name>WSO2_ANALYTICS_PROCESSED_DATA_STORE_DB</name>
<description>The datasource used for analytics record store</description>
<definition type="RDBMS">
<configuration>
<url>jdbc:oracle:thin:@<ip>:1521/pdbora12c</url>
<username>anaprodstor61</username>
<password>anaprodstor61</password>
<driverClassName>oracle.jdbc.driver.OracleDriver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<validationQuery>SELECT 1 FROM DUAL</validationQuery>
<defaultAutoCommit>false</defaultAutoCommit>
<initialSize>0</initialSize>
<testWhileIdle>true</testWhileIdle>
<minEvictableIdleTimeMillis>4000</minEvictableIdleTimeMillis>
<defaultTransactionIsolation>READ_COMMITTED</defaultTransactionIsolation>
</configuration>
</definition>
</datasource>
</datasources>

6. To share the user store among the nodes, modify user-mgt.xml ($EI_HOME/wso2/analytics/conf) as follows:

[Change <Property name=”dataSource”> as given]

 <Property name=”dataSource”>jdbc/WSO2UMDB</Property>

7. Add or modify “govregistry” datasource attribute in registry.xml ($EI_HOME/wso2/analytics/conf) as follows:

Replace <ip >with instance IP of which installed oracle server.

<dbConfig name="wso2registry">
<dataSource>jdbc/WSO2CarbonDB</dataSource>
</dbConfig>
<dbConfig name="govregistry">
<dataSource>jdbc/WSO2RegDB</dataSource>
</dbConfig>
<remoteInstance url="https://localhost:9443/registry">
<id>gov</id>
<cacheId>jdbc:oracle:thin:@<ip>:1521/pdbora12c</cacheId>
<dbConfig>govregistry</dbConfig>
<readOnly>false</readOnly>
<enableCache>true</enableCache>
<registryRoot>/</registryRoot>
</remoteInstance>
<mount path="/_system/governance" overwrite="true">
<instanceId>gov</instanceId>
<targetPath>/_system/governance</targetPath>
</mount>
<mount path="/_system/config" overwrite="true">
<instanceId>gov</instanceId>
<targetPath>/_system/config</targetPath>
</mount>

Configure clustering related configs in EI Analytic profile

8. Edit axis2.xml ($EI_HOME/wso2/analytics/conf/axis2/) in each node as follows.

  • Enable clustering.
<clustering class = "org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
  • Enable wka mode.
<parameter name="membershipScheme">wka</parameter>
  • Add other analytics profile nodes details under members tag.
<members>
<member>
<hostName>[node1 IP]</hostName>
<port>[node1 port]</port>
</member>
<member>
<hostName>[node2 IP]</hostName>
<port>[node2 port]</port>
</member>
</members>

Sample Config:

<members>
<member>
<hostName>192.168.**.***</hostName>
<port>4100</port>
</member>
</members>
  • For each node, enter the respective server IP address as the value for the localMemberHost property

<parameter name="localMemberHost">[Server_IP_Address]</parameter>

9. Configure the event-processor.xml ($EI_HOME/wso2/analytics/conf/) file as follows to cluster CEP in the Receiver.

  • Enable the HA mode

<mode name="HA" enable="true">

  • Disable the Distributed mode

<mode name="Distributed" enable="false">

  • For each node, enter the respective server IP address under the HA mode Config section.
<eventSync>
<hostName>*.*.*.*</hostName>

10. Update spark-defaults.conf ($EI_HOME/wso2/analytics/conf/analytics /spark/) file as follows to use the Spark cluster embedded within Analytic profile.

  • Keep the carbon.spark.master configuration as local. This instructs Spark to create a Spark cluster using the Hazelcast cluster.
  • Enter 2 as the value for the carbon.spark.master.count configuration. This specifies that there should be two masters in the Spark cluster. One master serves as an active master and the other serves as a stand-by master. (Change count according to number of nodes)
carbon.spark.master local
carbon.spark.master.count 2
  • Create a symbolic link to <$AnalyticProfile_Home>in both nodes, where paths of those symbolic links are identical. This ensures that if we use the symbolic link to access the analytic profile, we can use a common path.

In this scenario we are having analytics profile in two different instances which are having two different OSes. Therefore choose a common directory as follows:

/usr/share/das/das_symlink

Use following command to create symlink folder.

ln -s <$AnalyticProfile_Home> <$Symlink_Folder>

Sample config:

ln -s /home/debian/analytics2/wso2ei-6.1.0/wso2/analytics /usr/share/das/das_symlink61

Give permission to that folder to have access while creating analytics cluster. (User group and user name should be change accordingly). Use following command for that.

sudo chown -R username:group directory

Ex: sudo chown -R cloud-user:cloud-user /user/share/das/das_symlink61

Check for updated permission for the folder using command ls -al

[cloud-user@ei6s2-analytics1 das]$ ls -la

Sample output:

lrwxrwxrwx. 1 cloud-user cloud-user 69 මාර් 17 11:35 das_symlink61 -> /home/cloud-user/EI610Analytics1/wso2ei-6.1.0-SNAPSHOT/wso2/analytics

Now you are done with creating symlink.

Add symlink in spark-defaults.conf file.

carbon.das.symbolic.link /usr/share/das/das_symlink61/

11. Add followings in to wso2server.sh ($EI_Home/wso2/analytics/bin/)

-Dcarbon.das.symlink.config.dir.path=<SYM_LINK_PATH>/conf
-Dcarbon.das.symlink.external.lib.dir.path=<SYM_LINK_PATH>/../../lib
-Dcarbon.das.symlink.dropins.dir.path=<SYM_LINK_PATH>/../../dropins
-Dcarbon.das.symlink.internal.lib.dir.path=<SYM_LINK_PATH>/../lib
-Ddas.symlink.components.repo=<SYM_LINK_PATH>/../components/plugins

Sample config:

ANA_HOME=”/usr/share/das/das_symlink61"
...
-Dcarbon.das.symlink.config.dir.path=$ANA_HOME/conf \
-Dcarbon.das.symlink.external.lib.dir.path=$ANA_HOME/../../lib \
-Dcarbon.das.symlink.dropins.dir.path=$ANA_HOME/../../dropins \
-Dcarbon.das.symlink.internal.lib.dir.path=$ANA_HOME/../lib \
-Ddas.symlink.components.repo=$ANA_HOME/../components/plugins \

12. Add following in to load-spark-env-vars.sh file ($EI_Home/wso2/analytics/bin/).

export CARBON_CONFIG_DIR_PATH=/usr/share/das/das_symlink61/conf

Note: Replace path with given path.

You are done with configurations :) .

When you started, you can check for spark cluster to identify status. Default url for spark cluster : <ip_instance>:8082/

Sample URL:

http://192.168.**.**:8082/

Note: As we didn’t configure the deployment synchronizer, it is required to deploy any C-App use in the Analytics profile cluster set up to both the nodes.

Important Notes

  1. When changing datasources/clean up datasources within cluster, you should have to clean following folders too. ( rm -r * ).
  • solr/*
  • tmp/*
  • repository/data/*
  • repository/database/*

2. When you change symplinks, clear above folders and additionally followings too.

  • work/*

3. When you are going start cluster, start one node and in parallel you can start other node too. Otherwise you will get known issue as follows:

ERROR {org.wso2.carbon.event.processor.manager.core.internal.HAManager} -  CEP HA State syncing failed, No execution plans exist for tenant  -1234
org.wso2.carbon.event.processor.manager.core.exception.EventManagementException: No execution plans exist for tenant -1234
at org.wso2.carbon.event.processor.core.internal.CarbonEventProcessorManagementService.restoreState(CarbonEventProcessorManagementService.java:83)
at org.wso2.carbon.event.processor.manager.core.internal.HAManager.syncState(HAManager.java:336)
at org.wso2.carbon.event.processor.manager.core.internal.HAManager.access$100(HAManager.java:49)
at org.wso2.carbon.event.processor.manager.core.internal.HAManager$2.run(HAManager.java:276)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Refer [1] for more details and for known issues.

References:

  1. https://docs.wso2.com/display/CLUSTER44x/Minimum+High+Availability+Deployment+-+DAS+3.1.0