[WSO2] Clustering EI Analytics Profile

Image for post
Image for post

Setup

  • Enterprise Integrator 6.1.0

Data source Configuration

  1. Install Oracle database server.

Create

  • REGISTRY database (ei610regana),
drop user ei610regana cascade;
Create user ei610regana identified by ei610regana account unlock;
grant connect to ei610regana;
grant resource, create session, dba to ei610regana;
grant unlimited tablespace to ei610regana;
grant create table to ei610regana;
commit;
drop user ei610user cascade;
Create user ei610user identified by ei610user account unlock;
grant connect to ei610user;
grant resource, create session, dba to ei610user;
grant unlimited tablespace to ei610user;
grant create table to ei610user;
commit;
drop user anaevesto61 cascade;
Create user anaevesto61 identified by anaevesto61 account unlock;
grant connect to anaevesto61;
grant resource, create session, dba to anaevesto61;
grant unlimited tablespace to anaevesto61;
grant create table to anaevesto61;
commit;
drop user anaprodstor61 cascade;
Create user anaprodstor61 identified by anaprodstor61 account unlock;
grant connect to anaprodstor61;
grant resource, create session, dba to anaprodstor61;
grant unlimited tablespace to anaprodstor61;
grant create table to anaprodstor61;
commit;
drop user wso2carbonana1 cascade;
Create user wso2carbonana1 identified by wso2carbonana1 account unlock;
grant connect to wso2carbonana1;
grant resource, create session, dba to wso2carbonana1;
grant unlimited tablespace to wso2carbonana1;
grant create table to wso2carbonana1;
commit;
drop user wso2carbonana2 cascade;
Create user wso2carbonana2 identified by wso2carbonana2 account unlock;
grant connect to wso2carbonana2;
grant resource, create session, dba to wso2carbonana2;
grant unlimited tablespace to wso2carbonana2;
grant create table to wso2carbonana2;
commit;

Configure data source related configs in EI Analytic profile

3. Download Oracle JDBC driver. Add driver jar in to lib folder ($EI_HOME/lib)

4. Add carbon data source details in to master-datasources.xml ($EI_HOME/wso2/analytics/conf/datasources)

Replace <ip >with instance IP of which installed oracle server.

<datasources>
<datasource>
<name>WSO2_CARBON_DB</name>
<description>The datasource used for registry and user manager</description>
<jndiConfig>
<name>jdbc/WSO2CarbonDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:oracle:thin:@<ip>:1521/pdbora12c</url>
<username>wso2carbonana1</username>
<password>wso2carbonana1</password>
<driverClassName>oracle.jdbc.driver.OracleDriver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
<defaultAutoCommit>false</defaultAutoCommit>
</configuration>
</definition>
</datasource>
<datasource>
<name>WSO2_REG_DB</name>
<description>The datasource used for registry and user manager</description>
<jndiConfig>
<name>jdbc/WSO2RegDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:oracle:thin:@<ip>:1521/pdbora12c</url>
<username>ei610regana</username>
<password>ei610regana</password>
<driverClassName>oracle.jdbc.driver.OracleDriver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
<defaultAutoCommit>false</defaultAutoCommit>
</configuration>
</definition>
</datasource>
<datasource>
<name>WSO2_UM_DB</name>
<description>The datasource used for registry and user manager</description>
<jndiConfig>
<name>jdbc/WSO2UMDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:oracle:thin:@<ip>:1521/pdbora12c</url>
<username>ei610user</username>
<password>ei610user</password>
<driverClassName>oracle.jdbc.driver.OracleDriver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1 FROM DUAL</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
</datasources>

5. Add WSO2_ANALYTICS_EVENT_STORE_DB and WSO2_ANALYTICS_PROCESSED_DATA_STORE_DB data source details in to analytics-datasources.xml ($EI_HOME/wso2/analytics/conf/datasources).

Replace <ip >with instance IP of which installed oracle server.

<datasource>
<name>WSO2_ANALYTICS_EVENT_STORE_DB</name>
<description>The datasource used for analytics record store</description>
<definition type="RDBMS">
<configuration>
<url>jdbc:oracle:thin:@<ip>:1521/pdbora12c</url>
<username>anaevesto61</username>
<password>anaevesto61</password>
<driverClassName>oracle.jdbc.driver.OracleDriver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<validationQuery>SELECT 1 FROM DUAL</validationQuery>
<defaultAutoCommit>false</defaultAutoCommit>
<initialSize>0</initialSize>
<testWhileIdle>true</testWhileIdle>
<minEvictableIdleTimeMillis>4000</minEvictableIdleTimeMillis>
<defaultTransactionIsolation>READ_COMMITTED</defaultTransactionIsolation>
</configuration>
</definition>
</datasource>
<datasource>
<name>WSO2_ANALYTICS_PROCESSED_DATA_STORE_DB</name>
<description>The datasource used for analytics record store</description>
<definition type="RDBMS">
<configuration>
<url>jdbc:oracle:thin:@<ip>:1521/pdbora12c</url>
<username>anaprodstor61</username>
<password>anaprodstor61</password>
<driverClassName>oracle.jdbc.driver.OracleDriver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<validationQuery>SELECT 1 FROM DUAL</validationQuery>
<defaultAutoCommit>false</defaultAutoCommit>
<initialSize>0</initialSize>
<testWhileIdle>true</testWhileIdle>
<minEvictableIdleTimeMillis>4000</minEvictableIdleTimeMillis>
<defaultTransactionIsolation>READ_COMMITTED</defaultTransactionIsolation>
</configuration>
</definition>
</datasource>
</datasources>

6. To share the user store among the nodes, modify user-mgt.xml ($EI_HOME/wso2/analytics/conf) as follows:

[Change <Property name=”dataSource”> as given]

 <Property name=”dataSource”>jdbc/WSO2UMDB</Property>

7. Add or modify “govregistry” datasource attribute in registry.xml ($EI_HOME/wso2/analytics/conf) as follows:

Replace <ip >with instance IP of which installed oracle server.

<dbConfig name="wso2registry">
<dataSource>jdbc/WSO2CarbonDB</dataSource>
</dbConfig>
<dbConfig name="govregistry">
<dataSource>jdbc/WSO2RegDB</dataSource>
</dbConfig>
<remoteInstance url="https://localhost:9443/registry">
<id>gov</id>
<cacheId>jdbc:oracle:thin:@<ip>:1521/pdbora12c</cacheId>
<dbConfig>govregistry</dbConfig>
<readOnly>false</readOnly>
<enableCache>true</enableCache>
<registryRoot>/</registryRoot>
</remoteInstance>
<mount path="/_system/governance" overwrite="true">
<instanceId>gov</instanceId>
<targetPath>/_system/governance</targetPath>
</mount>
<mount path="/_system/config" overwrite="true">
<instanceId>gov</instanceId>
<targetPath>/_system/config</targetPath>
</mount>

Configure clustering related configs in EI Analytic profile

8. Edit axis2.xml ($EI_HOME/wso2/analytics/conf/axis2/) in each node as follows.

  • Enable clustering.
<clustering class = "org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
  • Enable wka mode.
<parameter name="membershipScheme">wka</parameter>
  • Add other analytics profile nodes details under members tag.
<members><member><hostName>[node1 IP]</hostName><port>[node1 port]</port></member><member><hostName>[node2 IP]</hostName><port>[node2 port]</port></member></members>

Sample Config:

<members>
<member>
<hostName>192.168.**.***</hostName>
<port>4100</port>
</member>
</members>
  • For each node, enter the respective server IP address as the value for the localMemberHost property

<parameter name="localMemberHost">[Server_IP_Address]</parameter>

9. Configure the event-processor.xml ($EI_HOME/wso2/analytics/conf/) file as follows to cluster CEP in the Receiver.

  • Enable the HA mode

<mode name="HA" enable="true">

  • Disable the Distributed mode

<mode name="Distributed" enable="false">

  • For each node, enter the respective server IP address under the HA mode Config section.
<eventSync>
<hostName>*.*.*.*</hostName>

10. Update spark-defaults.conf ($EI_HOME/wso2/analytics/conf/analytics /spark/) file as follows to use the Spark cluster embedded within Analytic profile.

  • Keep the carbon.spark.master configuration as local. This instructs Spark to create a Spark cluster using the Hazelcast cluster.
carbon.spark.master local
carbon.spark.master.count 2
  • Create a symbolic link to <$AnalyticProfile_Home>in both nodes, where paths of those symbolic links are identical. This ensures that if we use the symbolic link to access the analytic profile, we can use a common path.

In this scenario we are having analytics profile in two different instances which are having two different OSes. Therefore choose a common directory as follows:

/usr/share/das/das_symlink

Use following command to create symlink folder.

ln -s <$AnalyticProfile_Home> <$Symlink_Folder>

Sample config:

ln -s /home/debian/analytics2/wso2ei-6.1.0/wso2/analytics /usr/share/das/das_symlink61

Give permission to that folder to have access while creating analytics cluster. (User group and user name should be change accordingly). Use following command for that.

sudo chown -R username:group directory

Ex: sudo chown -R cloud-user:cloud-user /user/share/das/das_symlink61

Check for updated permission for the folder using command ls -al

[cloud-user@ei6s2-analytics1 das]$ ls -la

Sample output:

lrwxrwxrwx. 1 cloud-user cloud-user 69 මාර් 17 11:35 das_symlink61 -> /home/cloud-user/EI610Analytics1/wso2ei-6.1.0-SNAPSHOT/wso2/analytics

Now you are done with creating symlink.

Add symlink in spark-defaults.conf file.

carbon.das.symbolic.link /usr/share/das/das_symlink61/

11. Add followings in to wso2server.sh ($EI_Home/wso2/analytics/bin/)

-Dcarbon.das.symlink.config.dir.path=<SYM_LINK_PATH>/conf
-Dcarbon.das.symlink.external.lib.dir.path=<SYM_LINK_PATH>/../../lib
-Dcarbon.das.symlink.dropins.dir.path=<SYM_LINK_PATH>/../../dropins
-Dcarbon.das.symlink.internal.lib.dir.path=<SYM_LINK_PATH>/../lib
-Ddas.symlink.components.repo=<SYM_LINK_PATH>/../components/plugins

Sample config:

ANA_HOME=”/usr/share/das/das_symlink61"
...
-Dcarbon.das.symlink.config.dir.path=$ANA_HOME/conf \
-Dcarbon.das.symlink.external.lib.dir.path=$ANA_HOME/../../lib \
-Dcarbon.das.symlink.dropins.dir.path=$ANA_HOME/../../dropins \
-Dcarbon.das.symlink.internal.lib.dir.path=$ANA_HOME/../lib \
-Ddas.symlink.components.repo=$ANA_HOME/../components/plugins \

12. Add following in to load-spark-env-vars.sh file ($EI_Home/wso2/analytics/bin/).

export CARBON_CONFIG_DIR_PATH=/usr/share/das/das_symlink61/conf

Note: Replace path with given path.

You are done with configurations :) .

When you started, you can check for spark cluster to identify status. Default url for spark cluster : <ip_instance>:8082/

Sample URL:

http://192.168.**.**:8082/

Note: As we didn’t configure the deployment synchronizer, it is required to deploy any C-App use in the Analytics profile cluster set up to both the nodes.

Important Notes

  1. When changing datasources/clean up datasources within cluster, you should have to clean following folders too. ( rm -r * ).
  • solr/*

2. When you change symplinks, clear above folders and additionally followings too.

  • work/*

3. When you are going start cluster, start one node and in parallel you can start other node too. Otherwise you will get known issue as follows:

ERROR {org.wso2.carbon.event.processor.manager.core.internal.HAManager} -  CEP HA State syncing failed, No execution plans exist for tenant  -1234
org.wso2.carbon.event.processor.manager.core.exception.EventManagementException: No execution plans exist for tenant -1234
at org.wso2.carbon.event.processor.core.internal.CarbonEventProcessorManagementService.restoreState(CarbonEventProcessorManagementService.java:83)
at org.wso2.carbon.event.processor.manager.core.internal.HAManager.syncState(HAManager.java:336)
at org.wso2.carbon.event.processor.manager.core.internal.HAManager.access$100(HAManager.java:49)
at org.wso2.carbon.event.processor.manager.core.internal.HAManager$2.run(HAManager.java:276)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Refer [1] for more details and for known issues.

References:

  1. https://docs.wso2.com/display/CLUSTER44x/Minimum+High+Availability+Deployment+-+DAS+3.1.0

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store