Go to the profile of Pubudu D.P.
Pubudu D.P.

WSO2 API Manager — How to get rid of SVN related issues in a clustered deployment

When WSO2 API Manager is being used in a clustered environment it, is essential to use the Deployment Synchronizer. Deployment Synchronizer is responsible for maintaining and serving svn artifacts checked in via publisher amongst API Manager gateway nodes. This is a lot clearer if you take a look at the following clustering pattern.

In this setup, the user can publish an API to an external svn location (not shown in picture). Once the artifact is published, It is required to serve these artifacts when a request is made. This is where the gateway workers come into play. What gateway workers do is, to serve these APIs/artifacts when requested. In order to to this, gateway workers need to maintain a mechanism how to keep the artifacts up to date within each node.

Even though the serving part is done by the gateway workers, gateway manager is responsible for sending requests for the most appropriate worker node. Therefore, Deployment Synchronization should be configured for all the nodes in gateway cluster (i.e gateway manager and worker nodes).

Once the svn deployment synchronizer is successfully configured, sometimes it is possible to run into issues where workers are unable to keep the artifacts in sync. We can solve this by re-synchronizing the artifacts in svn location of the gateway nodes.

These artifacts are kept within $[Home]/repository/deployment/server folder. If you run into any svn related exceptions it is possible to solve it by using the approach given below.

1. Remove the .svn folder in $[HOME]/repository/deployment/server folder in each node in gateway cluster and then restart the servers. (if you have created tenants you will have to remove the .svn folder from $[HOME]/repository/tenants as well)

2. If the above approach didn’t solve the issue, then you can try to delete the server folder from each node (make sure to keep backup) and restart the servers. This will re-sync the artifacts back to gateway worker nodes during the startup.

3. If none of the above works it leaves us with the last option, where we will have to create a new svn location, and then add it to the carbon.xml file in each gateway worker nodes and manager node, replace the $[HOME]/repository/deployment/server folder and then restart all the servers in gateway cluster. The risk of this approach is that you will loose the artifacts created at the previous location and will have to republish those to the new location.

If you have created tenants and published artifacts, please keep in mind you have to clean everything in $[Home]/tenants folder from each gateway node (both manager and workers) before restarting the gateway cluster nodes

One of the issues I have seen related to Deployment Synchronization and were able to troubleshoot using these methods is shown below.

TID: [-1234] [] [2015–11–21 03:47:58,462] ERROR {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask} — Deployment synchronization update for tenant -1234 failed {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask}
java.lang.RuntimeException: org.wso2.carbon.deployment.synchronizer.DeploymentSynchronizerException: Error while attempting to create the directory: http://xxx.xxx.xxx.xxx/svn/testrepo/apim/-1234
at org.wso2.carbon.deployment.synchronizer.internal.DeploymentSynchronizerServiceImpl.update(DeploymentSynchronizerServiceImpl.java:98)
at org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.deploymentSyncUpdate(CarbonDeploymentSchedulerTask.java:179)
at org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.run(CarbonDeploymentSchedulerTask.java:137)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.wso2.carbon.deployment.synchronizer.DeploymentSynchronizerException: Error while attempting to create the directory: http://192.168.57.233/svn/testrepo/apim/-1234
at org.wso2.carbon.deployment.synchronizer.subversion.SVNBasedArtifactRepository.handleException(SVNBasedArtifactRepository.java:831)
at org.wso2.carbon.deployment.synchronizer.subversion.SVNBasedArtifactRepository.checkRemoteDirectory(SVNBasedArtifactRepository.java:250)
at org.wso2.carbon.deployment.synchronizer.subversion.SVNBasedArtifactRepository.init(SVNBasedArtifactRepository.java:175)
at org.wso2.carbon.deployment.synchronizer.repository.CarbonRepositoryUtils.newCarbonRepositorySynchronizer(CarbonRepositoryUtils.java:67)
at org.wso2.carbon.deployment.synchronizer.internal.DeploymentSynchronizerServiceImpl.update(DeploymentSynchronizerServiceImpl.java:76)
… 9 more
Caused by: org.tigris.subversion.svnclientadapter.SVNClientException: org.apache.subversion.javahl.ClientException: svn: E175002: timed out waiting for server
svn: E175002: OPTIONS request failed on ‘/svn/testrepo/apim’
at org.tigris.subversion.svnclientadapter.javahl.AbstractJhlClientAdapter.mkdir(AbstractJhlClientAdapter.java:2733)
at org.wso2.carbon.deployment.synchronizer.subversion.SVNBasedArtifactRepository.checkRemoteDirectory(SVNBasedArtifactRepository.java:248)
… 12 more
Caused by: org.apache.subversion.javahl.ClientException: svn: E175002: timed out waiting for server
svn: E175002: OPTIONS request failed on ‘/svn/testrepo/apim’
at org.apache.subversion.javahl.ClientException.fromException(ClientException.java:68)
at org.tmatesoft.svn.core.javahl17.SVNClientImpl.getClientException(SVNClientImpl.java:1492)
at org.tmatesoft.svn.core.javahl17.SVNClientImpl.mkdirRemote(SVNClientImpl.java:2294)
at org.tmatesoft.svn.core.javahl17.SVNClientImpl.mkdir(SVNClientImpl.java:597)
at org.tigris.subversion.svnclientadapter.javahl.AbstractJhlClientAdapter.mkdir(AbstractJhlClientAdapter.java:2730)
… 13 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: timed out waiting for server
svn: E175002: OPTIONS request failed on ‘/svn/testrepo/apim’
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:106)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:90)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:775)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:375)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:363)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:710)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1032)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.checkPath(DAVRepository.java:214)
at org.tmatesoft.svn.core.internal.wc2.remote.SvnRemoteRemoteMkDir.addURLParents(SvnRemoteRemoteMkDir.java:156)
at org.tmatesoft.svn.core.internal.wc2.remote.SvnRemoteRemoteMkDir.doRun(SvnRemoteRemoteMkDir.java:65)
at org.tmatesoft.svn.core.internal.wc2.remote.SvnRemoteRemoteMkDir.run(SvnRemoteRemoteMkDir.java:39)
at org.tmatesoft.svn.core.internal.wc2.remote.SvnRemoteRemoteMkDir.run(SvnRemoteRemoteMkDir.java:35)
at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1235)
at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
at org.tmatesoft.svn.core.javahl17.SVNClientImpl.mkdirRemote(SVNClientImpl.java:2292)
… 15 more
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.tmatesoft.svn.core.internal.util.SVNSocketConnection.run(SVNSocketConnection.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
… 3 more
TID: [-1234] [] [2015–11–21 03:48:13,514] WARN {org.wso2.carbon.deployment.synchronizer.internal.DeploymentSynchronizerServiceImpl} — A repository synchronizer has not been engaged for the file path: /mnt/192.168.48.40/gateway/wso2am-1.10.0-SNAPSHOT/repository/deployment/server/ {org.wso2.carbon.deployment.synchronizer.internal.DeploymentSynchronizerServiceImpl}