How to build a CI/CD pipeline for WSO2 ESB (WSO2 EI)

This article describes implementing a fully automated CI/CD process for WSO2 ESB (or WSO2 EI Integration profile). Since WSO2 Enterprise Integrator (EI) has become the successor of WSO2 ESB, I will be using both terms interchangeably within this article. The content is applicable for both versions.

WSO2 ESB is an integration product which can be used to build integrations within your enterprise using a configuration based approach. All your integrations can be implemented using synapse DSL which is an XML configuration language. Since it is XML, users can use any code editor or IDE to build these integration configurations. But WSO2 provides a tool called WSO2 Developer Studio (or WSO2 EI Tooling) which is an IDE built on top of eclipse. This IDE provides graphical editing of these integrations on top of source editing of XML files. It is recommended to use this tool for integration development.

Let’s think about a typical enterprise integration scenario with several developers building these integrations and you have a staging and a production environment which are built with latest WSO2 Enterprise Integrator (or WSO2 ESB) nodes. You can use WSO2 Update Manager (WUM) to make these nodes up to date. You need to automate the entire development/testing and deployment process so that whenever there is a change happens to the integration code from a developer, that change is deployed into staging, tested on staging and if succeeded only deployed into production. If you have more than one non-production environments (e.g. DEV/QA/UAT), you can add any number of environments into this pipeline.

WSO2 has come up with the idea of using maven for building integration artefacts into deployable units. A Composite ARchive (CAR) is a single deployable unit which contains all the artefacts which you develop for a certain integration use case. It can contain synapse configurations, connector artefacts, registry configurations, data services, custom mediators, security policies and any other configuration files which are required for runtime.

Let’s see how we can integrate these concepts into a single process which can be used to build a fully automated CICD pipeline.

WSO2 ESB CICD pipeline

As depicted in the above figure

1. Developers will develop integrations using WSO2 Developer Studio. The source code of the integrations will be maintained at a source repository like Github.
2. A Continuous Integration tool like Jenkins or TravisCI is configured to listen to this source repository for any change. If there is any change to the source code of the master branch, it will trigger a new build which will execute a maven build process.
3. The CI tool will build the maven project and use maven-car-deploy plugin to deploy the CAR files to the staging environment.
4. The CI tool will trigger a testing process by executing a set of testing scripts using a tool like newman.
5. This testing tool will eventually run a set of postman scripts to run tests on the staging servers and verify the results.
6. Once the testing is completed successfully, the CI tool will build and deploy the CAR files to the production environment.

The above process defines a clear step by step guide to implementing a CICD process for WSO2 ESB. It is easier to define a process than actually implementing it. Let’s see how we can actually implement this process using the tools which are mentioned above.

I’m using a simple proxy service which calls a backend and returns that response back to the client here. I’m using 2 WSO2 EI servers which are running in my local machine to mimic the 2 environments staging and production. There are several ways you can keep environment specific details within WSO2 ESB. In this case, I’m going to use separate registry projects to keep my environment specific details (endpoint address). I’m creating a “maven multi module” project from the WSO2 Developer Studio to include all the other projects. Under this project, I’m creating the following projects.

• ESB Config project
• Connector exporter project
• Registry resource project for staging
• Registry resource project for production
• Composite archive project (CAR) for staging
• Composite archive project (CAR) for production

Using the above projects, I’m going to create 2 separate CAR files for staging and production environments. Each of these CAR files contains

ESB config + Connector exporter + Registry resource project of the environment.

Once I have created the above projects, I’m going to implement the proxy service within the ESB config project and the endpoint URL is stored within registry projects for each environment. Within the CAR projects, I’m configuring the maven-car-deploy plugin to deploy the CAR files into the relevant environment.

I’m using “newman” to run tests for proxy service using postman scripts which are implemented in json files. The above mentioned 6 step process is defined within a shell script. In this implementation, I have skipped the Jenkins automation part. To showcase the process and the flow, you need to run this shell script manually. But if you have Jenkins or any other CI tool, it is just a matter of executing this shell script through that when there is a code change in the source repository.

All the source code and the instructions on running this process locally are described in the following Github repository. Clone it, run it and give it a star if you like it !.

chanakaudaya/WSO2-ESB-CICD