Building a CI/CD Pipeline for WSO2 Enterprise Integrator using GitLab Tools

WSO2 Enterprise Integrator is a comprehensive integration solution builder that comes with a handy set of development tools which provides a better developer experience to integration solution designers. If you have been working with WSO2 Enterprise Integrator (Formerly WSO2 Enterprise Service Bus) you would know that it comes with an Eclipse-based development studio which enables you to develop ESB Solution Projects as maven modules and package and deploy as a composite archive (CAR) file to the Enterprise Integrator.

Developer experience can be further improved and made efficient by implementing a CI/CD pipeline to deploy and test implemented integration solutions. Also, a CI/CD will hugely beneficial to manage multiple integration projects.

Let’s Begin

First set of tools to build this CI/CD pipeline comes with the WSO2 itself, which are developer studio and maven based project structure and maven-car-plugin which helps develop, build and package WSO2 EI Solutions.

For the next steps, I will be using the following tools,

1. Source Code Management — GitLab
2. Automated Testing — Postman/Newman. Alternatively, Apache JMeter or SoapUI can be used
3. Deploying — Ansible. Alternatively, maven-car-deploy-plugin can be used.
4. Pipeline Definition and Execution — GitLab CI/CD tools

I will be discussing how to build a basic pipeline assuming that WSO2 EI is deployed in multiple environments Test (QA), Staging and Production environments. Following flow chart describes the pipeline flow which will build, package and then deploy to Test, Staging and Production environments automatically.

CI/CD Pipeline Flow Chart

Let’s Implement

Step 1

First, we need to configure GitLab Runner which will be acting as a coordinator for the build jobs and listen to the source code repository for commits. There are three types of runners which are shared, specific and group runners. You can install the Gitlab Runner by going through the following documentation.

Note: If you decide to use shared runners, there is no need to install or register the runners. Therefore you can skip Step 1 and Step 2.

Step 2

After you have properly installed GitLab Runner, register a single runner or multiple runners with your project if you need to run parallel builds by following this documentation. When registering runners, you will be asked to choose an executor. There are multiple executor types are supported in GitLab Runner. I recommend using the docker executor which will spawn a Docker container for every job within the pipeline.

Step 3

Let’s now get into pipeline configuration. GitLab CI/CD pipelines are configured using a YAML file within each project.

I have created the following project demonstrating the pipeline implementation using GitLab Runner docker executor.

Madawa Soysa / wso2ei-cicd-demo

The above project contains a WSO2 ESB Solution Project which contains a simple REST API which echoes a message. The project also contains tests written using Postman as well as the ansible playbook which is used to deploy the CAR archive to WSO2 Servers.

Note: Related ansible source can be maintained in a different repository and can be refered from each project instead of duplicating the playbooks in each project.

Let’s have a closer look at the CI/CD configuration for the project.

The first line specifies the docker image to spawn the container and run the pipeline on.

image: registry.gitlab.com/madawa/wso2ei-artifact-builder:ubuntu_latest

In this configuration, I have used a custom image I created including all the packages (Java, Maven, Newman, Ansible) needed to build, test and deploy WSO2 EI artefacts. You can either use the same image or build a custom image as per the scenario. The image and the related docker files can be found in the below repository. The same image can be pulled from DockerHub as well.

Madawa Soysa / wso2ei-artifact-builder

Next section of the configuration includes the stages of the pipeline and these will be executed in order.

stages:
  - build
  - qa
  - test

Note: GitLab CI has default stages build, test and deploy and does not need to specify when default stages are used. However, in this case, we need to deploy the CAR to an existing environment before executing tests.

Next, the jobs are configured. Jobs define the tasks to run in each stage. Jobs will run in the order of the stages. For example, jobs belong to build will run before jobs belong to qa. There can be multiple jobs that belong to a single-stage which will run in parallel given that there are multiple runners available.

Building

This will build the project using maven and archives all the CAR files inside the project and share those with dependent jobs.

Deploying to QA

In the second job, we will be deploying the built CAR archives to the QA environment. This may seem different from a standard CI pipeline as here we do a deployment before testing. Since this is an integration solution, the actual systems should be available for testing. Hence we will first deploy the artefacts to one of the QA environments and run tests against that in the next stage.

Testing

This job will execute the tests against the deployed artifact on the QA environment. The idea here is to run tests belong to a single project which is equivalent to unit testing in the standard development lifecycle. This job will make sure that the solution is working as expected.

Extending the Pipeline

Up to now, we have built a CI/CD pipeline to build, deploy to QA and run unit tests against the QA environment. Depending on the requirements, we can extend the pipeline to manage the full development cycle until releasing to production. It is just a matter of adding a few more job definitions with some conditions to trigger only when a release is happening.

When specifying jobs in the GitLab CI configuration, we can use only/except parameter to specify conditions to run jobs. Here we can use tags to identify releases.

deploy_staging:
  stage: staging
  only:
    - tags

When configured as above, the job will only run when the commit is associated with a tag.

Sample CI/CD Configuration

This is only a simple example of adding conditions. GitLab CI/CD offers advanced configurations to support advanced use cases.

Conclusion

This article discussed how to implement a CI/CD pipeline to use with WSO2 Enterprise Integrator artifacts using GitLab tools and implementation details of a basic pipeline. This pipeline can be easily customized according to different use cases and requirements.

There are multiple advantages of using GitLab tools for the CI/CD pipeline.

1. Minimal configuration and easy to set up.
2. Everything is integrated with GitLab and everything can be managed via the source code repository.
   e.g. https://gitlab.com/madawa/wso2ei-cicd-demo/pipelines
3. Ability to customize pipelines in each project as the configuration is maintained within the project itself.

References

GitLab CI/CD | GitLab

How to build a CI/CD pipeline for WSO2 ESB (WSO2 EI)

Practical CI/CD for WSO2 Enterprise Integrator with Jenkins