Building a fully automated CI/CD process for API development with WSO2 API Manager

It is the age of automation where everyone tries to minimize the manual steps while building software. API management is no exception. The term “CI/CD” which expands to “Continous Integration/ Continous Deployment” is the more technical term for this automation process. According to this definition, it contains a 2 step process for automating your software development and delivery.

• Continuous Integration — This means that whenever you do a change to your source code, it needs to be well tested before integrating into the existing stable code base (or program)
• Continuous Deployment — This means that once the integration tests are passed, the deployment of the new piece of code to the relevant environments (staging, prod) needs to happen automatically as part of the build process

When it comes to development of enterprise software, enterprise architects design the software in such a manner that entire software system breaks down into several layers (layered architecture). The back end services which implements the business logic is developed with a general purpose programming language with the help of existing frameworks. Implementing CI/CD process on top of this kind of approach is quite straight forward since it is easier to write unit tests using the same programming language and source code can be managed in a github repository. With a tool like TravisCI, you can easily integrate the integration and deployment process together when there is a change in the source code.

WSO2 API Manager is a full lifecycle API management platform which comes with a componentized architecture to make the life easier for API designers, API consumers and API owners. WSO2 API Manager comes with a dedicated web application called API Publisher for designing and developing the APIs. Even though this is a pretty useful tool for manullay building the APIs through a proper governance workflow, applying a CI/CD process for this is not possible. But WSO2 API Manager is designed in such a manner that it exposes a set of REST APIs at the product core runtime level which can be used to build a proper CI/CD pipeline for API development.

This article is hugely inspired from the work done by @Manjula Rathnayake of WSO2 team where he has published a similar concept and source code at following Github location. I will be reusing his code to explain the process in more granular and API Manager focused manner.

https://github.com/manjulaRathnayaka/API-Management-CI-CD-Example

At a higher level, we are going to build a CI/CD pipeline to deploy an API to the WSO2 API Manager which connects to a given back end URL. In this scenario, the source code of the API definition is stored in a GitHub repository and there is a TravisCI build job configured to the master branch of that repository so that when there is a change occur in the GH repo, it will automatically deploy the API to the staging environment after testing on development environment. This process is explained in the above mentioned github repository and I’m using the diagram which was used there.

source: https://github.com/manjulaRathnayaka/API-Management-CI-CD-Example

If you want to see how things working first, you can follow the steps mentioned in below link.

https://github.com/manjulaRathnayaka/API-Management-CI-CD-Example#to-try-out-this-with-your-repository

I’m going to explain what is actually happening under the hood so that anyone can use the scripts which are implemented here and change them according to their requirements. Once you clone the above mentioned gihub repository, you can find there are 6 main directories under the root directory of the repository. The contents of those directories are explained below.

• BackendServiceImpl — This is the source code of the backend service which is going to expose through the API created in WSO2 API Manager. This has been implemented using WSO2 MSF4J framework. It contains a build script to build the flat jar for the microservice.
• DeployBackendService — This directory contains the shell script to deploy the backend service to the relevant environment based on the flag set when running this script. This can be used to deploy the backend service to either Dev environment (tenant) or Staging environment based on the passed in value.
• DeployAPI — This directory contains the important information about the API definition and it’s metadata required for WSO2 API Manager. The API definition is included in the swagger.json file in a standard format. Users can define their API with this file. This information is not enough to create an API within the WSO2 API Manager. It requires a set of metadata about API (e.g. endpoint URL, throttling, security, caching, mediation, etc.) to be passed in when creating an API. These metadata and the requests which needs to be sent to the API Manager runtime is defined within a postman collection file with the name “WSO2_API_PUBLISHER.postman_collection.json”. This is the step where we connect with the API Manager runtime and use the product level APIs to create the API definition within the WSO2 API Manager. It contains the following requests in the given order.
• Dynamic Client Registration request — First we need to register a dynamic client to create the API in the runtime. This is sent to the following URL

{{gatewayURL}}/client-registration/register

• Get the access token for API creation — Using the clientID which was generated in the above step, we need to get an access token with “api_create” scope to create an API. This step is for getting the access token by contacting the token endpoint

{{gatewayURL}}/token

• Get the access token for API view — Since we are verifying the API creation through test script, we need to have an access token with “api_view” scope. This step is for getting the access token by contacting the token endpoint

{{gatewayURL}}/token

• Check whether API is already created — Using the token received at previous step, check for the existence of the API which is going to be created. This allows us to decide whether to create the API or update the swagger definition of the existing API. Within the postman test step, it checks for the return value and if there is an API already created, it will skip to the swagger definition update step. The URL used in this step is

{{gatewayURL}}/api/am/publisher/apis?query=name:{{apiName}}

• Create an API — If the API is not created already, create an API using the token received at the second step above. In this step, we are passing all the metadata about the API within the “rawModeData” parameter in JSON format. The URL used in this step is

{{gatewayURL}}/api/am/publisher/apis

• Update the swagger definition — If the API is already created or newly created, we are updating the swagger definition with the content of the “swagger.json” file located in the same directory. The URL used in this step is

{{gatewayURL}}/api/am/publisher/apis/{{apiId}}/swagger

• Get the access token for API publishing — Once the API is created with the above step, it will be in the created state. We need to change the state of the API into “Published” state so that users can consume that API. In this case, we need that to be published for testing purposes before deploying into other environments. In this step, we are getting an access token with “api_publish” scope. The URL used in this step is

{{gatewayURL}}/token

• Publish the API — With the token received in the previous step, we call the publisher API to change the lifecycle state of the API to “published”. The URL used in this step is

{{gatewayURL}}/api/am/publisher/apis/change-lifecycle?action=Publish&apiId={{apiId}}

The above steps are sequentially configured in the postman collection which will be executed by “newman” tool. Once this script is executed, the API will be created in the relevant tenant and will be in the “published” state.

• TestAPI — This directory includes a postman collection which is used to test the deployed API. Before publishing into the upper environments, we need to make sure that tests are properly passing in the lower environments. We are executing the following steps within this postman collection.
• Dynamic Client Registration request — First we need to register a dynamic client to subscribe to the API in the runtime. This is sent to the following URL

{{gatewayURL}}/client-registration/register

• Get the access token for API subscription — Using the clientID which was generated in the above step, we need to get an access token with “api_subscription” scope to subscribe to an API. This step is for getting the access token by contacting the token endpoint

{{gatewayURL}}/token

• Check for application — We need to have an application to subscribe to an API. In this step we are checking whether there is an already created application with the name “testingApp”. If the application is there, jump to the next step where we get the API details. The URL used for this step is

{{gatewayURL}}/api/am/store/applications?query=testingApp

• Create application — If the application does not exist with the name “testingApp”, we are creating an application in this step. The application details are passed in “rawModeData” parameter within the postman request. The URL used for this step is

{{gatewayURL}}/api/am/store/applications

• Get the API details — Next step is to check the existence of the API which was deployed in the previous step and get the details about the API for subscription. With this step, we are doing both. The URL used for this step is

{{gatewayURL}}/api/am/store/apis?query=name:{{apiName}}

• Subscribe to the API — In this step, we are subscribing to the API with the token received at the second step. Subscription details are passed in through “rawModeData” parameter. The URL used for this step is

{{gatewayURL}}/api/am/store/subscriptions

• Get Application Details — Before consuming the API, we need to generate the access tokens for the application. In this step, we are retrieving the application details to check whether those tokens are already generated or not. If those are not generated already, we execute the subsequent steps to generate the application keys and access tokens for the subscription. The URL used for this step is

{{gatewayURL}}/api/am/store/applications/{{applicationId}}

• Generate application keys — If the application tokens are not generated based on the results of the previous step, we are generating the application keys in this step. Here we are specifying the token type and the timeout as input data. The URL used for this step is

{{gatewayURL}}/api/am/store/applications/generate-keys?applicationId={{applicationId}}

• Generate access tokens — If the access tokens are not generated when checking the application details, we are generating the access tokens within this step. Here we are using the “client_credentials” grant type and using the keys generated in the previous step to generate the access tokens. The URL used in this step is

{{gatewayURL}}/token

• Execute the protected API with access token — In this step, we are calling the API with the access token and verify the response which is coming from the backend. If the response is matched, the test is passed. The URL used in this step is the actual API URL in the relevant API manager environment

{{gatewayURL}}/t/{{tenantDomain}}/hello/1.0.0/

Once the above steps are executed with the postman collection, we move into the next step of the execution flow.

• DevEnvironment — This directory contains the environment variables related to the development environment. The file “Development.postman_environment.json” is passed in as an input parameter to newman when executing the postman collection stored in the “DeployAPI” section. The relevant URLs and tenant domain information are extracted from this file during the deployment. The contents within the “backendService.properites” are used when deploying the backend service within the bash script of “DeployBackendService” directory.
• StagingEnvironment — This directory contains the same content as the previous “DevEnvironment” directory which are related to staging environment.

Last but not least, we have the travis yaml descriptor file which defines the execution flow of the build and deploy pipeline. The sequence is mentioned below.

1. Build the backend service implementation
2. Deploy the backend service to integration cloud “Development” tenant (environment)
3. Deploy the API to the WSO2 API cloud “Development” tenant
4. Test the API deployed at “Development” tenant
5. Deploy the backend service to integration cloud “Staging” tenant
6. Deploy the API to the WSO2 API cloud “Staging” tenant
7. Test the API deployed at “Staging” tenant

With TravisCI, you can configure the above script to be executed when there is a change in the github repository attached to this script. You can follow the below steps to see this in action.

https://github.com/manjulaRathnayaka/API-Management-CI-CD-Example#to-try-out-this-with-your-repository

In this post, I have explained the steps to fully automate the deployment of APIs with WSO2 API Manager. You can tweak the steps mentioned in this post and define your own CI/CD pipeline. On top of this automation, if you have more than one node, you can have proper file sharing and database sharing mechanisms to deploy the APIs across all the nodes in the cluster.