How Identity Server is Made

Source Code Maintenance

WSO2 Identity Server maintains the source code and all the scripts that are used to build the product in GitHub repositories, under two GitHub organizations: wso2 and wso2-extensions. These two GitHub organizations carry code and scripts related to all WSO2 products, with hundreds of repositories.

The Git repositories used by WSO2 products are of two categories:

• Feature repositories: A feature repository consists of the source code relevant to a particular component. Each WSO2 product is built using combinations of these feature repositories. Therefore, if you want to do modifications to a particular function in a product, you need to clone the Git repositories relevant to that function. Note that each repository will also consist of Tag IDs, which correspond to the tag that was used for a particular product release.

Eg: wso2-extensions / identity-inbound-auth-oauth : Is the feature repository for Identity Server OAuth 2.0 inbound authentication feature

• Product-build repository: A product-build repository consists of the build scripts and profiles that are used for building the product. It also includes Integration test cases. You can simply clone this repository tag to your computer and build it to get a standard product distribution. When you build a product-build repository, all the component repositories that are required for that product release will be automatically fetched from Nexus. See the topic of Using Maven to build Git repositories.

Eg: wso2 / product-is Is the WSO2 Identity Server product repository,

Since there are hundreds of repositories sometimes it’s hard to find, which jar file comes from which repo. Hence we have built a repo explorer (rEx) for Identity Server (IS) to address this concern.

So if you are struggling to find a feature repository please use rEx, it will make your life easy.

Building Identity Server Locally from Source

Prerequisites

• Git
• JDK 8
• Maven

wso2 / product-is Is the WSO2 Identity Server product repository, you can simply clone this repository and build the product. However, if you have modified any of the feature repositories, you need to first build those feature repositories, and update product-is to match their new version, before building the product-is repository. Given below are the steps you need to follow.

1. Build all the feature repositories that you have cloned to your computer and edited.
2. Make sure the feature version you changed is reflected in the pom.xml in the product-is repository you have cloned.
3. Once you have built the feature repositories, build the product-is repository that you have cloned to your computer.
   When you build the product repository, Maven will first check in the local Maven repository on your computer and fetch the artifacts that you built in Step 1. Maven will then fetch the remaining dependencies from Nexus. This process will give you a new product distribution with your changes.
   You can find the new binary distribution (ZIP file) in the <PRODUCT_REPOSITORY_HOME>/modules/distribution/target directory.

Continuous Integration and Delivery Process

Overall Build System

Main Components of the Build System

• Jenkins — CI/CD platform
• Sonatype Nexus — Online maven artifact repository that hosts all the release artifacts
• SonarQube — Code quality verification

Sonar jobs are an exact duplicate of the maven jobs. The main difference is these jobs include post build action to analyse the code using findbugs. Once the code is analyzed the results are pushed to the SonarQube which generates the code quality statistics.

• GitHub — Source code

wso2 — Main organization

wso2-extensions — Products extensions such as IAM connectors

Code Quality Tools

1. FindBugs / FindSecurityBugs — Static code analysis for possible coding bugs
2. Checkstyle — Checks for coding style errors
3. Pull Request review mechanism — Each Pull request is peer-reviewed and is merged by another peer

Continuous Delivery Sequence

This section explains the effort went into automatically doing a release for each Git push by incrementing the micro part of the version number. This is one more step towards CI/CD.

The requirement is to let our CI/CD platform (Jenkins) take care of releasing all our Git repos. Some developers feel guilty to increment version numbers and this leads to a long release train being created when a product has to be released. Features are developed in feature branches. And when that feature is completed, we merge it to the master, and the release will happen automatically. It could be even bug fixes which are pushed to GitHub which will trigger the automatic release.

As part of this, we provided following two capabilities:

1. Automated release build on each Git Push / PR merge by committers
2. Triggering a Release through Jenkins on as-needed basis

Each repo can decide whether they want to go ahead with release per each Git Push, or do a release on an as-needed basis through Jenkins.

With #1, Jenkins automatically takes over the releases of the repo. It registers a Git Hook to listen for each Git push, and do the release. It creates a tag in GitHub, and uploads all the artifacts to the online WSO2 Nexus Maven repository.

With #2, the developers need to come to the Jenkins, and ask it to do a release for them. This is useful for repos that are still in infancy — with frequent commits. Releasing per Git Push does not make sense in this case.

Following is the sequence diagram of the process of this implementation. It shows the interactions between Git users, Git repos, Jenkins and the Sonatype Nexus. This process was implemented as a Jenkins plugin.

Automated Production Quality Check

To make sure the Identity server is production-ready, just running automated tests in a given platform is not enough, it has to be done in a combination of operating systems, databases, JDKs, etc. Furthermore, a combination of scenarios and infrastructure has to be tested for different deployment options as well. And manually doing this will not scale-up. Addressing this concern within WSO2 we have a three-dimensional automated platform which is called TestGrid.

Within the TestGrid, Infrastructure, deployment, and scenario dimensions will be independently automated. For example in the infrastructure dimension, it will set up the infrastructure (VMs, OS, DB, JDK, etc.). Once the infrastructure is done Identity server will be deployed for different deployment patterns. In the end Identity Server use cases will be tested on these platforms and will make sure it's ready to use in production.