Building OSGi bundles using Maven Bundle Plugin with embedded dependencies to be used in the WSO2 Identity Server.
The WSO2 Identity Server (WSO2 IS) runs in an OSGi environment and any extensions that are developed for it need to be bundled as an OSGi bundle using Maven Bundle Plugin to be integrated with it. For someone like me who didn’t know what any of these terms such as OSGi, bundle, maven bundle plugin meant when I started developing an extension for the WSO2 IS, it can be a very daunting task to get a grasp on these concepts.
This blog is intended to be a zero-to-hero guide for anyone with no pre-existing knowledge of these concepts. At the end of the blog, you will have learnt what the OSGi framework is, what maven-bundle-plugin is, how to embed dependencies in a bundle and finally how to build an extension as an OSGi bundle and integrate it with the WSO2 IS.
Note: If you have a basic understanding of OSGi and are only looking to learn how to integrate a bundle into the WSO2 IS, feel free to skip ahead to the end.
The OSGi Framework
OSGi (Open Service Gateway Initiative) is a Java framework for developing and deploying modular software programs and libraries. The WSO2 IS is built using the OSGi framework with modularity in mind to make it easier for custom components and extensions to be integrated into it. A very large application (such as the WSO IS) is broken down into many loosely coupled components (OSGi bundles) and are integrated using the OSGi framework. OSGi handles classloading to ensure that all the necessary classes required by a bundle from other bundles are loaded properly.
OSGi Bundles
In simple terms, an OSGi bundle is simply just a jar file that contains a MANIFEST.MF with some OSGi specific headers which tell how the jar should run in the OSGi environment. It is the smallest unit of deployable code in OSGi. A bundle has to explicitly declare what packages it needs to have access to through the manifest file and the OSGi platform will load these dependencies from other bundles already installed in the platform. The bundle also needs to explicitly declare what packages it exports, that is, the packages that are available for other bundles through this bundle. Furthermore, it is important to note that OSGi allows packages to have versions, which means that multiple versions of the same package can exist in the environment.
Some important headers in the MANIFEST.MF file are,
Export-Package: Expresses which Java packages contained in a bundle will be made available to the outside environment.
Import-Package: Indicates which Java packages will be required from the outside environment as dependencies.
You do not need to worry too much about the manifest file when building a bundle for the WSO2 IS as the Maven Bundle Plugin automatically creates the manifest file. This is further explained in the next section of this blog.
Note: OSGi classloading is a core concept of OSGi which I will not go into detail about in this blog as this is more focused on building bundles through maven. To get a thorough understanding of OSGi classloading, click this link.
Building bundles using Maven Bundle Plugin
What is Maven?
Maven is a build automation tool that is primarily used to build java projects. The necessary data for building should be included in a pom.xml file and Maven will automatically download the required dependencies along with the transitive dependencies required for the project.
In our use case, maven is the tool that is used to build the OSGi bundle. This is done using a plugin known as the maven bundle plugin.
Using the Maven Bundle Plugin
The following is an example pom.xml file used to build a bundle to be used within the WSO2 IS.
I will be explaining parts of the pom.xml file linked above in this section of the blog.
PS: It’d be easy for you to understand if you open the above pom.xml in a separate tab and refer to it while reading the rest of the blog 😄
The packaging type has to be set to bundle to instruct Maven to build an OSGi bundle.
<packaging>bundle</packaging>The dependencies that are required for this project should be specified under the <Dependencies> tag. Maven will automatically download these dependencies including all transitive dependencies as well.
Note: In the below code snippet, the versions are defined under a properties tag that is not shown.
....
<dependencies>
<dependency>
<groupId>org.wso2.securevault</groupId>
<artifactId>org.wso2.securevault</artifactId>
<version>${org.wso2.securevault.version.range}</version>
</dependency>
<dependency>
<groupId>org.wso2.carbon</groupId>
<artifactId>org.wso2.carbon.utils</artifactId>
<version>${carbon.kernel.version.range}</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>secretsmanager</artifactId>
<version>${software.amazon.awssdk.version}</version>
<exclusions>
<exclusion>
<groupId>software.amazon.awssdk</groupId>
<artifactId>netty-nio-client</artifactId>
</exclusion>
</exclusions>
</dependency>
....
Important: When a project is built using maven, the external dependencies mentioned above are downloaded and stored in a separate folder in the local environment it is built. When we build a bundle using maven to be used within the WSO2 IS, we need to make sure that those dependencies exist in the OSGi environment of the IS. If the dependencies do not exist, they need to be either added separately into the IS as packages/bundles or embedded within the bundle. This is further explained in the next section.
Maven should be instructed to use the maven bundle plugin by adding the plugin under the <plugins> tag under the <build> tag as shown below.
<build>
<plugins>
<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
<version>5.1.4</version>
<extensions>true</extensions>
<configuration>
<instructions>
......The tags added under <instructions> are what is directly used by the plugin to generate the MANIFEST.MF file which determines how the bundle behaves in the OSGi environment. Any instruction that starts with a capital letter will appear in the resulting bundle’s manifest file.
- Bundle-SymbolicName
The symbolic name of the OSGi bundle.
<Bundle-SymbolicName>${project.artifactId}</Bundle-SymbolicName>2. Export-Package
The packages mentioned under this tag is to denote the packages available within this bundle for other bundles to use. It can be specified with particular package names as well as package patterns using the * wildcard. Also, it is possible to exclude packages using negation by starting the package pattern with !.
For example, in the pom.xml in the given link, org.wso2.carbon.securevault.aws is mentioned under the <Export-Package> tag. This means that other bundles in the OSGi environment can access the classes belonging to the org.wso2.carbon.securevault.aws package in this bundle. In other words, this bundle exposes the org.wso2.carbon.securevault.aws package to the environment.
<Export-Package>
org.wso2.carbon.securevault.aws
</Export-Package>3. Import-Package
This specifies a list of packages that are required by the packages contained within our bundle. The packages that are required as dependencies by our bundle which already exists in the WSO2 IS OSGi environment are included here to be imported. However, it is important to note that only packages that are already existing in the OSGi environment can be imported.
The dependencies required by our bundle which are WSO2 packages can be imported as such. Furthermore, the WSO2 IS is shipped with most of the common apache packages such as logging, lang, http, etc and these can be imported using this tag as well.
Important: Extra care should be taken to ensure that the package versions existing in the IS are compatible with the requirements of our bundle.
<Import-Package>
org.apache.commons.lang;version=".......",
org.apache.commons.logging;version="......",
org.apache.http.client.*;version="......",
org.apache.http.*,
org.wso2.carbon.utils;version="......",
org.wso2.securevault.keystore;version="........",
org.wso2.securevault.secret;version="......."
</Import-Package>Those are the main instructions that should be specified in the pom.xml. However, a problem arises when our bundle requires dependencies that are not existing in the IS. The problem further compounds when there are transitive dependencies required by the external dependencies. In the next section, we will take a look at two approaches that can deal with this problem.
Embedding dependencies vs separately dropping jars to the IS
There are two approaches to use dependencies that are required by our bundle that are not available in the IS.
Dropping dependency jars into the IS
One way is to drop the jar files directly into the IS and then import these packages using the <Import-Package> tag similar to the one’s mentioned above.
If the dependency jar file is an OSGi bundle, it has to be dropped into <IS_HOME>/repository/components/dropins/
If it is not an OSGi bundle, the server will automatically convert it into an OSGi bundle. For this, the jar file has to be dropped into <IS_HOME>/repository/components/lib/
This is the preferred way of using dependencies since it doesn’t embed the dependencies within our jar file which allows us to use those dependent jars in other bundles as well and also helps in managing vulnerabilities. However, this approach becomes impractical if our bundle requires a large no. of dependencies and transitive dependencies since it will need a user of our bundle to manually download all the required jars.
Embedding dependencies within the bundle
In order to embed dependencies within the bundle, the dependency names should be included under the <Embed-Dependency> tag. It is important to note that only the artifactId of the dependencies should be included and not the full package name as in the import packages tag.
Keep in mind that what happens under the hood in this scenario is, this <Embed-Dependency> instruction embeds the mentioned dependencies which are also included under the <dependencies> tag in the pom.xml. It is very important to note that even if a dependency is mentioned under the <Embed-Dependency> instruction, if it is not a part of the dependencies of the maven project included under the <dependencies> tag, it will not get embedded.
Furthermore, to embed transitive dependencies, the <Embed-Transitive> tag must be set to true as shown below. It is important to also include the names of all the transitive dependencies under the <Embed-Dependency> instruction as well. Only when both of the above is done, the transitive dependencies of the dependencies of the maven project included under the <dependencies> tag will also get embedded within the bundle.
<Embed-Dependency>
secretsmanager,
aws-json-protocol,
third-party-jackson-core,
json-utils,
protocol-core,
sdk-core,
profiles,
reactive-streams,
auth,
eventstream,
http-client-spi,
regions,
annotations,
utils,
aws-core,
metrics-spi,
apache-client,
.......</Embed-Dependency>
<Embed-Transitive>true</Embed-Transitive>.......<dependencies>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>secretsmanager</artifactId>
<version>2.17.124</version>
<exclusions>
<exclusion>
<groupId>software.amazon.awssdk</groupId>
<artifactId>netty-nio-client</artifactId>
</exclusion>
</exclusions>
</dependency>
......</dependencies>
In the above example, secretsmanager is a dependency and all the other dependencies mentioned under <Embed-Dependency> are transitive dependencies of it. The above code will embed secretmanager along with all the necessary transitive dependencies it requires.
Once all the required configurations are set in the pom.xml, run the mvn clean install command to create the OSGi bundle as a jar file in the target directory within the project.
Integrating the created bundle into the WSO2 IS
Once an OSGi bundle is created it is quite simple to integrate it into the IS. You just simply have to copy the created jar file from the target directory within the project and insert it into the <IS_HOME>/repository/components/dropins/ directory in the WSO2 Identity Server.
However, in order to use the classes of the newly created bundle within the IS, configuration files have to be edited as required by the classes in your bundle.
For example, if the newly created bundle has a Secret Repository Providerclass that needs to be used instead of the default secret repository, the following configurations in the secret-conf.properties file located at <IS_HOME>/repository/conf/security/ must be set to point to the new class in our bundle,
secVault.enabled=true
secretRepositories=vault
secretRepositories.vault.provider=org.wso2.carbon.securevault.<custom>.<CustomSecretRepositoryProvider>This will depend on the bundle that you are creating.
Debugging OSGi within the WSO2 IS
The WSO2 IS allows debugging of the OSGi bundles when using the server. This is a great way to see what is causing your bundle to fail and any issues relating to imports.
To enter the OSGi debug terminal, start the WSO2 Identity Server using the following commands,
Linux: sh wso2server.sh -DosgiConsole
Windows: wso2server.bat -DosgiConsole
After the server starts up, press enter to get the OSGi console.
Useful debug commands
ss : List out all the bundles in the server osgi environment.
ss <bundle_name> : Search through the bundles and list out only the bundles mentioned. From this list you can get the bundle id
b <bundle_Id> : This shows all the details of the specified bundle. It shows the imported bundles and the exported bundles. This is useful to check the bundle names and the versions of the bundles that are imported into our bundle.
diag <bundle_Id> : This shows any unsatisfied constraints of the bundle. If the bundle isn't activating properly, this is useful to check what dependencies are causing the issue.
I hope this blog has provided you with some insight on how to bundle an extension to the WSO2 as an OSGi bundle along with all the required dependencies and integrate it into the IS. 😄
