API Catalog CLI

How to discover and catalog your REST services on real Java projects

Published in

Another Integration Blog

9 min readDec 5, 2022

Nowadays, we usually develop REST services following an API-First approach; however, that hasn’t always been the case and most companies have tens or hundreds of APIs designed directly in the code. This means that there is no declarative file with the API specification, also known as the contract.

Not having a specific file describing each of the company’s APIs, using a standard specification such as Swagger 2.0, OpenAPI 3.0, or RAML, is an important limitation when implementing a universal API management strategy.

There are products such as MuleSoft’s Anypoint Platform, that support this strategy by providing different components such as Exchange, API Governance, Flex Gateway, and API Catalog CLI to cover the lifecycle of any type of REST API regardless of the technology with which it has been implemented. This makes it possible to publish and categorize all the APIs of a company in a single repository, offering a complete vision of the available services to encourage reusability, as well as to apply the same governance model.

Therefore, we are going to analyze different use cases in real Java projects to see how we can take advantage of the capabilities that MuleSoft offers. This same approach can be extrapolated to other technologies where you may have APIs implemented, like Node.js, .NET, or Python.

Use Cases

API Catalog CLI allows us to discover and catalog REST APIs that have been defined in a specification file using OpenAPI or RAML, and YAML and JSON formats.

What can we do if our project does not have the API specification file? Let’s take a look at some possible scenarios when dealing with real Java projects, especially when those projects were developed some time ago:

REST API defined in a specification file: this is the ideal scenario, when the project is developed following an API-first approach and we can rely on the available API specification file. We might expect every new Java project to follow this approach.
JAX-RS to implement a Java RESTful API: in the past, most of the Java projects were built using Jersey or RESTEasy as the main implementations for JAX-RS interfaces to develop RESTful web services.
JAX-RS + Swagger annotations: as RESTful web services became more popular, Swagger emerged as an initiative to standardize the way we document our APIs, so we can also find services implemented with JAX-RS that use Swagger annotations to describe the API.
RESTful API developed with Spring Framework: when we talk about Java applications, most of the time we have Spring Framework as the de facto standard for implementing those services. Therefore, we have to provide support to RESTful APIs developed using Spring MVC controllers and SpringFox + Swagger annotations to deliver a web-based UI to document the API.

How to Generate the API Specification File

As we mentioned before, the main problem is that a project does not have an API specification file.

However, thanks to the swagger-maven-plugin, we can generate Swagger 2.0 specifications from the source code, supporting both JAX-RS and Spring MVC implementations.

First, add the swagger-core dependency in the pom.xml file:

<dependency>
    <groupId>io.swagger</groupId>
    <artifactId>swagger-core</artifactId>
    <scope>provided</scope>
    <version>1.5.3</version>
    <exclusions>
        <exclusion>
            <groupId>javax.ws.rs</groupId>
            <artifactId>jsr311-api</artifactId>
        </exclusion>
    </exclusions>
</dependency>

We only need this dependency to generate the API specification file, hence we configure it with a provided value for the scope field.

To automate the generation of the specification file as part of a CI/CD pipeline, we will also include the swagger-maven-plugin:

<plugin>
    <groupId>com.github.kongchen</groupId>
    <artifactId>swagger-maven-plugin</artifactId>
    <version>3.1.8</version>
    <configuration>
        <apiSources>
          ...
        </apiSources>
    </configuration>
    <executions>
        <execution>
            <phase>compile</phase>
            <goals>
                <goal>generate</goal>
            </goals>
        </execution>
    </executions>
</plugin>

This is the common configuration for the different scenarios that we want to cover, but depending on the technology used to implement the RESTful API and the documentation provided through annotations we will have to adjust the configuration of each ApiSource element:

JAX-RS Configuration

<apiSource>
    <springmvc>false</springmvc>
    <locations>com.jogomezr.examples.restapi</locations>
    <schemes>http,https</schemes>
    <host>mycompany.com</host>
    <basePath>/api</basePath>
    <info>
        <title>JAX-RS REST API Example App</title>
        <version>v1</version>
        <description>This is a sample for OpenAPI spec generation using swagger-maven-plugin</description>
        <contact>
            <email>jogomezro@mycompany.com</email>
            <name>Jose Alfredo</name>
            <url>http://mycompany.com</url>
        </contact>
        <license>
            <url>http://www.apache.org/licenses/LICENSE-2.0.html</url>
            <name>Apache 2.0</name>
        </license>
    </info>
    <outputFormats>yaml</outputFormats>
    <swaggerDirectory>generated/swagger-api-spec</swaggerDirectory>                           
    <!-- JAX-RS annotations -->
    <swaggerApiReader>com.github.kongchen.swagger.docgen.reader.JaxrsReader</swaggerApiReader>
</apiSource>

In this case, we set the field springmvc to false, as this is a JAX-RS API and we use the class com.github.kongchen.swagger.docgen.reader.JaxrsReader as the implementation to parse the source code.

You can download this sample project from jaxrs-api-example.

Swagger Configuration

<apiSource>
    <springmvc>false</springmvc>
    <locations>com.jogomezr.examples.restapi</locations>
    <schemes>http,https</schemes>
    <host>mycompany.com</host>
    <basePath>/api</basePath>
    <info>
        <title>Swagger REST API Example App</title>
        <version>v1</version>
        <description>This is a sample for OpenAPI spec generation using swagger-maven-plugin</description>
        <contact>
            <email>jogomezro@mycompany.com</email>
            <name>Jose Alfredo</name>
            <url>http://mycompany.com</url>
        </contact>
        <license>
            <url>http://www.apache.org/licenses/LICENSE-2.0.html</url>
            <name>Apache 2.0</name>
        </license>
    </info>
    <outputFormats>yaml</outputFormats>
    <swaggerDirectory>generated/swagger-api-spec</swaggerDirectory>                           
<!-- Swagger annotations -->
    <swaggerApiReader>com.github.kongchen.swagger.docgen.reader.SwaggerReader</swaggerApiReader>
</apiSource>

We set the field springmvc to false as well, as this is a JAX-RS API and we use the class com.github.kongchen.swagger.docgen.reader.SwaggerReader as the implementation to parse the source code.

You can download this sample project from swagger-api-example.

Spring Configuration

Finally, we have the specific configuration for a project based on the Spring Framework. More details of the API are provided through annotations and it is therefore not necessary to indicate them in the ApiSource configuration:

<apiSource>
    <springmvc>true</springmvc>
    <locations>com.jogomezr.examples.restapi</locations>
    <schemes>https</schemes>
    <outputFormats>yaml</outputFormats>
    <swaggerDirectory>generated/swagger-api-spec</swaggerDirectory>
    <!-- Spring MVC REST Controllers -->
    <swaggerApiReader>com.github.kongchen.swagger.docgen.reader.SpringMvcApiReader</swaggerApiReader>
    <securityDefinitions>
        <securityDefinition>
            <json>/securityDefinitions.json</json>
        </securityDefinition>
    </securityDefinitions>
</apiSource>

You can download this sample project from spring-boot-api-example.

Generation of API Specification

Once we have defined the configuration needed for our project we just need to execute the following command to generate the API specification:

mvn clean compile

As a result, a folder called generated/swagger-api-spec is created and that folder contains a file swagger.yaml with the API specification.

Enriching API Descriptor Information Automatically

To publish an API on Exchange, we need the API descriptor. This is a file that can be generated by executing the following command:

api-catalog create-descriptor

This command generates a catalog.yaml file, the default file name unless we use the -d or --file flags to provide a custom value. However, the content that is generated by default covers only basic aspects of the API, as we can see in the following screenshot:

The API descriptor supports multiple configurations, such as the definition of the asset versioning strategy, the inclusion of API documentation pages, categories, tags, and more. At this time, it is not possible to provide these values through the commands supported by API Catalog CLI.

Logically, these values could be included manually; however, in our case, we want to automate the entire process so that APIs can be discovered, cataloged, and published to Exchange as part of a CI/CD pipeline.

To do this, we will rely on a premise of our universal API management strategy. Each CI/CD pipeline is associated with a certain set of Java projects on which we can apply the same configuration to enrich the API descriptor. This allows us to define templates to merge them with the descriptor file generated by API Catalog CLI.

#%Catalog Descriptor 1.0
projects:
  - main: generated/swagger-api-spec/swagger.yaml
    documentation:
      home: generated/documentation.md
    versionStrategy: minorIncrease
    versionStrategyConditions:
      patchIncrease:
        branches:
          - 'develop'
    categories:
      API Type:
        - System API
    tags:
      - Java
      - Spring Boot

This template (system-api-template.yaml, in this example) is merged with the API descriptor by running the following command:

yp '.projects[0] *= load("system-api-template.yaml") .projects[0]' catalog.yaml > merged-catalog.yaml

The yp command is a YAML processor written in go that works with YAML and JSON files and simplifies this type of file transformation.

The result is an API descriptor that contains all the configuration values we want to apply to publish the API to Exchange:

#%Catalog Descriptor 1.0
projects:
  - main: generated/swagger-api-spec/swagger.yaml
    assetId: the-students-rest-api-example
    version: 2.0.0
    apiVersion: v2
    contact:
      name: Sponge-Bob
      email: sponge-bob@swagger.io
    documentation:
      home: generated/documentation.md
    versionStrategy: minorIncrease
    versionStrategyConditions:
      patchIncrease:
        branches:
          - 'develop'
    categories:
      API Type:
        - System API
    tags:
      - Java
      - Spring Boot

Associate Documentation Pages

Perhaps you have noticed that in the API descriptor we have added these configuration fields:

documentation:
  home: generated/documentation.md

This is the mechanism we can use to include one or several documentation pages that will be displayed in Exchange as part of the API specification. Each one of them is a markdown document.

Since we intend to automate this process as much as possible, such documentation will be automatically generated as part of the execution of the swagger-maven-plugin if we add the following fields:

templatePath: the path of a template file to generate documentation
outputPath: the path of the generated static document, including the file name

Therefore, the ApiSource configuration looks like this for the sample application developed with Spring Boot:

<apiSource>
    <springmvc>true</springmvc>
    <locations>com.jogomezr.examples.restapi</locations>
    <schemes>https</schemes>
    <templatePath>${basedir}/templates/markdown.hbs</templatePath>
    <outputPath>${basedir}/generated/documentation.md</outputPath>
    <outputFormats>yaml</outputFormats>
    <swaggerDirectory>generated/swagger-api-spec</swaggerDirectory>
    <!-- Spring MVC REST Controllers -->
    <swaggerApiReader>com.github.kongchen.swagger.docgen.reader.SpringMvcApiReader</swaggerApiReader>
    <securityDefinitions>
        <securityDefinition>
            <json>/securityDefinitions.json</json>
        </securityDefinition>
    </securityDefinitions>
</apiSource>

Adding Metadata to Exchange

Another aspect that you may have seen in the descriptor file is the use of metadata to categorize the API.

Specifically, we are adding a category and two tags to indicate that it is a System API implemented in Java with Spring Boot.

The tags are automatically reflected in Exchange, but categories must be define prior to API publication so that an error does not occur.

Publishing and Versioning the API on Exchange

After we have generated the API specification (in case we didn’t already have it), and once the descriptor file has been defined with all the necessary configuration and metadata, we are ready to publish the API on Exchange.

API Catalog CLI provides the following command to do this, taking into account that our descriptor file is merged-catalog.yaml:

api-catalog publish-asset -d merged-catalog.yaml

If everything is correct, we should see a message similar to the following confirming that the API has been published:

It is important to mention how API versioning works based on the provided configuration:

API version: this is based on the value provided as part of the API specification. If we generate that specification from the source code, the team in charge of that application must apply the needed updates in the code to reflect the new version value (v1, v2, v3, etc.) when there are breaking changes in the contract.
Asset version: we use semantic versioning (major.minor.patch pattern) to establish the version value for a given asset. Based on the provided configuration, this value will be calculated taking into account the branch where the changes were made. In our example, we will increase the patch version if we are in the develop branch and we will increase the minor version if we are in any other branch. The major version will be increased automatically as soon as a new API version is provided.

After several iterations introducing changes to the API specification, we might end up with a scenario like this example:

Summary

It is now possible to automate the discovery, categorization, and publication of APIs in a centralized repository for both projects that follow an API-first approach and have a specification file for each API, and for projects developed with a code-first approach that directly implement RESTful APIs without previously defining the contract.

MuleSoft supports this approach through various components, which greatly simplifies the design and implementation of a universal API management strategy.