Generating Clients/Models for RESTful services from OpenAPI Specification

Generating ready to use clients and models in seconds

Netflix’s service graph — notice how many connections there are! (src)

Communication Is Integral!

As microservices grow in popularity, interservice communication becomes an essential part of an organization’s architecture. No longer does an enterprise’s technology stack consist of three monolothic apps that rarely communicate, but instead, there are hundreds of tiny components that interact on a frequent basis. The most common way for these services to communicate is with REST.

Before we go any farther, I have to point out that I try to avoid making synchronous calls wherever I can (and REST is a form of synchronous calls), but if you have to make them, then you should follow these tips to make your service more resilient

Current (flawed) processes

Despite its importance, I have seen many organizations make interprocess calls in a way that leaves much to be desired. It often goes something like this:

• server side team types models by hand (let’s just say they use java)
• server side team tries to write the open api spec in a way that reflects the hand-written models created in the above step
• server side team creates their own client for testing
• consuming team(s) types models by hand
• consuming teams(s) create their own client(s) for consuming the service

Areas for improvement

This process is flawed in many ways; some of the major ones are noted below

• error-prone — there is no guarantee that the contract is correct (consider that the models have transactionId and the contract has transactionID)
• inefficient— server-side team already define the contract, why should they need to write the models as well? It’s a waste of developer time
• inefficient—server-side team writes a client for tests, then consumers each write their own client; again, another waste of developer time
• inefficient — consuming teams must spend time writing unit tests for the clients they write
• additional overhead —since each consuming team writes their own client,
• lack of standardization — there is no guarantee of best practices being followed across teams
• lack of extensibility— adding new features requires modifying tons of code maintained by different teams

How to improve: code generation

There is a way to improve upon many of the shortcomings listed above: by generating code (both client and server) from a specification. Since this article is about REST, we will use the open api specification, but you can do similar with other formats (such as GRPC with protobuffers, or SOAP with WSDL files). Our process is outline below:

• write the contract for our service
• generate the client in its own sdk/jar (in this case, we will use a gradle plugin), so we can use it in our tests and consumers can include it in their project
• generate the server side models and interface (again, with same gradle plugin, just different configuration) — can be done in parallel with the above step
• Write cucumber tests for our functionality (using our generated client)

Notice how this improves on the shortcomings of the above solution:

• Guarentee of correctness — since the models are generated from the open api specification; they will be 100% correct
• More efficient — the server-side team no longer needs to write models; which saves hours of time
• More efficient — consumer(s) use same client as server-side team uses in functional tests (rather than writing their own), which saves hours of time per consumer
• More efficient— consumers no longer have to write unit tests for clients, saving hours of time per consumer
• Standardization — each client will follow a similar structure, which makes debugging and usage much easier, saving hours of time
• More easily extensible — teams can pass interceptors to modify the request and response; or they can customize the mustache templates to modify the generated code, which benefits all teams

NOTE: this process is similar to what is embraced by google for years with grpc, it existed (at least code generation) for SOAP webservices (with WSDL), as well as if you look at aws’ client sdks, they are all generated. Finally, numerous other successful technology companies, such as salesforce and paypal have all been using a client generation process for a long-time. Moral of the story; it’s tried and true and your company will be left behind if you don’t use it.

End product

At the end of this article, you will have set up a process to automatically generate java model objects representing any models declared in the open api specification (for request/response bodies) as well as fully functional java clients with a method for each operation declared in the specification. It involves 10 lines of code in your build.gradle (you don’t have to write a single line of java code — it is all generated) and you can use it to make REST calls to your service.

• Any model specified in the open api spec has a Java POJO created for it; constructed with same names and nested as appropriate

The specification for an assessment object in the open api document (visualized via swagger editor)

The generated java class representing an assessment

• Any operation (combination of url and method) has a client with a method generated. This method is a fully functional way to call the api, using generated models for request and response. If you wish, you can use tags to generate different APIs; and if you tag the same operation with different tags, you can have the same functionality exposed across different classes.

These are the operations specified in the open api specification (visualized via swagger editor)

You also get a generated java client for with a method for each operation declared in the specification

Implementation details

We will now see how to implement the above process in Java for a client library (note that a very similar process can be followed for other languages). The process is similar for server side stubs, except they give you an interface to implement to use your business logic instead of a fully functional client:

• apply the open api generator plugin

plugins {
    id "java-library"
    id "org.openapi.generator" version "4.3.1" 
}

• add configuration block — the only required important fields are the inputSpec, generatorName, and library

openApiGenerate {
    generatorName = "java"
    inputSpec = "$rootDir/infrastructure/openapi-spec.yml"
    outputDir = "$projectDir"
    apiPackage = "com.connor.reading.client"
    modelPackage = "com.connor.reading.client.dto"
    configOptions = [
            dateLibrary: "java8",
            generateBuilders: "true",
            library: "native",
            useRuntimeException: "true",
            generatePom: "false",
            sourceFolder: "build/generated/sources/" 
    ]
}

• add the dependencies used in the generated *code

dependencies {
    implementation "com.fasterxml.jackson.core:jackson-core:$jacksonVersion"
    implementation "com.fasterxml.jackson.core:jackson-annotations:$jacksonVersion"
    implementation "com.fasterxml.jackson.core:jackson-databind:$jacksonVersion"
    implementation "com.fasterxml.jackson.datatype:jackson-datatype-jsr310:$jacksonVersion"
    implementation "org.openapitools:jackson-databind-nullable:$jacksonDatabindVersion"
    implementation "com.google.code.findbugs:jsr305:$findBugsVersion"
    implementation "io.swagger:swagger-annotations:$swaggerAnnotationsVersion"
}

• add the generated code to the classpath — checks for compilation and allows it to be used

sourceSets.main.java.srcDirs += "$outputDir"

• make the generation run before code compilation, so if there are any errors in the generated code, the project will not compile

tasks.compileJava.dependsOn tasks.openApiGenerate

• (Optional) Update the .openapi-generator-ignore to prevent generating extra files

End Product

The complete build.gradle is shown below (find it on gitlab here) at this point, you have done all the work needed to setup code generation (the next section will show how to run it)

plugins {
    id "java-library"
    id "org.openapi.generator" version "4.3.1"
}

ext{
    findBugsVersion = "3.0.2"
    jacksonDatabindVersion = "0.2.1"
    jacksonVersion = "2.9.9"
    openApiGeneratorVersion = "4.3.1"
    swaggerAnnotationsVersion = "1.5.22"

    outputDir = "build/generated/sources/"
}

sourceCompatibility = 11
targetCompatibility = 11

sourceSets.main.java.srcDirs += "$outputDir"

repositories {
    mavenCentral()
}

dependencies {
    implementation "com.fasterxml.jackson.core:jackson-core:$jacksonVersion"
    implementation "com.fasterxml.jackson.core:jackson-annotations:$jacksonVersion"
    implementation "com.fasterxml.jackson.core:jackson-databind:$jacksonVersion"
    implementation "com.fasterxml.jackson.datatype:jackson-datatype-jsr310:$jacksonVersion"
    implementation "org.openapitools:jackson-databind-nullable:$jacksonDatabindVersion"
    implementation "com.google.code.findbugs:jsr305:$findBugsVersion"
    implementation "io.swagger:swagger-annotations:$swaggerAnnotationsVersion"
}

task deleteExtraGeneratedDirectories(type: Delete) {
    delete ".openapi-generator"
    delete "api"
    delete "src"
}

tasks.deleteExtraGeneratedDirectories.dependsOn tasks.openApiGenerate
compileJava.dependsOn tasks.deleteExtraGeneratedDirectories

openApiGenerate {
    generatorName = "java"
    inputSpec = "$rootDir/infrastructure/openapi-spec.yml"
    outputDir = "$projectDir"
    apiPackage = "com.connor.reading.client"
    modelPackage = "com.connor.reading.client.dto"
    configOptions = [
            dateLibrary: "java8",
            generateBuilders: "true",
            library: "native",
            useRuntimeException: "true",
            generatePom: "false",
            sourceFolder: "build/generated/sources/" 
    ]
}

Additional points of emphasis

There are a few things I’d suggest when generating the code, which I’ve listed below:

• generate client as it’s own jar — this allows consumers to easily include the client in their project
• generate to the build directory, which is excluded from git via the .gitignore— this will avoid it being checked in to git (and from showing up in pull requests), and it can be excluded from static code analysis and unit test coverage
• Use the generated client in your functional tests — this ensures the sdk you give to your clients is functional (and you have to test your deployed service anyways)
• Publish the jar — then have your clients use it (just like aws, salesforce, okta, and other industry leaders do)
• Modifying the generated (java) code does nothing — it will be rewritten by the build process. If you want to modify this, modify the mustache templates (see the next article in the series)

Let’s see it in action!

When you run a gradle build, the java files for the models and clients for the endpoints described in the open api specification will be generated (via the openApiGenerate task in the plugin). This is shown in the below video. Notice that the build directory does not exist, and after I run the build, the directory exists, and it contains models (for both success and error responses) as well as clients we can use for calling the endpoints.

This video shows the code generation process (first file is a generated success response, second is generated error response and third is a generated client)

Usage

Now that we have the generated clients and models, the only thing left to do is use it:

• get an instance of the generated client by specifying the host (and other optional fields)

String host = getHost(); //get the host from service discovery
ApiClient apiClient = new ApiClient()
        .setScheme(SCHEME) //http or https
        .setHost(host) //i.e. dev.mycompany.com
        .setPort(PORT); //http or https port (8080 or 443) 
return new AssessmentsApi(apiClient);

• Call the appropriate method on the client and use the response (contains response body, response headers, etc)

responseBody = assessmentsApi.getAssessmentByIsbn(isbnUsedInRequest);

• Catch ApiException and use it to examine the status code, response body, headers on failures

try{
    responseBody = assessmentsApi.getAssessmentByIsbn(isbnUsedInRequest);
    scenario.log("Successfully made request (and got response with 2xx status code)");
}catch (ApiException e){
    exceptionThrownOnNonSuccess = e;
    scenario.log("Exception thrown when making request (NOTE: doesn't mean bad, if running negative tests this is actually expected behavior).  The response body was " + e.getResponseBody());
}

you can get the http status code, response body, and response headers from the thrown api exception

Conclusion

We have recognized processes that do not generate code from a specification are inefficient, inaccurate, and hard to extend. We have improved on these drawbacks to make our service 100% acccurate to our contract, saving hours of time, and allowing us to easily extend existing functionality and share best practices across teams by generating fully functional clients (and models) for RESTful services defined via an OpenAPI specification. This requires only10–20 lines of code in our build.gradle, and it takes seconds to generate the models and clients.

We then use this client library in our functional tests, and then distribute it for usage by any number of consumers. This reduces stress and allows engineers to focus on writing business logic.

Stay tuned for the next part of the series where I show you how to modify the code generation templates (stored in mustache files) to customize the generated code to automatically retry failed calls (except for 4xx responses).

Related Links

https://medium.com/@connorbutch/enhancing-openapi-code-generation-with-custom-features-58d38e38222

Files · master · Connor Butch / reading-comprehension

OpenAPI Generator · Generate clients, servers, and documentation from OpenAPI 2.0/3.x documents

Successful Microservice Architecture

Stop making synchronous calls!

*if you ever want to see what versions to include, temporarily remove the pom.xml from the open api generator ignore file, run a build, copy the dependencies from there, and then delete the pom file