How to Generate Source Code and Documentation from Protocol Buffers schema using Gradle Build tool
There are multiple ways to serialize structured data such as XML, JSON, ProtoBuf, etc., and they play a very significant role for an application when it comes to exchanging data over the network or in terms of their storage. Each of the structured data serialization formats comes with its own pros and cons. In this blog, I am not going to delve into those, and will cover that in detail in another blog.
Here I will talk about creating the Source Code (Java/Python) and the corresponding documentation (HTML/Markdown/JSON/DocBook) file from .proto file using Protobuf Gradle plugin that leverages “Protoc-gen-grpc-java” and “Proto-gen-doc plugins” using Gradle as the build tool, which would make it easy to integrate as part of the developers’ SDLC process as well as build CI (Continuous Integration) process and tools.
Overview
The Protobuf Gradle plugin compiles Protocol Buffer (aka. Protobuf) definition files ( *.proto) in the project. It’s a two-step process:
- It assembles the Protobuf Compiler (protoc) command line and uses it to generate Java source files out of your proto files.
- It adds the generated Java source files and document file to the input of the corresponding Java compilation unit (sourceSet in a Java project), so that they can be compiled along with your Java sources.
Getting Started
These instructions will help to configure your “gradle.build” file to set up the Protobuf Gradle plugin, and also get you a copy of the reference project up and running on your local machine for development and testing purposes.
Prerequisites
Ensure that the following are already installed on your machine:
Configuring the Gradle Build File
Adding the Plugin to the Project
Customizing the SourceSet
The Protobuf plugin assumes Protobuf files (*.proto) are organized in the same way as Java source files, in sourceSets and the the plugin will look for the proto files by default under the ‘src/main/proto’ folder. The Protobuf files of a sourceSet are compiled in a single protoc run, and the generated files are added to the input of the Java compilation run of that sourceSet ().
But in addition to the default folder, if any custom folder needs to be added, then it can be done as below using the sourceSets:
Adding Compile Dependencies
Ensure that you include the below Protobuf dependency in order to compile the generated Protobuf Java file:
Customizing the Protobuf Compilation
The plugin adds a Protobuf block to the project and it provides all the configuration settings. By default, the plugin will search for the “protoc” executable in the system search path. But here we are leveraging the pre-compiled “protoc” from Maven Central.
Also by default generated Java files are under: $generatedFilesBaseDir/$sourceSet/$builtinPluginName, where $generatedFilesBaseDir is $buildDir/generated/source/proto by default, and is configurable as below:
We would be running the protoc with codegen plugins, and are configured under the “plugin” block within Protobuf, here we have configured the “protoc” and “doc” plugins. Please note, the doc plugin is referring to local instance and hence the usage of “path” instead of “artifact”, as at present the plugin doesn’t exist for this, you can follow the instruction in
https://github.com/pseudomuto/protoc-gen-doc#installation — Connect to preview for installing it on to your local.
Once installed, ensure to copy the “protoc-gen-doc” executable under the root project directory same as “gradle.build” or update the path accordingly to point to it correctly.
Generating the Source file and Documentation from the .proto files
Once the plugins are defined, the tasks for generating the Source file and Documentation associated to plugin are configured under the “generateProtoTasks” block as detailed below:
Both Java and Python builtins are added here, but depending on your project need you can either use one or both. Also from a documentation standpoint, all 4 supported formats HTML, Markdown, JSON and DOCBOOK are supported, and depending on the project need either of them can be used.
Running the Gradle Project
The complete Gradle Build file would be as follows:
From the command line, please execute the following Gradle command:
gradle clean install
This would ensure that it would:
- Pick up the *.proto files from “$projectDir/src/main/protobuf” and “$projectDir/src/main/protobuffers” folder
- Generate the Java and Python source files under “$projectDir/src/main/java” and “$projectDir/src/main/python” folders respectively
- Generate a documentation file called “index.html” under “$projectDir/src/main/doc/index.html”
Deploy & Run — Reference Projects
The sample application can be pulled from the Git repository https://github.com/rajeship/tech-playground/tree/master/protobuf-gradle Ensure all the prerequisites as stated above in this Readme are set up.
Once checked out into your local, open the command line and run the following:
1 ./gradlew clean install
This would ensure that it would,
- Pick up the “addressbook.proto” and “book.proto” files from “/src/main/proto” and “/src/main/protobuf” folders respectively
- Generate the Java and Python source files under “/src/main/java” and “/src/main/python” folders respectively
- Generate a documentation file called “index.html” under “/src/main/doc/index.html” detailing “addressbook.proto” and “book.proto”
