Using Protocol Buffer to Generate SDK at Hootsuite
What is Protocol Buffer?
Protocol buffer (protobuf) is an IDL (Interface Definition Language) by Google that describes the API. IDL sets up the interface for clients and servers so they can communicate using RPC (remote procedure call). RPC allows calling remote services as simple as calling local functions. gRPC is Google’s implementation of RPC that uses protobuf by default, and uses HTTP/2 for transport. The reason why we use protobuf as our serialization framework for SDK generation, instead of another framework like Apache Thrift or Fast Buffer, is because of protobuf’s support of gRPC. gRPC is much faster and safer than REST while requiring less boilerplate code, allowing the code to be less verbose.
At Hootsuite, we use protobuf because of its support of gRPC, along with its efficiency and flexibility. Protobuf, unlike some other data serialization formats (such as XML or JSON), utilizes binary serialization which makes .proto file small and fast to generate and parse. Additionally, protobuf is both language and platform independent, which supports Hootsuite’s diverse technology stack.
Why does Hootsuite use Protobuf to Generate SDK?
Using protobuf to generate SDK helps to create faster to produce, less erroneous, yet more maintainable and flexible code. By generating the SDK instead of hand writing it, we save a lot of time. Theoretically, generated code should have less errors, although that is not always true in reality. Protobuf helps document and describe the API specifications, so additional documentations like swagger would not be necessary. As well, using protobuf to describe the API makes it possible to refactor existing services for new services, as long the it conforms to the proto defined API. Another advantage protobuf has is its ability to generate SDKs in multiple programming languages, Protobuf’s easy to refactor nature and multiple languages support allow Hootsuite to continue to embrace new technologies.
How do we use Protobuf to Generate an SDK?
Now that we understand how useful generating SDK from protobuf can be, the question becomes: How do we use protobuf to generate an SDK? First we need to create the .proto IDL file. Afterwards, we use the IDL to generate the SDK with the help from Hootsuite’s internal IDL-to-SDK repository.
Diagram 1. IDL to RESTful SDK generation
The IDL-to-SDK repository contains a Makefile that compiles the IDL repository using protoc in order to generate swagger. Swagger is a set of rules for documenting API to ensure all API are standardized and nicely visualized for easy understanding. At Hootsuite, we use swagger to describe our RESTful API. Protoc is the Protocol Buffer compiler that generates stub gRPC classes in various languages. These languages include Go, Scala, Php, and more. We need to first use IDL to generate swagger since protobuf does not support REST/HTTP v1.1 SDK generation directly. However, protobuf can generate gRPC SDK directly.
While REST and gRPC both allow clients to communicate with servers, they have distinct differences. REST uses HTTP v1.1 whereas gRPC use HTTP/2. gRPC allows you to define any kind of function calls (ie. synchronous, asynchronous, unidirectional, bidirectional) whereas REST, being a design pattern, is constricted to a limited set of function calls (ie. GET, DELETE). Also, since gRPC calls remote service calls like local calls, it is faster than REST implementation.
After the swagger gets generated, it is passed to swagger-code-gen to generate the REST SDK in a new repository. Swagger-code-gen is a tool used to generate documentation, server stubs and client SDK by parsing swagger. Afterwards, the SDK repository is ready to be used in production.
Despite the advantages gRPC offers, and the additional steps it take to generate REST SDK from protobuf, at Hootsuite we continue to use REST SDK instead of making a complete switch to gRPC SDK. The reason is because a lot of other external and internal services still rely on REST. Thus, in order to maintain flexibility, Hootsuite generates REST SDKs and in the future, will transition to gRPC by generating gRPC SDKs.
Diagram 2. Hootsuite’s current IDL to SDK generation that supports only RESTful SDK compared to Hootsuite’s future IDL to SDK generation will support both RESTful and gRPC SDK
What are the Disadvantages to Using Protobuf to Generate SDK?
Protobuf to SDK generation has its advantages, but also disadvantages. Some downsides to using protobuf to generate SDK involves its generated nature and its lack of existing support and usage within the Hootsuite ecosystem. While generate SDK is time efficient, it could be harder to understand, debug, and test because it is not written by a human. Involving newer technology into older codebase can require extra steps. In this case because there is no direct protobuf to REST generation, we need to generate swagger as an intermediate step. Eventually, Hootsuite’s goal is to transition towards using gRPC. However, until the transition is complete, both gRPC SDK and the REST SDK has to be maintained which will require more resources than using a single SDK.
Overall, protobuf allows for a flexible and fast way to generate SDK with ease. Despite Hootsuite’s lack of support for gRPC services and SDKs currently, this is the direction we are heading towards.
Acknowledgements
Sam Reh, Brandon McRae, Adam Arsenault
About the Author
Joy Zhang is a Software Developer Co-op on the Strategic Integration team at Hootsuite. During her work term, she worked on a major LinkedIn V2 migration and saw it to completion. She studies Computer Science at the University of British Columbia. When she’s not at school or at work you can find her attending hackathons, seeking out delicious food and travelling the world. Connect with her on LinkedIn, Github , Twitter ,Medium and Devpost.
