Protobuf definition best practices
Protocol buffers are a mechanism for sending data through the series of tubes known as the Internet. One common use of them is to define gRPC specifications — essentially a form of remote procedure calls. With gRPC service definitions, you create a “service” that has RPC methods. These RPC methods take a request “message” and return a response “message”.
service FooService {
rpc GetFoo (GetFooRequest) returns (GetFooResponse);
}message GetFooRequest {
string id = 1;
}message GetFooResponse {
string fooName = 1;
int64 fooValue = 2;
}
Protobuf and gRPC are powerful tools and can be used in a number of ways. What follows is a set of recommended best practices to use when defining your gRPC services using protocol buffers.
1. Consistent naming
RPC methods should follow a consistent naming convention — this makes it easier to find methods since you can infer the name of what you’re looking for. It also allows for self-documenting code — it’s a lot easier to digest an API if the methods clearly state what they do.
rpc GetFoo (GetFooRequest) returns (GetFooResponse);
rpc GetBar (GetBarRequest) returns (GetBarResponse);
rpc CreateOrUpdateFoo (CreateOrUpdateFooRequest)
returns (CreateOrUpdateFooResponse);
rpc CreateOrUpdateBar (CreateOrUpdateBarRequest)
returns (CreateOrUpdateBarResponse);The specific convention you follow doesn’t actually matter as much — what’s more important is that you follow a convention, so that consumers of your service don’t have too difficult a time finding the right methods to call.
Another thing to watch out for is naming messages by what they contain. That can cause problems down the road, since protobuf messages can change over time, and you can easily end up in a scenario where the name no longer matches up with what the object contains.
message FooIdAndName { // bad
int64 id = 1;
string name = 2;
}
message FooObject { // better - at some point Foo will have more
int64 id = 1;
string name = 2;
}2. Data structure enforcement at the API layer
Protobuf messages define structured objects. As a service owner, you can be sure that data you are receiving comes parsed into a specific structure. As a service consumer (i.e. as a client), you can be sure that the response comes back parsed in a specific structure, with fields you expect.
message GetFooRequest {
int64 fooId = 1;
}message GetFooResponse {
string fooName = 1;
boolean active = 2;
repeated string tags = 3;
In the example above, a service owner always knows that a GetFooRequest will come in with a field called “fooId”, which is a 64-bit integer. A client knows that it will receive a response that contains specific fields with their specific types. This strong specification is of benefit to both parties.
It can be tempting to ignore the structure and stuff a serialized object into a single string or bytes field, but this negates a lot of the benefits of using protocol buffers. For instance, protobuf allows for extensibility by providing for adding and removing fields over time (as long as you don’t re-use the field number of a removed field), but if you implement your own serialization that’s a problem you’ll have to worry about. Utilizing the different field types in protobuf allows us to hand off serialization and parsing, and keeping our messages structured provides clarity around API input and output.
3. Unique messages for RPC requests and responses
RPC definitions in protobuf take in a request and return a response. It can be tempting to re-use the input or output message across multiple RPC methods and it may save you time getting set up, but it’s important to keep in mind that APIs may change over time, and you probably don’t want to couple two separate RPC calls tightly together. If you do end up re-using the same message object across RPC calls, you will likely end up in a state where some fields are set for one call, but ignored for another. This makes the life of the RPC clients more difficult, as they need to think about what fields the message object has versus what they actually need to set on it. The server also has a more difficult time, as it need to know to ignore certain fields if, for example, using the message to update a row in the database.
// don't do this!
message CreateFoo (Foo) returns (Foo);
message SetFooName (Foo) returns (Foo);
message DeactivateFoo (Foo) returns (Foo);message Foo {
int64 id = 1; // set when returned but ignored when creating
boolean fooActive = 2; // ignored in CreateFoo/DeactivateFoo
string name = 3; // ignored in DeactivateFoo
Instead, if you use purpose-built messages for each RPC definition, you can evolve them independently over time.
message CreateFoo (CreateFooRequest) returns (CreateFooResponse);
message SetFooName (SetFooNameRequest) returns (SetFooNameResponse);
message DeactivateFoo (DeactivateFooRequest)
returns (DeactivateFooResponse);message CreateFooRequest {
string name = 1;
}
message CreateFooResponse {
int64 id = 1;
string name = 2;
bool active = 3;
}
message SetFooNameRequest {
int64 id = 1;
string name = 2;
}
message SetFooNameResponse {
}
message DeactivateFooRequest {
int64 id = 1;
}
message DeactivateFooResponse {
}
In the above example, if you wanted to add a user name to the DeactivateFoo call for auditing purposes, you could add it to the DeactivateFooRequest with no issues, since DeactivateFooRequest is specific to that use case.
An important point to note that goes along with this — keep your message structs simple. If your message struct starts evolving by having a few fields that can optionally be set, you should stop and think about whether or not the functionality need to be offered via a separate RPC call, with its own purpose-built message definitions. It can become a little tricky to document a struct that has a lot of optional fields and so a lot of the time you really want to break out the functionality into different RPC methods.
4. Use structs liberally
One of the strengths of protocol buffers is that it supports the addition of fields without breaking existing clients. This allows for the addition of functionality and optional features to existing RPC methods without having to re-implement the method. However, to be able to avail this feature, you have to be careful to use structs in places you may want to add fields.
rpc GetFooHistory (GetFooHistoryRequest)
returns (GetFooHistoryResponse)message GetFooHistoryResponse {
repeated FooHistoryResponseItem items = 1;
string cursor = 2;
}
message FooHistoryResponseItem {
int64 fooId;
}
In the example above, we use a struct for the repeated items instead of a plain old int64 because this allows us to add more fields to the list of response items in the future. For instance, if we wanted to add the foo names to each items, we could simply add it to the FooHistoryResponseItem struct and it would be available alongside the fooId. If we had used a raw repeated int64 instead, this would not be possible.
Another important point here — protobuf allows you to have an empty request or response by using the google.protobuf.Empty message. While this seems convenient at first, it puts you in a corner where it comes to extending that RPC method in the future — you can’t add fields to it, so you are essentially stuck with that being empty for the lifetime of the RPC method. To future-proof your service definition, use a purpose-built message instead, even if it starts off empty. This gives you the flexibility to extend it in the future.
5. Don’t have one field in a struct influence the meaning of another
Protocol buffer messages usually have multiple fields. These fields should always be independent on each other — you shouldn’t have one field influence the semantic meaning of another.
// don't do this!
message Foo {
int64 timestamp = 1;
bool timestampIsCreated; // true means timestamp is created time,
// false means that it is updated timeThis causes confusion — the client now needs to have special logic to know how to interpret one field based on another. Instead, use multiple fields, or use the protobuf “oneof” feature.
// better, but still not ideal because the fields are mutually
// exclusive - only one will be set
message Foo {
int64 createdTimestamp;
int64 updatedTimestamp;
}// this is ideal; one will be set, and that is enforced by protobuf
message Foo {
oneof timestamp {
int64 created;
int64 updated;
}
}
6. Enum definitions should have an “unknown” value at 0
Protobuf doesn’t support the concept of “empty” enums — getting the value of an enum field will always return some value, even if the value was never set by the caller/recipient.
// don't do this
enum ChangeType {
CREATE = 0;
UPDATE = 1;
DELETE = 2;
}message Foo {
ChangeType changeType = 1;
}
If the person who creates an instance of Foo doesn’t set changeType, the client using that Foo instance will see a default value. For an enum, protobuf defaults to the enum constant at ordinal 0 — so in the above example, the client will see a value of CREATE.
For this reason, a common practice is to create an “unknown” enum type at position 0, like so:
enum ChangeType {
UNKNOWN_CHANGE_TYPE = 0;
CREATE = 1;
UPDATE = 2;
DELETE = 3;
}This means that if the value wasn’t explicitly set (to a non-UNKNOWN_CHANGE_TYPE value), the client will see UNKNOWN_CHANGE_TYPE.
On a side note, the scoping rules of protobuf dictate that enum constant values must be unique within a protobuf file, hence calling it UNKNOWN_CHANGE_TYPE and not UNKNOWN.
