Bring gRPC payloads and domain models closer with ScalaPB transformations
At Teads, for more than two years, we have been migrating our business logic to a central component. This component exposes an API that is consumed by tens of services (BFF, cron jobs, AWS lambdas…) and also engineers via a command-line interface.
gRPC and protocol buffers
Our industry is moving quickly and so is our API. Every week, new endpoints are created and the existing ones are updated. Every day, the component serving the API is deployed several times. Not breaking clients is obviously a necessity. We have chosen gRPC as a protocol, for the backward-compatibility features and the strong contractual guarantees it brings.
The expressiveness offered by the protocol buffers (aka protobuf, the Interface Definition Language used by gRPC) is also a key advantage to express clearly our business concepts. Last but not least, thanks to gRPC the schema drives the development, helping us to think API-first.
Scala and ScalaPB
For the implementation of our API, we decided to use Scala for several reasons. First of all, most of our engineers are at least familiar with the language. Secondly, Scala is a very expressive language with a powerful type system, it’s perfect for implementing rich domain models and clear business rules. And finally, thanks to ScalaPB, the support of gRPC is very good.
ScalaPB “generates Scala case classes, parsers and serializers for your protocol buffers”
ScalaPB offers an important number of features: customization of the generated code (using protobuf options), JSON conversion, sbt integration to name a few.
protoc-gen-validate
To get an even richer contract, we take advantage of protoc-gen-validate (PGV), a set of protobuf options allowing to add constraint rules on protobuf fields. Consequently, the protobuf file not only defines the structure of the messages and the types of the fields but also more precise business invariants.
Annotating the API with PGV options makes the API more documented and more explicit since the rules are not hidden in the implementation.
ScalaPB converts the PGV rules to validators. A validator ensures that the message honors the specified rules otherwise it raises an exception mapped to an explicit gRPC status code, making it impossible for the API implementation to send or receive payloads that would violate the contract.
Example of a protobuf message with some protoc-gen-validate rules:
The case class generated by ScalaPB for the above protobuf message:
And the validator generated by ScalaPB, which checks the message content regarding the protoc-gen-validate rules:
ScalaPB transformations
All of this stack works really well but we felt that we can go even further. As you may know, Scala emphasizes static typing and promotes encoding the business logic in types (making illegal states unrepresentable).
The ScalaPB generated validators, while being correct, enforce at runtime the fulfillment of rules on low-level types. There are only String, Int, Seq… in the generated code while our domain model contains Email, URL, NonEmptySet[Role]…
But the latest version of ScalaPB introduces an amazing feature (sponsored by Teads 😎) called Transformations which:
“Allow you to automatically apply ScalaPB field-level options when a given field matches certain conditions”
Thanks to this feature, we are now able to customize the generated code (apply ScalaPB field-level options) when some PGV rules are present (a given field matches certain conditions). In other words, implementation details (field types) are derived from semantic rules (PGV).
How does it look in our example?
Transformations are expressed using a domain-specific language in a simple protobuf file.
Defining transformations of PGV rules into domain types:
And the new generated case class with the custom types:
Note: This use of transformations has a dedicated section in the documentation.
It just gives a taste of what is achievable with ScalaPB transformations. You can match on any information of the field (name, type, options) and set any ScalaPB option. You can even reference rule values when applying the ScalaPB options.
Benefits of the approach
As you can see the transport layer is now richer and close to our domain model. And since the fields have the same types, going from an incoming request to our domain model is very much simplified. On the other hand, producing a response requires the implementation to conform to the business rules at compile time. Consequently, fewer runtime errors can occur.
Our transformations are defined in an auxiliary file that is not included in the API schema exported for clients. The fields of our messages are not annotated with ScalaPB customization options and that’s a good thing. Transformations are an implementation detail.
Another unexpected advantage of using these transversal transformations is a more consistent API. Indeed, when two endpoints refer to the same business concept, if the PGV rules are not the same, the generated code is different and thus the drift between the two endpoints is more visible to our engineers.
For all these reasons, we believe that our engineers will use more PGV rules, resulting in a richer and well-documented API. By bringing gRPC payloads and domain models closer, the implementation becomes more robust and less error-prone.
As mentioned earlier, Teads sponsored the Transformation feature, this has been a great experience to work with Nadav Samet (@thesamet) and find a way to actively contribute to an open source project that is a major component in our API stack.
Thanks again to Nadav for all his work.
