Building APIs With gRPC and Go

Structure your project to support many versions

API design is hard. Often with new projects, we are limited by the information or knowledge of the problem we are trying to solve. Once consumers come on board, insights which were previously overlooked can become known, resulting in the evolution of our API.

gRPC and Protocol Buffers do a great job in helping us evolve our API in a way which maintains compatibility in both ways:

• Backwards compatibility can be achieved by ensuring all existing message fields, and service definitions remain unchanged.
• Forwards compatibility is achieved by the way protocol buffers are serialised and referenced by the field index on the message type.

However, sometimes backwards compatibility cannot be achieved. gRPC makes it easy for us to expose different versions of a services’ API concurrently on the same server. Doing this allows our consumers to migrate to the new version in their own time, without forcing a breaking change on their end.

The chances are, you might have found yourself in a similar position if you have ever built a service with a public facing API. The question I have for you is this:

Did the evolution of your API version require a complete or partial rewrite of your existing code?

Problem

The go programming language allows us to structure code however we want. This is fantastic, but when done incorrectly your code can quickly turn into spaghetti.

When building APIs using gRPC and go, the implications of a poor project layout can be devastating to your service. This can be amplified even further if you haven’t taken into consideration how you are going to support concurrent versions of the same service, for when this use case arises.

Effective project structures for go / gRPC services is a topic which I have found little information on, and is always a point of conjecture. To avoid the pains of dependency cycles, or having to maintain different database layers for the different versions of your service, I’m going to highlight an approach which I use to solve this problem. Read on to find out more.

Solution

To aid the explanation of the following solution, assume there is a Greeter service which has two API versions exposed on the same server.

V1 and V2 implementations of the Greeter Service registered to grpc server.

For a project like this, I would normally structure the internal or pkg directory such that it contains the following packages.

pkg/
    database/
    serializer/
    service/
        greeter/
            v1/
            v2/

The service package will hold the implementation of each service which is registered against your server. Each service will be contained within its own package and have an import path matching that of the API version it is implementing. Doing this provides a clear distinction between what services exist in the project, and the underlying API version the package is implementing.

The database package will define an interface to your database layer, and will hold implementations of that interface inside it. The database layer should not know anything about the service layer. If it does, it likely means your database is tightly coupled to an individual service. If that is the case, introducing more services, or different versions of a service will inevitably require a rewrite of the database package.

Interface into the Database Layer.

My golden rule: Structs defined inside your service layer, or from your generated API code, should never be seen inside your database layer.

The serializer package will then act as a translation layer between your service and database layers. It will define an interface which mimics that of the database layer, but returns response types that implement their own interface. The methods contained inside this interface allow the response to be transformed into the response object which can then be sent over the wire.

SayHelloSerializer / SayGoodbyeSerializer interfaces transform database responses to API responses to be sent across the wire.

Our services will then hold reference to a serializer rather than a reference to the database itself. The serializer will be held responsible for delegating the call to the database, and returning the response in a wrapper which implements the required serializer interface. It is inside this wrapper where the mapping from database response to service response is defined.

ToV1() and ToV2() handle mapping to proto generated response structs.

Our service then calls what appears to be the same interface as the database layer, but has a response which exposes a set of methods allowing it to map to the response needed to be sent across the wire.

Conclusion

Structuring your project like this will set you up for success when the time arises that you have to support another version of your service, or introduce another service to your gRPC server.

A single database layer can be defined which serves the needs of all the services within your project. Instead of rewriting this layer, simply extend the interfaces inside the serializer package to provide transformations to your new version or service.

You will find yourself spending less time untangling the code as your project evolves in size and complexity, and more time building out your services by extending and implementing the already defined interfaces.

Finally, if you would like to take a deeper look into the code accompanying this post, feel free to clone the repo below. Also feel free to leave any comments for ideas / suggestions you might have, or if you find anything wrong with the example code, thanks!

https://github.com/rokane/grpc-demo