Why choose between gRPC and REST?

Options for letting your service expose both REST and gRPC API’s on a single port.

Summary

In this article, I explore the options for making a service, written in Go, expose both a gRPC and REST API on a single port. I explain how one may use the popular grpc-gateway to auto-generate a REST-full JSON API from gRPC definitions, and then show how to make both API’s use the same port using an HTTP multiplexer. I then conclude there are severe limitations to this approach and show an alternative using CMUX, a TCP multiplexer. This results in a cleaner and better architecture.

Introduction

I found myself working on an open-source project called Moulin, a reliable queue for batch processing written in Go and backed by Redis. As there would be different use-cases I want to provide users the choice between a fast gRPC option, and a more traditional REST API. I also want to be able to add a web interface later.

From the gRPC docs page I read that there exists a “popular project” called GRPC Gateway. It allows you to annotate your Protobuf definitions, and it will then auto-generate code for the HTML handlers, take care of conversion from JSON to Protobuf and back, and forward it to an upstream gRPC server.

I decided it would be great if both API’s could share the same port (i.e. 80 or 443). That way in deployment the user may map a single hostname and it would “just work”. The reasoning is that gRPC uses http2, which makes it all HTTP.

Brendon Philips, from CoreOS has published an article titled “Take a REST with HTTP/2, Protobufs, and Swagger” where he explains this approach. I originally decided to write an article on this same topic because I felt that the topic of creating and sharing a TLS certificate with this approach had issues and was not well documented. I later found a better solution (for my needs) altogether.

How it works:

Both gRPC and HTTP clients connect to the same port on the application, served by go’s http request multiplexer.

The multiplexer checks if the http version is http2 and if the content-type is `application/ grpc`.

If it is, we just pass the request onto the gRPC handler. If it is not we pass the HTTP message to the gRPC gateway, which then does its magic to make it into a gRPC message, creates a new connection and then forwards it to the gRPC server. Sure, there is a little overhead, but gRPC is fast, and this saves a lot of code (…).

Here you can find the full source-code:

dhrp/grpc-rest-go-example

Here is the function that does the actual discrimination:

It was not easy to setup, but it works.

It also left me unsatisfied…

Caveats

The thing is that this solution comes with a couple of caveats...

At first I thought I could get away with disabling TLS for the gRPC server, because while gRPC encourages TLS, it does have a `WithInsecure()` function.. The core of the problem is that we’re using a http multiplexer, and so have to use `grpc.ServeHTTP()` instead of `grpc.Serve()`. This uses go’s http2 libraries under the hood, and they do require TLS. So we’ll have to deal with TLS internally. This is frustrating. In the illustration above it is the connection between the grpc-gateway and the grpc handlers. You can read more about this issue here: https://github.com/grpc/grpc-go/issues/555

OK, So.. The (internal) connection between the REST proxy and the gRPC server has to be over TLS, but the gRPC+REST port also needs to be remotely accessible. The real pain is that, therefore, the certificate needs to be issued not for localhost, but some actual hostname or IP. Whenever you deploy, you need to recreate the certificate and you need to configure the proxy internally to use that new address or hostname, otherwise the connection will be refused.

This is a hassle, and depending on your network setup may also cause a network traffic loop back over your load balancer. If you are developing an internal API or hosted (proprietary) solution, and expect to gain a lot of benefit of not having to recreate the REST endpoints, it may still be worth it. But beware that sharing development efforts may be more difficult. It could then actually be better to keep the classic setup where the grpc-gateway (proxy) is served from a different host.

In trying to use the gRPC gateway I have also found other limitations, mostly caused by REST not cleanly mapping onto gRPC, the result is that the REST API is kinda’ ugly. For example, you can’t use query parameters, or form-upload a file.

Alternatives

So, while we have shown (and given code) to make it work, we also recognise it’s not pretty. So what is the alternative?

Don’t use the grpc-gateway

The problem with the internal TLS connection only exists because we are setting up a new gRPC connection after we’ve received a HTTP request.

If we don’t use the gRPC-gateway we may still use the HTTP multiplexer to send traffic to either gRPC handlers or regular HTTP handlers.

Using (only) the HTTP multiplexer

This way, as shown in the illustration, we now still have one open port on which the HTTP request multiplexer listens, but two sets of handlers.

The HTTP multiplexer checks what type of request it is and forwards it to the appropriate handlers.

We can then write simple HTTP handler functions that simply call the same business logic as the gRPC calls do.

A little example for how to do this in code can be found here.

Because of the use of http2, TLS is still required on the external http multiplexer port, but this is not a real problem. But if you want, you can work around that, too.

Use a TCP multiplexer (cmux)

It is also possible to multiplex the connection at the TCP level. There is a great little go library called cmux which allows mixing and filtering of different protocols on a single port, not just filtering on different routes or versions of HTTP.

It works by reading the first couple of bytes from the incoming request to determine which protocol it is. If it is HTTP, it forwards it to the HTTP handlers, if it’s gRPC it forwards it to the gRPC handlers. But it’s not limited to HTTP and gRPC. It could also do SSH, or FTP, or whatever your heart desires. For example both HTTP and HTTPS on the same port. This makes it very flexible.

In a different branch I have ported the example code I showed first to use this approach, and it works like a charm.

dhrp/grpc-rest-go-example_nogateway_branch

Conclusion

After spending time to figure out the details of these different options I believe the GRPC-Gateway is not a great choice for my needs. While it is a cool project and idea, the setup of internal TLS is required and annoying, and in this configuration it may cause unexpected network issues and overhead. The grpc-gateway also has some other limitations that influence the design of your REST API.

Without it, our application will need to implement the event handlers twice. Once for gRPC and once for HTTP calls. Depending on your application structure, and the amount of endpoints that you have the overhead is limited. In my situation the savings are not worth the trouble. Whether you choose for the HTTP or TCP muxer matters less, but the TCP muxing option gives you the most flexibility and is what we’ll go for, for now.

Thanks for reading!

Acknowledgements / links

• GRPC Gateway: . This is really what inspired me to explore this option.
• A tale of two ports; CockroachDB was (among the) first to push for the use-case of mixing gRPC and REST on one port in Golang.
• Take a REST with HTTP/2, Protobufs, and Swagger; Who showed how to make it work on one port by Brandon Philips (CoreOS)
• Golang: Run multiple services on one port; Article by Ashwin Ramesh of Dgraph showing how to use cmux.
• CMUX, TCP multiplexer for Golang.
• AlesSI for pointing to Cmux on the go-grpc issue.
• And thanks to Mark Gituma for giving me feedback on the article!