gRPC like Interceptor for Apache Thrift
gRPC Interceptors act as middleware to address cross-cutting concerns on Server or Client. A cross-cutting concern is a piece of code that applies in many different areas and could lead to code duplication, system dependencies etc. For example: additional validation checks, exception handling, logging, rate limiters around each endpoint on a Service are all cross-cutting concerns.
Apache Thrift doesn’t have a middleware mechanism like gRPC out of the box. With middleware, we could intercept each Thrift request and add more behavior, for example: concurrency limiting, dropping requests from disconnected clients. Thrift doesn’t let us detect client disconnections, so we could process a request sitting on the Server’s queue long after client has abandoned the request.
To add middleware mechanism to Thrift, I started out by looking at different ways to intercept Thrift requests. All Thrift Server (TServer) implementations handle client requests in a more or less similar way. The Server socket accepts a connection from a client, creates a new socket to communicate with the client and moves on to listening for more client requests. If the server is single threaded, the server socket would process the request on the same thread as the one used to listen for connections. If the server has a multi-threaded worker pool, the new socket created for the request would be handed off to that worker pool through a queue. Non-blocking server types are also available that allows fairness to all connected clients.
Since we cannot address all the client requests simultaneously on a Server, we have to queue them. But before queuing a request, based on the current Server status, we need to decide whether the request can be processed or not. This allows clients along the request chain to fail fast and propagate the error to the end user, thus allowing graceful degradation of the service.
For each request, if the client is able to provide extra details as request parameters, for ex: a timeout value in Epoch ms after which it abandons the request. We can use the timeout to drop the request if required after picking it up from the requests queue. Non-blocking Thrift Server provides the required functionality to access request parameters before processing the request.
Let’s take an example:
A simple Non-blocking Server interface would look like this in Java.
On the above asynchronous multiply endpoint, the non-blocking Server doesn’t wait for the method to finish processing. Instead, the Server gives us a callback AsyncMethodCallback<Integer> that is backed by NIO channel. When the result is ready, we need to set the result on this callback as shown below and the non-blocking Server will communicate that to the client.
To provide middleware mechanism, we introduce an interfaceRequestInterceptor that allows easy interception of Thrift requests (see below gist). Since we are using a non-blocking Server, we cannot haveRequestInterceptor implementations, let’s call them interceptors, that block the Server thread. Through AsyncMethodCallbackListener, the interceptors also get a chance to be notified about the result/exception that is set after processing the Thrift request. This helps us in clearing the state if maintained for a Thrift request. For example: for a concurrency limiting interceptor, we would like to know whether the request is processed successfully or not so that we can adjust the concurrency limit. An interface for an Interceptor looks like this:
Finally, we use Java dynamic proxy to wrap the MultiplicationHandler with multiple RequestInterceptor implementations. There are two types of RequestInterceptors in the below code: requestCreationInterceptors and requestPreProcessInterceptors on line 12 and line 16 respectively. requestCreationInterceptors are those that want to intercept as soon as a request is received. For example: a concurrency limiter would be able to decide whether a request can be served or not. After the requestCreationInterceptors, we delegate the task to an underlying Executor which is responsible for processing the request and setting the final result. Next, requestPreProcessInterceptors are those that want to intercept right before we begin processing the request. For example: request timeout interceptor would be able to decide whether the request timed out waiting on the queue.
And this is how we prepare the final handler. This Server is from a concrete example available here
This middleware mechanism can be applied on the client side in a similar fashion if the client is able to work with Asynchronous Iface of a Service.
Users might have different mechanisms around how they use Apache Thrift. So, the above RequestInterceptor logic could easily be updated to fit your use case. Overall, the base idea is to intercept requests using Java dynamic proxy and Thrift Asynchronous Server.
This code is hosted on Github along with an example: https://github.com/udaysagar2177/thrift-interceptor
I would love to hear from you if you have anything similar in your Thrift setup or if you found this helpful.
