Dropwizard Client

Dropwizard is a lightweight RESTful Java framework based on Jetty app server, Jersey REST framework and Jackson JSON parser. The dropwizard-client module provides an easily configurable way to create clients in a Dropwizard project to call external REST (json) services.

The Dropwizard client is actually a Jersey client using Apache HttpClient providing an simple client builder api and a default configuration that suits better for a production environment.

See an example client and related code at dwexample-client with Dropwizard v1.1.0.

The main class to build a client is the io.dropwizard.client.JerseyClientBuilder (it’s different than Jersey’s org.glassfish.jersey.client.JerseyClientBuilder).

The builder typically needs:

• A Dropwizard Environment instance to bind itself to it’s lifecycle. The client has a thread pool executor service for asynchronous requests that is also started/stopped with the Dropwizard environment.
• A MetricRegistry that comes with the Dropwizard environment so you don’t have to specify another instance. Dropwizard’s metric system collects statistics about the client usage. It’s possible to create a client without a Dropwizard environment (e.g. for unit tests), but then a new MetricRegistry instance is needed.
• A Jackson ObjectMapper used for json marshaling. By default the ObjectMapper instance from the Dropwizard Environment is used, but you can create an own instance for the client. There is a useful factory method provided by Dropwizard:

• A JerseyClientConfiguration (extends HttpClientConfiguration) object that contains the actual configuration for the client. The configuration should be set in the Dropwizard yaml configuration file, but of course it can be built as regular POJOs (e.g. for unit tests).
• A name string that must be unique, it’s used in client async request thread names and in the MetricRegistry.

Based on my experience it’s much easier to build a REST client with Dropwizard than using the Jersey/Apache HttpClient configuration APIs. The default configuration also suits better for production than in the upstream libraries, but it’s still important to understand each configuration entries.

Socket timeouts

The connection related timeout settings in JerseyClientConfiguration:

• timeout: socket read timeout, practically how long the client waits for the server response. The default 500 ms is low and setting 1–10 sec may be needed based on your called service’s response time.
• connectionTimeout: timeout to establish a new connection. The default 500 ms should be increased to a few seconds especially with resources running in virtual/cloud or high network latency environments.

Connection pool

The client manages a (tcp) connection pool under the hood and it’s important to configure it properly for a microservices to achieve high throughput and avoid unnecessary connection overhead.

• maxConnection/maxConnectionsPerRoute: max number of connections (per hostname:port). The default value is more than enough, it allows 1024 parallel http requests.
• connectionRequestTimeout: determines how long we wait for a connection to become available and be returned by the pool if all the max connections are used in the moment of another request. The default is 500 ms and the client usually doesn’t have to wait at all if the pool size is big enough. The waiting time only starts if the pool couldn’t create a new connection because the max count was already reached.
• keepAlive: it should be set to a positive value (5–10 sec) to enable HTTP keep-alive and reuse connections. By default it’s disabled and every connection is closed after an http requests/reponse that makes the connection management not effective. The keepAlive value determines how long a persistent connection is kept open on the client side without being used (between two http requests). This value must be set less than the server side idleTimeout (defaults to 30 sec in Dropwizard server) otherwise the connection can be closed on the server’s end resulting a org.apache.http.NoHttpResponseException when the client tries to reuse the connection. (See ApacheHttpClient: Connection management).
• validateAfterInactivePeriod: could be enabled to help detecting the connections in the pool that have already been closed on the server side, but it’s not really needed if the keepAlive is set less than the server side idleTimeout.
• timeToLive: maximum time how long a connection can exist in the pool. This is only effective if the keepAlive is set (e.g. 5 sec) and the connection is used for http requests periodically (e.g. every second) for a long time. The connection can stay open up to the timeToLive limit, the default is 60 mins.

Retry

The http client level retry can be enabled by the “retries” property. The request will be retried the given number of times if there is an exception during the http transaction. (This means no retry in case of non-200 http responses or body parsing errors as the http request-response was valid.)

By default some exception types are ignored and the request is not retried, that might not suit your environment (e.g. running in containers) and still want to retry in these cases. To change the list of non-retried exceptions a custom org.apache.http.client.HttpRequestRetryHandler should be added to the client.

Enabling retry can also cause problems with POST requests resulting org.apache.http.client.NonRepeatableRequestException because the input stream for the request body can’t be read multiple times. The issue is related to chunked encoding that is enabled by default and a non-buffered stream is used by the client. Make sure to disable chunked encoding in the client if retry is enabled and POST requests are used to avoid the problem.

Async

The client can be used to execute http requests asynchronously. The request is sent and the response is received by another thread while the caller thread gets back a Future<javax.ws.rs.core.Response> object that can be used to get the actual response or exception later.

The client manages a java.util.concurrent.ThreadPoolExecutor executor service created based on the given properties:

• minThreads: number of active worker threads under normal load
• maxThreads: max number of worker threads started only if the queue is full
• workQueueSize: task queue size, the threads take tasks from the queue.

Dropwizard configuration limits the queue size at 16K, which is surprisingly low and the task queue can get full quickly if the called service became temporarily slow, but the threads calling the client keep requesting async calls at a high pace. This can cause a RejectedExecutionException when the caller tries to submit the async request.

(!) Note: In Dropwizard 1.0.x (and maybe earlier versions) the thread pool parameters above are ignored and an unlimited number of threads are used. The tasks are not queued, but a new thread is created for each request if needed. This can result a high number of worker threads if the called service is slow and eventually crash the JVM. The issue is fixed in Dropwizard v1.1.0

It’s possible to create your own ExecutorService and add to builder, but consider the impact of the settings:

• using an unbounded queue can fill up the heap
• allowing an unlimited number of threads can crash the JVM
• using another RejectedExecutionHandler causes message loss (DiscardPolicy) or practically switches back to synchronous execution (CallerRunsPolicy)