ThreadLocal is a java class which provide thread-local variables, i.e. variables specific to thread. Popularly used a lot for storing request context, for example requestId (in MDC for error reporting to sentry), or storing New Relic token for monitoring.
Hystrix
For those who don’t know much about hystrix, it is open source library with circuit breaker functionality. You can read more about it here.
Those of us who are used to distributed systems (for example Aggregation layer, or micro-service architecture) know that it is very much possible for a down stream to go down, or respond at higher latency than expected.
So how to prevent our service from deviating from its latency. This is where need for a circuit breaker pattern arises. It not only saves our service from becoming unavailable, also it gives the downstream services some breathing space.
Hystrix has two strategies, Semaphore and Thread. In this blog we focus mainly on Thread Strategy. So first of all lets discuss working of hystrix in short. So hystrix has 2 functions
- run() : contains primary execution code
- getFallback() : contains code to be run in case of fallback. There can be three reasons when this method is triggered, listed below.
In Thread strategy, hystrix runs the primary execution in different thread. Lets name that thread hystrix-command-*. Now there is a timer thread (named as hystrix-timer-*) this thread waits for t ms (specified timeout) for run() to execute. The thread on which fallback is executed depends on the reason due to which fallback was triggered
Now that we have discussed about how hystrix works with its thread, the problem that comes to mind is that how do we make sure that the threadLocal variables are transferred safely to these threads.
First thing that comes to mind is using InheritableThreadLocal(). Now I will give a short advice about its usage, DO NOT USE InheritableThreadLocal!!!
Why???? Because hystrix internally maintains thread pool, which allows thread to be reused. So the thread locals inherited first time in the hystrix thread will be there for all other calling threads, no matter what.
So now what?
Well hystrix developers had already thought about this problem. Hence they provided several plugins to tackle this. I will discuss two approach for this, the first one is a short one, with a little downsides, however the second one is best way for doing it.
Using HystrixConcurrencyStrategy
This plugin can simply be used to wrap the callable for hystrix run().
This works in a similar way like TaskDecorator. We extract the threadLocal id from caller thread and set it into the new thread invoked by hystrix.
So if this is so easy then whats the problem?
The issue with HystrixConcurrencyStrategy is that it wraps only the run() function. In simpler words if you use HystrixConcurrencyStrategy, ThreadLocal variables won’t be available in fallback methods thread (in case the thread for fallback is different, i.e. timeout).
If you don’t need threadLocal variables in fallback, then you are good to go. But If you need it there, then you need to use HystrixCommandExecutionHook.
Using HystrixCommandExecutionHook
HystrixCommandExecutionHook allows you to override several functions like onExecutionStart, onFallbackStart, etc. This gives you full control over the hystrix execution flow. I will now describe briefly some of the functions we need. If you want more about them, you can read it here.
- onStart(): executes when the hystrixCommand is called. Runs on the caller thread itself
- onSuccess(): executes if hystrixCommand.execute() is successful, i.e. either run() or getFallback() function have been successful. Runs in caller thread
- onError(): executes when hystrixCommand.execute() fails, i.e. both run() and getFallback() functions have failed. Runs in caller thread
- onExecutionStart(): executes just before run() code. Runs on the hystrix-command-thread
- onExecutionSuccess(): executes whenever the run() method executes successfully (without error or timeout). Runs on the hystrix-command-thread
- onExecutionError(): executes when run() method fails (timeout or any exception). Runs on the hystrix-command-thread
- onFallbackStart(): executes when fallback starts. Runs on the hystrix-command-thread OR hystrix-timer-* or caller thread.
- onFallbackSuccess(): executes when fallback ends successfully. Runs on the hystrix-command-thread OR hystrix-timer-* or caller thread.
- onFallbackError(): executes when fallback ends with any exception. Runs on the hystrix-command-thread OR hystrix-timer-* or caller thread.
So now that we have knowledge about full flow, we just need to pull out the threadLocals from calling thread and set it in any of the subsequent thread.
Simple isn’t? Well nice guess NO! :D There are several issues with the about code.
- Same HystrixHook object is shared across all the threads. So the value set to idTL can be changed if any other threads executes the hystrixCommand at the same time.
- As I mentioned before the fallback can run in calling thread as well, so onFallbackError()/onFallbackSuccess() methods can run in the caller thread, and hence the threadLocal variables will be cleared from the calling thread which is not required.
Final Solution
So what now? Don’t worry this is the final solution, promise ;)
Same HystrixHook object is shared across all the threads.
HystrixRequestVariableDefault is a kind of threadLocal, provided by hystrix for such usage. But unlike threadLocals, it has an scope to a hystrix request. . Since I will skip details about it, so you can read more about it here.
So this is how the final code for HystrixHook will look like
