Hystrix Circuit Breaker — How To Set It Up Properly

In one of the previous posts I have briefly described what is the Hystrix Circuit Breaker and why it’s worth to consider using it. Now I go more into details and explain what are the key parameters and how they affect its behavior.

October 19, Mielec, Park in the City — https://www.instagram.com/p/BabyPP0DAMg

When you apply circuit breaker concept with Hystrix into your source code you should set up appropriate values for these three key parameters:

• request volume threshold
• sleep window in milliseconds
• error threshold percentage.

What they mean and how they actually work?

Request Volume Threshold

A total number of requests that goes through the circuit breaker after which the breaking strategy is applied. When a number of requests are lower than this value, the breaking will never happen.

By this number, you can make sure circuit breaker starts its work on a certain throughput of requests.

What is important is that this number is calculated separately for each sleep window. So for example when you set it to 20 (default value), requests are short-circuited only after 20 requests volume in the sleep window. When the number of them is lower, they are all passed to downstream service — even if they all fail.

Error Threshold Percentage

This number controls when the requests should be short-circuited and not passed to downstream anymore. The default value is 50%.

So when you already reached the volume threshold of requests in a given sleep window, this number is used to determine whether subsequent requests should be cut. If at least 50% of previous requests failed, then no other request will be passed further and the fallback response will be served back.

If you want to be more sensitive and stop passing requests to a downstream resource you should set this number to the lower value. Then for example responses will be short-circuited after 10% of failed requests.

Sleep Window in Millis

The last very important property in Hystrix Circuit Breaker. It controls the size of the window of received requests. The default value is 5 seconds.

In order to circuit breaker work, all things must happen in that window. When in 5 seconds you will receive at least 20 requests (volume threshold) and the certain percentage of them will fail (error threshold) circuit will be opened for consecutive 5 seconds. After that time, the first request will be served to a downstream resource, and in case of success, the circuit will be closed again.

If you set up sleep window too high, then in case of failures your circuit will be opened for a long period of time. So if you want to measure execution in 30 seconds based window, keep in mind that it may result in 30-second circuit open period.

There are several ways of specifying those numbers. You can set up defaults with properties file, like this:

hystrix.command.default.circuitBreaker: 
  requestVolumeThreshold: 20 
  errorThresholdPercentage: 50 
  sleepWindowInMilliseconds: 5000

You can set them per command:

hystrix.command.<command_key>.circuitBreaker:   
  errorThresholdPerce ntage: 10

Or you can set them programmatically as well:

HystrixCommandProperties.Setter()
  .withCircuitBreakerErrorThresholdPercentage(int value)

For more information go directly to official docs.

Originally published at dmydlarz.com on October 31, 2017.