Spring Cloud Circuit Breaker Implementation Using Netflix Hystrix and Reactive Feign
Currently I’m working in Orchestration service that hit many downstream services. Sometimes, when one of my downstream service got issue that makes his response slower, response time of my service got slower too. Since response time got slower, sometimes my service return read timeout error.
Based on behavior that I mention above, those behavior has disadvantage for both my upstream and downstream services. For my upstream service, they need more times to get response from my service. And when they got the response, sometimes they got error response, since my service waiting to long for my downstream response. When this case is happening, my upstream service will retry the request, and wait again for the response.
For my downstream service, since he got issue that makes his response slower, he need time to recovery. Since my upstream service always retry the error request, there are still many request comes to my downstream service. Many request that comes give burden to my downstream service, since he need times to recovery but we still give them many loads.
To solve my problem, I use circuit breaker pattern. Simply, it’s a pattern that allowing us to cut the request to our downstream service if many error occurs from that downstream service. And we can as soon as possible give proper response to our upstream service. With this solution, I can solve my problem. I can give proper response to my upstream service that says I cannot process your request, as soon as possible. And I can reduce burden for my downstream service, since I reduce the load that come to him.
Spring Cloud Netflix Hystrix
Currently hystrix is no longer in active development, and is in maintenance mode. For spring cloud integration, hystrix is only supported until spring 2.3.x. Above 2.3.x version, we cannot use hystrix with spring cloud.
Application Dependencies
We will use several spring dependencies to support our example.
- org.springframework.boot:spring-boot-starter-webflux → dependency for spring webflux
- org.springframework.boot:spring-boot-starter-actuator → dependency for spring boot actuator that provides several production-ready endpoint that help us to monitor our application
- org.springframework.cloud:spring-cloud-dependencies → spring cloud dependency that contain all dependencies to use spring cloud features
- org.springframework.cloud:spring-cloud-starter-netflix-hystrix → dependency for netflix hystrix
- org.springframework.cloud:spring-cloud-starter-netflix-hystrix-dashboard → dependency for netflix hystrix dashboard that used to monitor current condition of hystrix
- com.playtika.reactivefeign:feign-reactor-webclient → dependency for feign with reactor webclient
- com.playtika.reactivefeign:feign-reactor-cloud2 → dependency to support feign reactor and hystrix integration
- com.playtika.reactivefeign:feign-reactor-spring-configuration → dependency to support feign reactor and spring integration
You can generate skeleton of project, using start.spring.io. Then add additional dependencies above.
This is the pom.xml file example (please adjust some part from this snippet based on your needs).
Downstream Service
We will use https://httpbin.org/#/Dynamic_data/get_delay__delay_ to test our implementation. That API is an open API that allow us to set delay for the API response, it is suitable for the scenario that we will try.
Firstly we need to create feign client that will hit Delay Open API.
From reactive feign client that we created above, spring reactive feign will automatically create the implementation to call the downstream service.
Application Controller and Service
In previous section, we already create a feign client to hit our downstream service. So on this section we will create a controller and service class that will be hit by our upstream service.
Wrap It Up!
At this point, all codes that we need to implement hystrix is complete. We only need to set several annotations in our spring boot application class, and set several properties.
From snippet code above, we add several annotations.
- @EnableFeignClients → Tell spring that we implement feign client and need help to auto-configure the feign client
- @EnableReactiveFeignClients → Tell spring that we implement reactive feign client and need help to auto-configure the reactive feign client
- @EnableHystrix → Tell spring that we implement hystrix and need help to auto-configure hystrix
- @EnableHystrixDashboard → Tell spring that we implement hystrix dashboard and need help to auto-configure the hystrix dashboard. We will discuss about it in later section.
Beside the annotation that we set, we need to set several properties also to custom the behavior of our application.
From snippet code above, we add several properties.
- server.servlet.context-path → we set context-path for our web application, so if we will hit our endpoint, we will hit it under /demo path.
- spring.cloud.loadbalancer.ribbon.enabled → this is property to enable ribbon client load balancing. As default it will true if we have netflix ribbon dependency. Currently we have netflix ribbon dependency from reactive feign spring configuration, so we need to set it as false, since we not use it.
- management.endpoint.health.show-details → this is property to tells actuator to enable endpoint to show health information of our application or not. By default, actuator never show detail of our application health information, so we need to set it as always to make it always show detail of our application health.
- management.endpoints.web.exposure.include → actuator provide many endpoints to make us able to monitor our application. For testing purpose we will show all of actuator endpoints.
- reactive.feign.client.config.{reactive-feign-client-name}.options → this property is used to set option properties for our reactive feign client. It used feign client name that we set in ReactiveFeignClient annotation. We use this property to set read timeout and connect timeout threshold for our downstream service.
- hystrix.dashboard.proxy-stream-allow-list → this property is used to set list of hosts that we will allowed to set connection for our hystrix data stream. This is for hystrix dashboard purpose, we will discuss it in later section.
- hystrix.command.default.execution.isolation.strategy → this property is used to set what execution strategy that we used for hystrix. We used “SEMAPHORE” strategy as default configuration, since we build non-blocking application that not depends on many threads. You can read more about this property in this page.
- hystrix.command.{command-name}.execution.timeout.enabled → this property is used to enable or disable execution timeout for specific hystrix command. By default hystrix have it’s own execution timeout configuration, but in this article, we only depends on client timeout configuration, so we disable this configuration.
- hystrix.command.{command-name}.circuitBreaker.requestVolumenThreshold → this property used to set minimum number of request that should be occurs before circuit breaker start to check statistics. You can read more about this property in this page.
- hystrix.command.{command-name}.circuitBreaker.errorThresholdPercentage → this property used to set percentage threshold for error that occurs. If percentage of error that occurs exceed the threshold, circuit breaker will open it circuit. You can read more about this property in this page.
- hystrix.command.{command-name}.metrics.rollingStats.timeInMilliseconds → this property used to set number of time of window before it will move to new window and reset it’s statistics. You can read more about this property in this page.
- hystrix.command.{command-name}.metrics.rollingStats.numBuckets → this property is used to set number of window. You can read more about this property in this page.
From properties above, we need command-name to set several properties. When implement hystrix with reactive feign, each of reactive feign client will wrapped by hystrix command. Each hystrix command name will follow reactive feign client class name, method name, and method parameter types.
{Feign-Class-Name}#{Method-Name}({Parameter-Types}{,})
Command name will follow above structure. So if we have getBinWithDelayInSeconds with int parameter in HttpBinClient, hystrix command name will be HttpBinClient#getBinWithDelayInSeconds(int).
Testing
Based on our configuration in application.yaml, to test circuit breaker implementation, we need to sent at least 10 request during 100s range. To make circuit breaker open, at least 50% of the request that we sent is failed.
Our API url for testing is http://localhost:8080/demo/get
To hit with success response, we can try using 1s for delayInSeconds query parameter. It will give us success response.
http://localhost:8080/demo/get?delayInSeconds=1To hit with failed response due to read timeout. We can try using 3s for delayInSeconds query parameter, since our read timeout configuration is 2s.
http://localhost:8080/demo/get?delayInSeconds=3If you hit with failed response 8 times more, it will give you different error response since our circuit breaker is OPEN and give us circuit breaker is OPEN failed response.
Hystrix Dashboard
Other benefits using Netflix Hystrix is they come with user friendly ready-to-use dashboard. Dashboard will show statistics of each hystrix command, including the health of circuit breaker, success command, failed command, command that got timeout, etc.
To enable hystrix dashboard, we simply need put org.springframework.boot:spring-boot-starter-actuator and org.springframework.cloud:spring-cloud-starter-netflix-hystrix-dashboard in our service dependencies. Then put @EnableHystrixDashboard in our configuration class to tell spring to auto-configure the hystrix dashboard. We already do this in previous section.
To show dashboard we can go into http://localhost:8080/demo/hystrix, it will shows this page.
We need to put hystrix stream url in the dashboard, our hystrix stream url is http://localhost:8080/demo/actuator/hystrix.stream. You can find hystrix stream url from actuator path, http://localhost:8080/demo/actuator. If all field is filled, you can click monitor stream, it will show you the dashboard.
Dashboard will be empty if your service didn’t receive any request yet. You can try to hit an API and the data will shown in the dashboard. This is the example when I try to hit get path with failed response. Dashboard show me that I already sent request but failed, and all requests that come is failed.
Conclusion
Netflix hystrix is great fault tolerance library that support circuit breaker pattern. But since it’s in maintenance mode and not in active development, and if you used it in application that in active development, I prefer you use another library that is in active development, like Resilience4J. Resilience4J also already supported by spring cloud.
