Spring Cloud Circuit Breaker Implementation Using Resilience4j and Reactive Feign

image source : https://files.readme.io/8701241-small-Resilience4j.png

Last time, I wrote an article about Circuit Breaker implementation using Netflix Hystrix in Spring Boot with Reactive Feign. Netflix Hystrix is one popular of circuit breaker library, but currently it’s in maintenance mode and not in active development.

So in this article I came with other popular circuit breaker library, Resilience4j. Resilience4j is lightweight, circuit breaker library that inspired by Netflix Hystrix. You can read more about Resilience4j in this page. As you can see in Resilience4j page, they have many features. But in this article I only write about their circuit breaker feature.

Application Dependencies

We will use several spring dependencies to support our example.

1. org.springframework.boot:spring-boot-starter-webflux → dependency for spring webflux
2. org.springframework.boot:spring-boot-starter-actuator → dependency for spring boot actuator that provides several production-ready endpoint that help us to monitor our application
3. org.springframework.cloud:spring-cloud-dependencies → spring cloud dependency that contain all dependencies to use spring cloud features
4. org.springframework.cloud:spring-cloud-starter-circuitbreaker-reactor-resilience4j → dependency for resilience4j circuit breaker spring cloud starter
5. com.playtika.reactivefeign:feign-reactor-webclient → dependency for feign with reactor webclient
6. com.playtika.reactivefeign:feign-reactor-cloud2 → dependency to support feign reactor and hystrix integration
7. com.playtika.reactivefeign:feign-reactor-spring-configuration → dependency to support feign reactor and spring integration

You can generate skeleton of project, using start.spring.io. Then add additional dependencies above.

This is the pom.xml file example (please adjust some part from this snippet based on your needs).

Resilience4j Circuit Breaker Config

First thing that we need to prepare is ReactiveResilience4JCircuitBreakerFactory Bean. This factory bean is used to create Circuit Breaker Object for each Reactive Feign Client that we used. That factory bean will also set all configurations that needed by Circuit Breaker.

Actually, that bean is already created by spring auto-configuration. But somehow, when I tried using default bean from spring auto-configuration. I cannot change the timeout configuration, so I always use default timeout configuration. This bother me, since if my downstream need more time to process the request, I cannot extend the timeout configuration, so it always got timeout.

From our custom ReactiveResilience4JCircuitBreakerFactory Bean above, we also able to create specific event handler to handle any events that published by Circuit Breaker. As you can see in line 38, currently I set handler to write log if any events occurs in Circuit Breaker.

There are many events that published by Circuit Breaker like on success, on error, on state transition, etc. In line 38 I set handler to listen for all events that published, not for specific event. You can customize according to your needs.

Get Default Circuit Breaker Config from Properties

Another challenge to use Resilience4j Circuit Breaker is to add default config for all Reactive Feign Client from properties.

The case is I want to set default config according to my needs that applied to all Reactive Feign Client. To handle that case, I need to set config for all Reactive Feign Client manually. That solution is takes times, and will make my properties file full of Circuit Breaker config even though all config is same.

So I came up with solution to customize logic to set Circuit Breaker config. I make Circuit Breaker config always read “default” config from properties. So I didn’t need to set config for all Reactive Feign Client manually.

Reactive Feign Circuit Breaker Config

Each Circuit Breaker that create will have it’s own id. For Reactive Feign Client, Circuit Breaker will build id from Feign Client name. Below is Reactive Feign Client name convention.

{Feign-Class-Name}#{Method-Name}({Parameter-Types}{,})

As you can see, that Reactive Feign Client name will have name that contains ‘#’, ‘(‘, ‘)’, and ‘,’ characters. Unfortunately, Resilience4j Circuit Breaker id cannot contains those characters. So the id will be invalid, and we cannot set specific Circuit Breaker config based on id.

To handle that issue, I came up with solution to change the Reactive Feign Client name before it used by Circuit Breaker as id. I will convert all of those characters to “_” character, and remove latest “_” character. So the id will be like below convention.

{Feign-Class-Name}_{Method-Name}_{Parameter-Types}{_}

Downstream Service

We will use https://httpbin.org/#/Dynamic_data/get_delay__delay_ to test our implementation. That API is an open API that allow us to set delay for the API response, it is suitable for the scenario that we will try.

Firstly we need to create Feign Client that will hit Delay Open API.

From Reactive Feign Client that we created above, Spring Reactive Feign will automatically create the implementation to call the downstream service.

Application Controller and Service

In previous section, we already create a Feign Client to hit our downstream service. So on this section we will create a controller and service class that will be hit by our upstream service.

Wrap It Up!

At this point, all codes that we need to implement Circuit Breaker Resilience4j is complete. We only need to set several annotations in our spring boot application class, and set several properties.

From snippet code above, we add several annotations.

1. @EnableFeignClients → Tell spring that we implement Feign Client and need help to auto-configure the Feign Client
2. @EnableReactiveFeignClients → Tell spring that we implement Reactive Feign Client and need help to auto-configure the Reactive Feign Client

Beside the annotation that we set, we need to set several properties also to custom the behavior of our application.

From snippet code above, we add several properties.

1. spring.webflux.base-path → we set base-path for our web application, so if we will hit our endpoint, we will hit it under /demo path.
2. spring.cloud.loadbalancer.ribbon.enabled → this is property to enable ribbon client load balancing. As default it will true if we have netflix ribbon dependency. Currently we have netflix ribbon dependency from reactive feign spring configuration, so we need to set it as false, since we not use it.
3. management.endpoint.health.show-details → this is property to tells actuator to enable endpoint to show health information of our application or not. By default, actuator never show detail of our application health information, so we need to set it as always to make it always show detail of our application health.
4. management.health.circuitbreakers.enabled → this is property to tells actuator to enable endpoint to show health of our Circuit Breaker. Circuit Breaker health information show statistics for each Circuit Breaker, including failure rate, failure rate threshold, failed calls, Circuit Breaker state, etc.
5. reactive.feign.loadbalancer.enabled → this is property to tells Reactive Feign Client to not use load balancer. As default it will true, since Reactive Feign Client come up with Spring Cloud Starter Load Balancer dependency.
6. reactive.feign.client.config.{reactive-feign-client-name}.options → this property is used to set option properties for our reactive feign client. It used feign client name that we set in ReactiveFeignClient annotation. We use this property to set read timeout and connect timeout threshold for our downstream service.
7. resilience4j.circuitbreaker.configs.default.registerHealthIndicator → this property is used to tells Circuit Breaker to register it’s health indicator in actuator application health information. We set config in ‘default’, so all of our Circuit Breaker will have this config.
8. resilience4j.circuitbreaker.instances.{Circuit-Breaker-Id}.minimumNumberOfCalls → this property is used by Circuit Breaker to tell when they start to calculate the failure rate. So if we set the value as 10, Circuit Breaker will wait until they receive 10 request before calculate the failure rate. This property is configure for specific Circuit Breaker id, as example we set it for HttpBinClient_getBinWithDelayInSeconds_int id. You can read more about this property in this page.
9. resilience4j.circuitbreaker.instances.{Circuit-Breaker-Id}.failureRateThreshold → this property is used to set percentage threshold for error that occurs. If percentage of error that occurs exceed the threshold, circuit breaker will open it circuit. You can read more about this property in this page.
10. resilience4j.timelimiter.configs.default.timeoutDuration → this property is used to set how long Circuit Breaker will wait until it cancel the process and throw timeout error. We set config in ‘default’, so all of our Circuit Breaker will have this config. You can read more about Time Limiter from this page.

Testing

Based on our configuration in application.yaml, to test Circuit Breaker implementation, we need to sent at least 10 request, and after 100 request Circuit Breaker will reset the data. To make Circuit Breaker open, at least 50% of the request that we sent is failed.

Our API url for testing is http://localhost:8080/demo/get

To hit with success response, we can try using 1s for delayInSeconds query parameter. It will give us success response.

http://localhost:8080/demo/get?delayInSeconds=1

To hit with failed response due to read timeout. We can try using 3s for delayInSeconds query parameter, since our read timeout configuration is 2s.

http://localhost:8080/demo/get?delayInSeconds=3

If you hit with failed response 8 times more, it will give you different error response since our circuit breaker is OPEN and give us circuit breaker is OPEN failed response.

If you open application actuator health information, you will find statistics about your Circuit Breakers.

Our application actuator health information url is http://localhost:8080/demo/actuator/health

Conclusion

Since Netflix Hystrix is currently in maintenance mode, we can use Resilience4j as replacement. Resilience4j is easy to use and have many features other than Circuit Breaker. If you have interest and want to read more about Resilience4j, you can go to this page.