Building Resilient Applications with Circuit Breaker Pattern

(portuguese version)

In the previous post, we present the Retry resilience pattern. However, what happens if the destination server is failing by an overhead? Making attempts will overload the server. The Circuit Breaker pattern can help us in these scenarios.

In electrical circuits a circuit breaker is a device that protects an electrical installation from overloading electrical currents as well as short-circuit. In distributed systems, the implementation of the circuit breaker pattern will help protect the target service.

This pattern is more complex than Retry because it is based on a state machine as shown in the following figure:

Circuit Breaker State Machine

State Machine:

• Closed: In electrical circuits, when a circuit breaker is closed indicates that electric current is passing. In the resilience pattern it means that you are passing “requests”;

• Open: in this state, the circuit does not pass the requests to the destination, only returns to the requester that the service is out, avoiding to overload the destination with new requests, besides providing feedback faster, without having to wait for the end of a request that probably will fail. The circuit goes to the open state after reaching a predefined threshold of faults and will also remain open for a predefined time; and

• Half-Open: This is a check state. After reaching the time in the Open state, the system goes to this state and the next request will be forwarded to the destination. If the destination returns a fault, the circuit returns to the Open state, if it returns successfully it goes to the Closed state.

The following animation illustrates the circuit breaker in action with a threshold of two consecutive faults to transition to the Open state (if you prefer a gif, click here):

Circuit Breaker with Polly

Again, Polly comes to make life easier for developers in implementing the circuit breaker pattern in their code, see how simple it is:

In the code, a circuit breaker policy has been set up for when there are two exceptions (failures) HttpRequestException consecutive, the circuit stops sending requests to the destination for 1 minute. During this time the circuit will raise a BrokenCircuitException.

Note in the code that the policy must be the same instance between the calls. This is because it is necessary to maintain the instance of the state machine.

Let’s see the code of a client consuming this service:

The Test_5_Calls method will have the following outputs:

Console Output

Notice that after the second consecutive failed call (HttpRequestException), the service starts to fail fast by returning a BrokenCircuitException.

Advanced Circuit Breaker Configuration

Imagine the scenario where the target service is being balanced and one of the servers is overloaded. In this scenario, it is likely that we will not have a number of consecutive failures to open Circuit Breaker.

The Polly framework provides the AdvancedCircuitBreaker policy that will allow a more robust configuration in which it will analyze a sampling of requests during a period and open the circuit if the number of failures reaches a certain percentage. It is worth exploring this policy, for more details see documentation.

Conclusion

The circuit breaker policy is very important to protect the target server as well as favor the user experience with a fast failure. Implementing this policy with Polly is relatively simple.

I hope you have enjoyed it, leave your feedbacks and continue to follow.