The Circuit Breaker Pattern

In this article we will give an overview of the circuit breaker pattern and explore one Golang package providing an implementation.

Circuit Breaker Definition

Context

Resiliency is one of the architectural qualities in systems design. It is the ability of a given system to adapt and recover from failures. The circuit breaker is a pattern intended to increase the resiliency of a system.

Use cases

In a system it is possible that a key service becomes unresponsive.

The risk when one service is unresponsive is that the bottleneck created can bring other components of the system down.

Example scenario

Pizzeroo the on-demand pizza company has launched the 1 million free pizzas marketing campaign at 12am on their website. At 12.15 the traffic is 10 times higher than usual and users are experiencing longer and longer response times on the website because the Order service has reached its maximum capacity. Auto-scaling is in place but it takes 10 minutes for new instances to be created. Users are getting frustrated and keep hammering the submit button to not miss out on the offer, which creates even more requests. At 12:20 system administrators start noticing that services consuming the Order service start running out of memory because requests pile up in memory. At 12:22, the website is still up but all requests are now timing out, users are furious and start posting on social media how upset they are at Pizzeroo, investors take notice and the company’s shares start to plummet…

This scenario is completely fictional of course however it is realistic enough to give a good use case for a circuit breaker.

Solution

Administrators and developers at Pizzeroo have studied this production incident and decided that there needs to be a circuit breaker around the Order service, they also realised that scaling up the service would not have helped because Order is dependent on legacy third party APIs that cannot handle a very high load.

A circuit breaker can be used to detect when a downstream dependency starts failing and returns an error immediately instead of waiting until the requests time out. Stopping requests going through prevents the problem from spreading and notifies clients that the system is temporarily down instead of slowly collapsing.

Next time there is a high load on Pizzeroo’s website and the Order service reached its maximum capacity, requests will fail immediately and the frontend can even disable the submit button and display a friendly message while the circuit breaker is activated. The Order service is given some time to process requests and has some capacity again and users can submit their forms.

A good analogy : the Order service getting a break while it processes current requests…

The 3 States of a Circuit Breaker

As we have seen the circuit breaker essentially decides whether to let a request go through or not. This decision is conditioned by 3 different states which can be thought of as a state machine.

• Closed : Non-intuitively this state means that requests will go through, as no faults or timeouts were detected.
• Open : In this state all requests are temporarily rejected for a set period of time. This gives the opportunity for the failing service to recover.
• Half-open : This state always follows the Open state, a limited number of requests are allowed to go through, if they fail we go back to the Open state, if they succeed we assume the dependency has recovered and we go to the Closed state.

Circuit Breaker Decision Diagram

Service Meshes

For cloud native applications the ideal place to have a circuit breaker is at the infrastructure level with a service mesh. It comes out of the box and most of the time only needs configuration without any specific service logic. Sometimes a mesh is not an option in a given project and that’s when the following section intends to help.

Implementation in Golang

Adding a circuit breaker in your code is sometimes the only option available.

Cons : If many services need it then the same code would have to be copied over which is definitely an anti-pattern. Ideally it would sits in one custom package with its own repository.

Sony offers the https://github.com/sony/gobreaker package. Which in my experience is very reliable, compact and easy to use. Below is a functional example, the circuit breaker decorates http.Client and executes before and after the actual http request is made.

This gist will print :

0 closed
1 closed
Get https://404.google.com: lookup 404.google.com: no such host
2 closed
Get https://404.google.com: lookup 404.google.com: no such host
3 open
circuit breaker is open
4 open
5 half-open
6 closed
Get https://404.google.com: lookup 404.google.com: no such host
7 closed
Get https://404.google.com: lookup 404.google.com: no such host
8 open
circuit breaker is open
9 open
10 half-open

Alerting and Monitoring

The team responsible for a service with a circuit breaker as well as infrastructure engineers will probably want to know whenever the circuit breaker is open. It is important then to have an event emitted when the state changes. Don’t forget to add another alert to notify when the circuit breaker is in closed state, everyone likes good news.

I hope this article was useful to you and has provided some material for making your system more resilient.