Architectural Patterns: Circuit Breaker

A deep dive into Circuit Breaker architectural pattern, with examples on Go

Welcome to the world of architectural patterns! In this article, we will explore one of the most essential architectural patterns used in distributed systems — the Circuit Breaker. The name “Circuit Breaker” comes from the analogy with electrical fuses, which protect electrical circuits from overload. Similarly, the Circuit Breaker protects distributed systems from overloads, failures, and other disruptions.

In distributed systems, the Circuit Breaker is used to control the availability and reliability of remote services. It helps to avoid overloads and system crashes when one of the services is temporarily unavailable. In cloud systems, the Circuit Breaker is an integral part of microservices architecture. It allows monitoring the availability and reliability of microservices and automatically managing the load on them. Using the Circuit Breaker pattern provides us with several advantages, such as improved availability, system reliability, reduced risk of failures, and better load management on services — avoiding unwanted requests to an unavailable service and ensuring the system works as expected if one or more services are temporarily unavailable.

In a microservice architecture, each of our services depends on others. If one of the internal or external services becomes unavailable, the services that depend on it will also malfunction. For example, if service G is unavailable, services D and E will also function improperly. Services A and B will only work partially since they depend on services D and E. In short, we must design a fault-tolerant architecture. The Circuit Breaker pattern can help us with that. If a remote service that we depend on is temporarily unavailable, we don’t crash with an error, but instead, execute a predefined action to return a value from the cache, send a request to an alternative service, return a default value, etc…

How it works

The Circuit Breaker pattern is based on a pattern state machine, where it switches between three different basic states, depending on what actions are performed in the application:

• Closed State — is the initial state in which all requests to the service are executed without change. If the application receives an error response, it repeats the request a specified number of times. If we do not get an answer from the service, it switches to the open state.
• Open State — when passing to the open state Circuit Breaker stops all requests to the service and starts returning predefined answer, which can be some kind of stub (predefined action). This allows to reduce the load on the service and give it time to recover. After a certain amount of time, Circuit Breaker goes to the third HalfOpen State.
• Half Open State — When moving to the HalfOpen State, Circuit Breaker begins making one request to the service to see if the service has recovered. If the service successfully completes the requests, Circuit Breaker switches back to the closed state, otherwise it switches back to the open state.

The Circuit Breaker pattern thus helps to prevent service congestion and increase application fault tolerance.

Practice

Let’s move on to practice. This pattern can be done in any programming language. But today I will implement it in Go. You can find a repository with an example at this link.

WARNING: The code below is for educational purposes and can’t be used in production.

We will write two elementary applications. The first will be a server application that will switch between successful and unsuccessful responses every 4 seconds.

package main

import (
 "fmt"
 "log"
 "net/http"
 "time"
)

var isStopped bool

func main() {
 http.HandleFunc("/hello", func(writer http.ResponseWriter, request *http.Request) {
  if isStopped {
   writer.WriteHeader(http.StatusInternalServerError)

   return
  }
  writer.Header().Add("Result", "Hello")
  writer.WriteHeader(http.StatusOK)
 })

 go func() {
  for range time.Tick(4 * time.Second) {
   isStopped = !isStopped
   fmt.Println("Is server works: ", isStopped)
  }
 }()

 log.Fatal(http.ListenAndServe(":8081", nil))
}

Here we create a handler at the address “/hello” that will return to us a successful or failed response, depending on what position the isStopped flag is set in. Then in a separate goroutine we will change the state isStopped to the opposite every 4 seconds in an infinite loop. Finally we start our server listening on port 8081.

The client application is a bit more complicated. In it we will send 10_000_000 requests to our server, but not directly but via our Circuit Breaker. If the server answers ok, we will print the result as it is and if the server returns an error we will do a predefined default behavior.

package main

import (
 "errors"
 "fmt"
 "net/http"
 "time"

 circuitBreaker "github.com/KRR19/CircuitBreaker/client/circuit-breaker"
)

func sendRequest() (string, error) {
 res, err := http.Head("http://localhost:8081/hello")
 if err != nil || (res.StatusCode < 200 || res.StatusCode > 499) {
  return "", errors.New("server failed")
 }

 return res.Header.Get("Result"), nil
}

func defaultAction() (string, error) {
 return "DEFAULT", nil
}

func main() {
 cb := circuitBreaker.NewCircuitBreaker(5, 3*time.Second, defaultAction)
 for i := 0; i < 10000000; i++ {
  result, err := cb.Call(sendRequest)
  if err != nil {
   fmt.Println("Error")
   continue
  }

  fmt.Println(result)
 }
}

At the start of the client we create a new instance of the CircuitBreake structure with the parameters: 5 — number of retries before we go to state open, 3 seconds — we will be in state open after we go to state half-open, and default action that we execute when we get an error when we run the main request.

And here’s how our circuitBreaker package looks like:

package circuitbreaker

import (
 "time"
)

type State int

const (
 StateClosed = iota
 StateOpen
 StateHalfOpen
)

type CircuitBreaker[T any] struct {
 State         State
 FailureCount  int
 Threshold     int
 Timeout       time.Duration
 LastFailure   time.Time
 DefaultAction func() (T, error)
}

func NewCircuitBreaker[T any](threshold int, timeout time.Duration, defaultAction func() (T, error)) *CircuitBreaker[T] {
 return &CircuitBreaker[T]{
  State:         StateClosed,
  FailureCount:  0,
  Threshold:     threshold,
  Timeout:       timeout,
  LastFailure:   time.Now(),
  DefaultAction: defaultAction,
 }
}

func (cb *CircuitBreaker[T]) Call(action func() (T, error)) (T, error) {
 switch cb.State {
 case StateClosed:
  return cb.stateClosedBehaviour(action)
 case StateOpen:
  return cb.stateOpenBehaviour(action)
 case StateHalfOpen:
  return cb.stateHalfOpenBehaviour(action)
 }

 panic("unknown circuit breaker state")
}

func (cb *CircuitBreaker[T]) stateClosedBehaviour(action func() (T, error)) (T, error) {
 success, err := action()
 if err != nil {
  cb.FailureCount++
  cb.LastFailure = time.Now()

  if cb.FailureCount >= cb.Threshold {
   cb.State = StateOpen

   return cb.DefaultAction()
  }

  return cb.stateClosedBehaviour(action)
 }

 cb.FailureCount = 0

 return success, err
}

func (cb *CircuitBreaker[T]) stateOpenBehaviour(action func() (T, error)) (T, error) {
 if time.Since(cb.LastFailure) >= cb.Timeout {
  cb.State = StateHalfOpen

  return cb.stateHalfOpenBehaviour(action)
 }

 return cb.DefaultAction()
}

func (cb *CircuitBreaker[T]) stateHalfOpenBehaviour(action func() (T, error)) (T, error) {
 success, err := action()
 if err != nil {
  cb.State = StateOpen
  cb.LastFailure = time.Now()

  return cb.DefaultAction()
 }

 cb.State = StateClosed
 cb.FailureCount = 0

 return success, err
}

Here we write our Circuit Breaker structure. I made it generic to make it as universal as possible for different response models. The next one is the factory method that will return the already configured instance of the struct. And of course our Call method that takes the action and passes it to the desired function depending on what state the Circuit Breaker is in.

Let’s take a closer look at those functions:

func (cb *CircuitBreaker[T]) stateClosedBehaviour(action func() (T, error)) (T, error) {
 success, err := action()
 if err != nil {
  cb.FailureCount++
  cb.LastFailure = time.Now()

  if cb.FailureCount >= cb.Threshold {
   cb.State = StateOpen

   return cb.DefaultAction()
  }

  return cb.stateClosedBehaviour(action)
 }

 cb.FailureCount = 0

 return success, err
}

We execute the given action. If it runs successfully, we reset the FailureCount to zero and return the result. And if it runs with an error, we increase the FailureCount. Set LastFailure to the current time. If FailureCount is greater than or equal to the pre-specified Threshold then we change the state to open and return the result of the default action. Otherwise, we recursively call the stateClosedBehaviour function until the base script for our recursion works.

func (cb *CircuitBreaker[T]) stateOpenBehaviour(action func() (T, error)) (T, error) {
 if time.Since(cb.LastFailure) >= cb.Timeout {
  cb.State = StateHalfOpen

  return cb.stateHalfOpenBehaviour(action)
 }

 return cb.DefaultAction()
}

If we are in an open state, we check how much time has passed since the last unsuccessful request. If the passed time is more than the time specified in Timeout, we move the HalfOpen state and call the stateHalfOpenBehaviour(). If not enough time has passed since the last failed request, we will immediately call the default action.

func (cb *CircuitBreaker[T]) stateClosedBehaviour(action func() (T, error)) (T, error) {
 success, err := action()
 if err != nil {
  cb.FailureCount++
  cb.LastFailure = time.Now()

  if cb.FailureCount >= cb.Threshold {
   cb.State = StateOpen

   return cb.DefaultAction()
  }

  return cb.stateClosedBehaviour(action)
 }

 cb.FailureCount = 0

 return success, err
}

When Circuit Breaker is in HalfOpen state, we try to execute the request once, if it is successful, we change the state to Closed, reset the FailureCount to zero and return the result of the request. If it fails, we change the state to Open, set LastFailure to the current time and execute the default action.

Conclusion

The Circuit Breaker pattern is an important design pattern. It improves the fault tolerance of distributed systems. This pattern is particularly useful for preventing cascading failures in systems that depend on multiple services or components.

By implementing Circuit Breaker circuitry, you can get many advantages, including increased resiliency, fault tolerance, and reduced response time. This helps prevent cascading failures in distributed systems and increases the resiliency of the system as a whole. The Circuit Breaker Pattern allows a service or component to degrade smoothly, preventing the entire system from failing. If you are working on a distributed system, you should consider using Circuit Breaker Pattern to improve its reliability and fault tolerance.