Protecting your Kotlin microservices with Circuit Breaker pattern

Energy Circuit Breaker

This blog post is part of a section on my blog where I aim to explain “complex” concepts in a simple manner, using practical examples in Kotlin and keeping the explanations concise. Additionally, I will provide references and additional readings for those who wish to delve deeper into the topic.

A Circuit Breaker is a fault-tolerant design pattern, designed to help an application handle error conditions and prevent it from crashing when a system it relies on is performing poorly or has failed. It operates in a manner similar to an electrical circuit breaker, which safeguards home appliances from burning out during overcurrent events and can be reset once normal operation is restored. The principle behind this pattern is straightforward: when an application’s external dependency experiences an outage, the best course of action is to cease calling it.

Now, in the context of software, a Circuit Breaker (CB) is essentially a state machine that monitors external system calls and decides whether the circuit should be in one of three states:

1. Closed: This state occurs when normal calls are executed without any issues.
2. Open: It’s activated when calls cannot be executed, often due to a certain number of errors occurring within a specific time frame.
3. Half-open: In this state, the CB is prepared to attempt a new request, and it can either close or open again. Determining if the CB is in a half-open state can be accomplished through various methods, such as transitioning it after a set period or when a specific number of blocked calls have occurred.

circuit breaker states

Now, let’s delve into the implementation to gain a clearer understanding. This state property pertains to a time-based reset Circuit Breaker, expanding upon what we discussed earlier.

val state: State
    get() = when {
        failureCounts >= failureThreshold && nowEpochMillis() - lastFailureTime > resetTimeout -> HALF_OPEN
        failureCounts >= failureThreshold -> State.OPEN
        else -> CLOSED
    }

Now, we need to create a wrapper function for executing operations, which will monitor errors and update failure counts. Additionally, when the Circuit Breaker (CB) is in a half-open state and we receive a successful result, we can reset it to the closed state. In this context, we can utilize Kotlin’s runCatching.

private fun <T> recordExecution(originalCall: () -> T): T {
    return runCatching { originalCall() }
        .onFailure {
            failureCounts ++
            lastFailureTime = nowEpochMillis()
        }
        .onSuccess {
            if (state == HALF_OPEN) reset()
        }
        .getOrThrow()
}

That concludes the more intricate aspects of the implementation. Now, all that remains is to create our class and implement the function that wraps the original call.

class CircuitBreaker(
    private val failureThreshold: Int,
    private val resetTimeout: Long,
) {
    private var failureCounts = 0
    private var lastFailureTime: Long = 0

    private val state: State
        get() = ...

    fun <T> call(originalCall: () -> T) : T {
        when(state){
            State.OPEN -> throw IllegalStateException("Circuit Break is Open")
            else -> return recordExecution(originalCall)
        }
    }

    private fun <T> recordExecution(originalCall: () -> T): T = ...

    private fun reset(){
        failureCounts = 0
        lastFailureTime = 0
    }

    enum class State {OPEN, CLOSED, HALF_OPEN}

    private fun nowEpochMillis() = now().toEpochMilli()
}

That covers the fundamentals in a simple, brief, and practical manner. However, it’s essential to note that this code isn’t production-ready, as it lacks thread safety. In a multi-threaded environment, failure updates could be lost due to race conditions. Fortunately, there are several libraries available (see the references and extras section) that provide production-ready implementations of Circuit Breaker patterns for seamless integration into your application.

A few additional comments.

• Similarly to how electrical circuit breakers prevent overcurrent, it is common to find CB implementations with built-in support for preventing request timeouts, delays, or other network issues, in addition to handling generic exceptions.
• It is important for the system to detect when a CB is in an ‘open’ state. A common practice is to generate a system alert, metric, or log that can be monitored by the system or its administrators. Libraries provide usually event listeners for this purpose.

References & extras

• “Martin Fowler’s website features an excellent blog post explaining this concept.
• The book Release It! provides an in-depth explanation of the pattern.
• You can seamlessly integrate a Circuit Breaker into your Java/Kotlin code using libraries such as Resilience4j and Failsafe.