Failing fast with a circuit breaker

library(garnish)

retry() helps when a failure is brief. It does not help when a service is down for minutes: every caller keeps waiting on it, retrying it, and adding load to something that is already struggling. circuit_breaker() handles that case. After a run of failures it stops calling the function and fails immediately, then tries again after a cooldown. It is a port of Python’s pybreaker.

It takes two settings:

A service that is down

The examples use a function that always fails, and count how many times it actually runs:

calls <- 0
service <- function() {
  calls <<- calls + 1
  stop("service down")
}

guarded <- circuit_breaker(failures = 2, reset_after = 1) %@% service

The first two calls reach service() and fail with its own error:

guarded()

#> Error in <environment>$fn(): service down

guarded()

#> Error in <environment>$fn(): service down

That was the second failure in a row, so the circuit is now open. The next call is rejected without running service() at all:

guarded()

#> Error:
#> ! Circuit is open: the function failed repeatedly and is not being called.

calls

#> [1] 2

calls is still 2. The caller got an answer straight away instead of waiting on a service that is known to be down.

The three states

The circuit is always in one of three states:

Waiting out the cooldown shows the trial call. Here the service comes back while the circuit is open, so the trial succeeds and the circuit closes:

healthy <- FALSE
fn <- function() if (healthy) "back up" else stop("service down")
guarded <- circuit_breaker(failures = 2, reset_after = 0.5) %@% fn

try(guarded(), silent = TRUE)
try(guarded(), silent = TRUE)
guarded() # open: rejected

#> Error:
#> ! Circuit is open: the function failed repeatedly and is not being called.

healthy <- TRUE
Sys.sleep(0.6) # wait out the cooldown
guarded() # half-open: the trial call succeeds and closes the circuit

#> [1] "back up"

guarded() # closed again

#> [1] "back up"

If the service had still been down, the trial would have failed and the circuit would have opened for another full cooldown.

What counts as a failure

Only errors count. A call that returns normally, whatever its value, is a success and resets the count. So the circuit opens on failures errors in a row, not failures errors in total:

outcomes <- c("error", "ok", "error", "ok")
i <- 0
fn <- function() {
  i <<- i + 1
  if (outcomes[[i]] == "error") stop("failed") else "ok"
}
guarded <- circuit_breaker(failures = 2) %@% fn

try(guarded(), silent = TRUE)
guarded() # the success resets the count

#> [1] "ok"

try(guarded(), silent = TRUE)
guarded() # still closed: the errors were never consecutive

#> [1] "ok"

Handling an open circuit

A rejected call signals an error of class garnish_circuit_open. Catch it separately from the function’s own errors to fall back to something else, such as a cached value, instead of failing:

fn <- function() stop("service down")
guarded <- circuit_breaker(failures = 1, reset_after = 60) %@% fn
try(guarded(), silent = TRUE) # opens the circuit

fetch_or_default <- function() {
  tryCatch(
    guarded(),
    garnish_circuit_open = function(cnd) "default value"
  )
}
fetch_or_default()

#> [1] "default value"

Errors the function raises itself still reach you unchanged, so you can tell “the service failed” apart from “we did not try”.

Using it with retry

The two decorators answer different questions, and they work well together. retry() covers a brief failure. circuit_breaker() covers a long one.

Put circuit_breaker() outside retry():

fetch := circuit_breaker(failures = 3, reset_after = 30) %@%
  retry(stop = stop_after_attempt(3), wait = wait_exp(min = 0.1)) %@%
  function(url) {
    # ...
  }

In a %@% b %@% f, a is outermost. With the breaker outside, an open circuit rejects the call before any retries happen. Each call to fetch() that exhausts its three attempts counts as one failure, so three failed calls (nine attempts) open the circuit.

With the breaker inside, it would count every individual attempt. Retries would trip it quickly, and an open circuit would then be retried against, which defeats the point.

If you also use rate_limit(), it goes inside retry(), so the order from outside to inside is circuit_breaker(), retry(), rate_limit().

With promises

async_circuit_breaker() is the version for functions that return a promise, such as httr2::req_perform_promise() or a mirai task. It takes the same arguments and keeps the same three states, and a call fails when its promise rejects. An open circuit returns a promise that is already rejected with a garnish_circuit_open error, without calling the function.

down <- function() promises::promise_reject(simpleError("service down"))
fetch <- async_circuit_breaker(failures = 2, reset_after = 60) %@% down

# Failures are counted when a promise settles, so let each call settle
try_once <- function() {
  promises::then(
    fetch(),
    onRejected = function(err) cat(class(err)[[1]], "\n")
  )
  while (!later::loop_empty()) later::run_now(0.1)
}
try_once()

#> simpleError 

try_once()

#> simpleError 

try_once() # the circuit is open: down() is not called

#> garnish_circuit_open

Asynchronous calls overlap, which adds a few rules. A failure is counted when its promise rejects, so calls made before earlier ones have settled all go through. While the circuit is half-open only one trial call runs, and other calls are refused until it settles. A call that started before the circuit opened and finishes after it does not close the circuit. A trial that never settles is abandoned after reset_after seconds, so put async_timeout() inside to bound it sooner.

Stack it outside async_retry() in the same way as above, and see the retry vignette for async_timeout().

Notes