Retrying failing calls

library(garnish)

Some failures are temporary: a network blip, a rate limit, a lock held by someone else. retry() calls the function again until it works or you tell it to give up. It is a port of Python’s tenacity, and is built from three questions:

Each has a default, so retry() on its own retries any error, immediately, forever. That is rarely what you want, so you will usually set stop.

A function that fails at first

The examples use a function that fails twice and then succeeds:

make_flaky <- function(fail_times) {
  calls <- 0
  function() {
    calls <<- calls + 1
    if (calls <= fail_times) stop("not yet (call ", calls, ")")
    paste("worked on call", calls)
  }
}

Apply retry() with %@%, or by calling it on the function:

flaky <- retry(stop = stop_after_attempt(5)) %@% make_flaky(2)
flaky()

#> [1] "worked on call 3"

Deciding when to stop

stop_after_attempt(n) makes at most n attempts. When it fires, a retry_error condition is signalled:

never_works <- retry(stop = stop_after_attempt(3)) %@% make_flaky(Inf)
never_works()

#> Error: Retry failed after 3 attempt(s): not yet (call 3)

The condition records what happened, so you can catch it and look:

fresh <- retry(stop = stop_after_attempt(3)) %@% make_flaky(Inf)
err <- tryCatch(fresh(), retry_error = identity)
err$attempt

#> [1] 3

conditionMessage(err$error)

#> [1] "not yet (call 3)"

If you would rather see the original error, set reraise = TRUE:

loud <- retry(stop = stop_after_attempt(2), reraise = TRUE) %@% make_flaky(Inf)
tryCatch(loud(), retry_error = function(e) "retry_error", error = conditionMessage)

#> [1] "not yet (call 2)"

Or return a fallback instead of throwing. retry_error_callback receives the final state, a list with attempt, wait, elapsed, error and value:

safe <- retry(
  stop = stop_after_attempt(2),
  retry_error_callback = function(state) "fallback"
) %@% make_flaky(Inf)
safe()

#> [1] "fallback"
Stop condition Gives up when
stop_never() never (the default)
stop_after_attempt(n) n attempts have been made
stop_after_delay(s) s seconds have passed since the first attempt
stop_before_delay(s) the next wait would pass s seconds in total
stop_any(a, b) either a or b fires
stop_all(a, b) both fire

Combine them to say “try up to 10 times, but never for more than 30 seconds”:

stop_any(stop_after_attempt(10), stop_after_delay(30))

#> <garnish::stop_any>
#>  @ stops:List of 2
#>  .. $ : <garnish::stop_after_attempt>
#>  ..  ..@ max_attempt_number: num 10
#>  .. $ : <garnish::stop_after_delay>
#>  ..  ..@ max_delay: num 30

Deciding how long to wait

By default retry() does not wait. For anything talking to a server you want a pause, and usually one that grows. wait_exp() doubles the wait each time:

patient <- retry(
  stop = stop_after_attempt(4),
  wait = wait_exp(multiplier = 0.01)
) %@% make_flaky(3)
system.time(res <- patient())[["elapsed"]] > 0.05

#> [1] TRUE

res

#> [1] "worked on call 4"

Waiting the same length of time means many clients that failed together retry together. Adding randomness spreads them out, and + combines strategies:

wait_exp(multiplier = 0.1) + wait_rand(0, 0.1)

#> <garnish::wait_combine>
#>  @ waits:List of 2
#>  .. $ : <garnish::wait_exp>
#>  ..  ..@ multiplier: num 0.1
#>  ..  ..@ min       : num 0
#>  ..  ..@ max       : num 4.61e+18
#>  ..  ..@ exp_base  : num 2
#>  .. $ : <garnish::wait_rand>
#>  ..  ..@ min: num 0
#>  ..  ..@ max: num 0.1
Wait strategy Waits
wait_none() no time (the default)
wait_fixed(s) s seconds
wait_rand(min, max) a random time between min and max
wait_exp() multiplier * exp_base^(attempt - 1), between min and max
wait_exp_jitter() exponential, with "full", "equal" or "decorrelated" jitter
wait_chain(a, b, ...) the n-th strategy for the n-th attempt
wait_combine(a, b) or a + b the sum of the strategies

Deciding what to retry

By default every error is retried. A bad password will not fix itself on the fifth attempt, so retry only what is temporary.

retry_if_error() matches condition classes:

calls <- 0
fetch <- retry(
  stop = stop_after_attempt(3),
  retry = retry_if_error("http_error")
) %@% function() {
  calls <<- calls + 1
  rlang::abort("bad request", class = "client_error")
}
try(fetch(), silent = TRUE)
calls

#> [1] 1

It made one attempt, because client_error is not an http_error. retry_if_error_message() matches the text instead, either exactly or by regular expression:

retry_if_error_message(match = "timed? ?out")

#> <garnish::retry_if_error_message>
#>  @ message: NULL
#>  @ match  : chr "timed? ?out"

A function can also fail without erroring, by returning something unusable. retry_if_result() retries when a predicate on the value is TRUE:

polls <- 0
poll <- retry(
  stop = stop_after_attempt(5),
  retry = retry_if_result(is.null)
) %@% function() {
  polls <<- polls + 1
  if (polls < 3) NULL else "ready"
}
poll()

#> [1] "ready"

retry_never() turns retrying off, which is useful for switching it off from a setting without removing the decorator.

Seeing what is happening

Silent retries are hard to debug. The hooks run around each sleep:

logged <- retry(
  stop = stop_after_attempt(4),
  wait = wait_fixed(0.01),
  before_sleep = before_sleep_log(function(msg) cat(msg, "\n")),
  after_sleep = after_sleep_log(function(msg) cat(msg, "\n"))
) %@% make_flaky(2)
logged()

#> Attempt 1 failed: not yet (call 1). Retrying in 0.01 seconds. 
#> Slept 0.01 seconds after attempt 1. Retrying. 
#> Attempt 2 failed: not yet (call 2). Retrying in 0.01 seconds. 
#> Slept 0.01 seconds after attempt 2. Retrying. 
#> [1] "worked on call 3"

The defaults are before_sleep_none() and after_sleep_none(). The loggers take any function that accepts one string, so you can send the message to whatever logging package you use.

Bounding hung calls

stop_after_delay() only checks the clock between attempts. If one attempt hangs, it never gets the chance. timeout() bounds a single call: when the time is up it signals a garnish_timeout_error, which is an ordinary error, so retry() treats it like any other failure.

Stack it inside retry() so every attempt is capped and a timeout is retried:

slow_work <- function() {
  # Many short steps, so R can stop between them (see the notes below)
  for (i in 1:100) Sys.sleep(0.02)
  "finished"
}

attempts <- 0
fetch := retry(stop = stop_after_attempt(3), wait = wait_fixed(0.01)) %@%
  timeout(0.2) %@%
  function() {
    attempts <<- attempts + 1
    if (attempts < 3) slow_work() else "fast enough"
  }
fetch()

#> [1] "fast enough"

attempts

#> [1] 3

The first two attempts were cut off after 0.2 seconds and the third finished in time. The error records the limit and the function name:

slow_call := timeout(0.2) %@% slow_work
err <- tryCatch(slow_call(), error = identity)
class(err)

#> [1] "garnish_timeout_error" "rlang_error"           "error"                
#> [4] "condition"            

conditionMessage(err)

#> [1] "`slow_call` timed out: it did not finish within 0.2 seconds."

Put timeout() inside retry(), as above. Outside it, retry() catches the time limit error and the limit is already spent, so later attempts would run without one.

Passing the time on

Some libraries have their own timeouts, which can clean up properly. timeout_remaining() returns the seconds left before the current deadline (Inf outside any timeout()), so you can hand them the budget:

budgeted <- timeout(10) %@% function() {
  # e.g. httr2::req_timeout(req, timeout_remaining())
  round(timeout_remaining())
}
budgeted()

#> [1] 10

Timeouts can nest. The soonest deadline always wins, so an inner timeout(60) inside an outer timeout(5) is cut off after 5 seconds.

What it cannot stop

The default method, timeout_r(), uses base R’s setTimeLimit(), which R only checks at interrupt checkpoints between R expressions. It cannot interrupt:

For these, pass timeout_remaining() to a native timeout, or use a method that runs the work in another process and kills it.

Writing a timeout method

A method is an S7 class that inherits from timeout_base, with a method for the timeout_fn(class, call, seconds) generic. call() runs the decorated function and seconds is the time left. It must return the value of call() unchanged, or signal a garnish_timeout_error. timeout() handles built-in and your own methods the same way, adding the function name and limit to the message. See ?extending_timeout for the details.

This one uses the callr package to run the call in a separate R process, which is killed when the time is up, even if it is stuck in compiled code. It also sets GARNISH_TIMEOUT_DEADLINE in the child, so timeout_remaining() works there too:

library(S7)

timeout_callr <- new_class("timeout_callr", parent = timeout_base)

method(timeout_fn, timeout_callr) <- function(class, call, seconds, ...) {
  deadline <- format(as.numeric(Sys.time()) + seconds, digits = 17)
  env <- c(callr::rcmd_safe_env(), GARNISH_TIMEOUT_DEADLINE = deadline)
  tryCatch(
    callr::r(function(call) call(), args = list(call), timeout = seconds, env = env),
    callr_timeout_error = function(e) {
      rlang::abort("Timed out.", class = "garnish_timeout_error")
    }
  )
}

hang := timeout(0.5, method = timeout_callr()) %@% function() {
  Sys.sleep(10)
  "never reached"
}
tryCatch(hang(), error = function(e) conditionMessage(e))

#> [1] "`hang` timed out: it did not finish within 0.5 seconds."

The function runs in a fresh process, so it cannot change variables in your session, and its arguments and result must be serialisable. callr is only suggested by garnish, so install it first if you want to use this.

Retrying without blocking

retry() sleeps between attempts, so nothing else can run in your R session while it waits. When the function already works with promises, as in Shiny, plumber or httr2::req_perform_promise(), use async_retry(). It takes the same arguments and the same stop, wait and retry strategies, including ones you wrote yourself, but schedules the next attempt with later instead of sleeping, and returns a promise.

flaky <- async_retry(
  stop = stop_after_attempt(5),
  wait = wait_fixed(0.05)
) %@% make_flaky(2)

p <- flaky()
promises::then(p, function(value) cat("resolved:", value, "\n"))

# Shiny and plumber run the event loop for you. In a script, run it yourself.
while (!later::loop_empty()) later::run_now(0.1)

#> resolved: worked on call 3

The function can return a plain value, as above, or a promise. A rejected promise counts as a failure, so a mirai task or an httr2 async request is retried the same way. When stop fires, the promise rejects with the same retry_error that retry() would signal.

async_retry() needs the promises and later packages. Put it outermost: a decorator inside it that calls the function synchronously, such as timeout(), cannot wait for or bound a promise.

Bounding a slow promise

async_timeout() is the promise version of timeout(). If the promise has not settled after the limit, the promise it returns rejects with the same garnish_timeout_error, which async_retry() retries:

# A stand-in for a request: resolves with `value` after `delay` seconds
request <- function(value, delay) {
  promises::promise(function(resolve, reject) {
    later::later(function() resolve(value), delay)
  })
}

calls <- 0
fetch := async_retry(stop = stop_after_attempt(3)) %@%
  async_timeout(0.1) %@%
  function() {
    calls <<- calls + 1
    if (calls < 3) request("too slow", 1) else request("in time", 0.01)
  }

promises::then(fetch(), function(value) cat("resolved:", value, "\n"))
while (!later::loop_empty()) later::run_now(0.1)

#> resolved: in time 

calls

#> [1] 3

It tells the caller to stop waiting but does not stop the work, so the first two requests above are still running when the third finishes. A method that also cancels the work, for example a mirai task, can be written with async_timeout_base and async_timeout_fn(), the same way as for timeout(). It cannot bound a function that blocks the session before it returns a promise. Use timeout() for that.

Writing your own strategies

Everything above is built from five kinds of strategy, and you can add your own to any of them. They are S7 classes, so if you have not used S7, the idea is small: make a class that inherits from a base class, then say what it does by registering a method for a generic. The built-in strategies work the same way, and yours combine with them.

retry() argument Inherit from Method of Return
stop stop_base stop_fn(class, state) TRUE to give up
wait wait_base wait_fn(class, attempt, prev_wait) seconds to sleep
retry retry_base retry_fn(class, outcome) TRUE to try again
before_sleep before_sleep_base before_sleep_fn(class, state) nothing
after_sleep after_sleep_base after_sleep_fn(class, state) nothing

A wait that grows by a fixed step each attempt:

library(S7)

wait_linear <- new_class(
  "wait_linear",
  parent = wait_base,
  properties = list(step = class_numeric),
  constructor = function(step = 1) new_object(wait_base(), step = step)
)

method(wait_fn, wait_linear) <- function(class, attempt, prev_wait = NULL, ...) {
  class@step * attempt
}

wait_fn(wait_linear(2), attempt = 3)

#> [1] 6

It composes like the built-in waits, with + and in wait_chain():

wait_fn(wait_linear(1) + wait_fixed(2), attempt = 3)

#> [1] 5

A stop condition gets the state list described in ?extending_retry, which has attempt, wait, elapsed, error and value. This one gives up once an attempt fails with a message matching a pattern:

stop_on_message <- new_class(
  "stop_on_message",
  parent = stop_base,
  properties = list(pattern = class_character),
  constructor = function(pattern) new_object(stop_base(), pattern = pattern)
)

method(stop_fn, stop_on_message) <- function(class, state, ...) {
  !is.null(state$error) && grepl(class@pattern, conditionMessage(state$error))
}

attempts <- 0
fatal <- retry(
  stop = stop_any(stop_after_attempt(5), stop_on_message("permission denied"))
) %@% function() {
  attempts <<- attempts + 1
  stop("permission denied")
}
try(fatal(), silent = TRUE)
attempts

#> [1] 1

It stopped after one attempt, though stop_after_attempt(5) would have allowed five, because stop_any() fires when either condition does.

A retry condition gets outcome, a list with value and error, one of which is NULL:

retry_if_negative <- new_class("retry_if_negative", parent = retry_base)

method(retry_fn, retry_if_negative) <- function(class, outcome, ...) {
  is.null(outcome$error) && outcome$value < 0
}

vals <- c(-1, -1, 5)
i <- 0
next_value <- retry(retry = retry_if_negative()) %@% function() {
  i <<- i + 1
  vals[[i]]
}
next_value()

#> [1] 5

i

#> [1] 3

If you write strategies in a package, call S7::methods_register() in its .onLoad() so the methods are registered when it loads.

Putting it together

fetch := retry(
  stop = stop_any(stop_after_attempt(5), stop_after_delay(30)),
  wait = wait_exp_jitter(base = 0.5, jitter = "full"),
  retry = retry_if_error_message(match = "timeout|503"),
  before_sleep = before_sleep_log(),
  reraise = TRUE
) %@% function(url) {
  # ...
}

Notes