8
Backend
Circuit breaker vs retry when a dependency degrades under load
Failure isolation ↔ Recovery attempts
Context
Keep checkout working when the fraud-scoring service slows down.
Checkout calls a fraud-scoring service. Under load its p99 goes from 200 ms to 8 seconds, and our checkout requests pile up behind it.
Option A: wrap the call in a circuit breaker: after a burst of timeouts, stop calling it for a while and use a fallback (approve small orders, queue big ones for review).
Option B: keep calling with retries and exponential backoff, so transient blips don't fail orders.
Which protects us better?
Constraints
- Dependency
- Fraud scoring, p99 jumps from 200 ms to 8 s under load
- Checkout budget
- 2 s end to end
- Fallback
- Allow orders under €100 without a score
Circuit breaker with fallbackvsRetry with backoff
Community verdict
Circuit breaker with fallback 67%Retry with backoff 33%
9 engineers · 5 opinions
With these constraints, what would you choose?
One choice per engineer. You can change it any time.
Circuit breaker with fallback
Retry with backoff
Recovers transient errors →
Trade-offs
Dimensions
Circuit breaker with fallbackRetry with backoffout of 5
- Protects the dependency
- 5Circuit breaker with fallback scores 5 of 52Retry with backoff scores 2 of 5
- Recovers transient errors
- 2Circuit breaker with fallback scores 2 of 55Retry with backoff scores 5 of 5
- Latency during failures
Community discussion
5 comments
Have you made this decision in production? Share your reasoning.
Sign in to commentMost of our failures were single dropped connections during deploys. A quick retry fixed 99% of them; a breaker would have tripped for no reason with a bad threshold.
Tune the breaker on error rate over a window with a minimum request count. Single blips should never open it.
Get the fallback signed off by the risk team. "Approve under €100" is a business decision, not an engineering one.
Retries make an overloaded dependency more overloaded: every slow call becomes two or three. That is how a slowdown becomes an outage.
Use both, in order: a tight timeout, at most one retry with jitter for transient errors, all inside a circuit breaker that opens when the error rate crosses a threshold. The breaker is what stops the pile-up.
And alert on the breaker opening. A silent fallback can approve risky orders for hours.