DEVOURED
How Uber Protects Against Retry Storms
Uber's new 'error ownership' mechanism in its service mesh prevented 9.5 million unnecessary retry requests during a major outage.
What: Uber engineers introduced an error-attribution system that identifies whether a service is the origin of an error or merely propagating it. By allowing only the 'originator' to handle retries, Uber reduced the retry-storm radius from 25 hops to 3, stopping cascading failures during service degradation.
Why it matters: This demonstrates a shift from static retry budgets to context-aware infrastructure that protects the stack by preventing upstream services from hammering degraded dependencies.
Deep dive
- Developed an 'Error Ownership' protocol where nodes claim or unclaim errors based on dependency analysis.
- Upstream nodes check for 'error claim' headers before initiating retries.
- If a service returns an error without a dependency failure, it owns the error and assumes responsibility for handling it.
- Coincidental errors (caller failing while downstream fails) are handled by maintaining a memory of failure patterns to avoid suppressing legitimate retries.
- Reduced total request volume during outages by eliminating redundant retries across the call graph.
Decoder
- Retry Storm: A phenomenon where failing services are overwhelmed by repeated retry requests from upstream callers, worsening an outage.
- Service Mesh: A dedicated infrastructure layer for handling service-to-service communication.
- Fan-out: A pattern where one service call triggers multiple downstream service calls.
Original article
Full article content is not available for inline reading.
Explore all 57 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi