DEVOURED
Client-Side Load Balancing at a Million Requests Per Second
Zalando reclaimed control over its Product Read API by moving load balancing in-process, bypassing a shared edge router to handle one million requests per second.
What: Senior Principal Engineer Conor Gallagher led Zalando's transition to a client-side load balancer for internal fan-out traffic, using Kubernetes watches for service discovery, N-ring fade-in for smooth scaling, and occupancy-based load shedding. This change cut infrastructure costs by $1,000 daily and removed the shared Skipper ingress as a source of latency spikes.
Why it matters: Relying on shared cluster-wide ingress controllers for high-volume internal fan-out traffic creates 'noisy neighbor' issues and limits visibility. This underscores the necessity of moving critical routing logic into the service itself when performance requirements exceed standard infrastructure capabilities.
Deep dive
- Replaced shared Skipper ingress for internal fan-out with an in-process load balancer.
- Implemented hash-ring parity with the existing Skipper config to ensure identical routing.
- Switched from polling Kubernetes EndpointSlices to a persistent watch with 2s debounce to protect the API control plane.
- Developed 'N-ring fade-in' for autoscaling, using a power-law curve to warm caches without causing DynamoDB spikes.
- Replaced 'in-flight' request counting with 'occupancy' (seconds of work per second) based on Little's Law.
- Added latency-weighting to the load balancer to automatically route around slow pods and stuck nodes.
- Hardened retries by excluding previously tried destinations to prevent retry storms.
Decoder
- Fan-out: A pattern where a single request triggers multiple parallel downstream requests to gather data.
- EndpointSlice: A Kubernetes API resource that provides a more scalable way to track network endpoints compared to standard Service endpoints.
- Little's Law: The queuing theory formula L = λW, which links average inventory (concurrency) to arrival rate and waiting time.
Original article
Full article content is not available for inline reading.
Explore all 57 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi