DEVOURED
Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned
Netflix built a multi-stage streaming pipeline to transform raw network hop data into clean service-to-service topology maps at scale.
What: Netflix uses a three-stage Kafka/SSE pipeline to process millions of network flows per second, utilizing consistent hashing to shuffle data by 'intermediaries' (load balancers) to build direct application dependency edges.
Why it matters: This highlights the 'data concentration' problem in distributed systems, where naively partitioning data by destination creates hot nodes that crash under heavy traffic.
Takeaway: When building streaming pipelines, split compute-heavy resolution stages from I/O-heavy persistence stages to prevent hot-node bottlenecks.
Deep dive
- Stage 1 ingests and windows flow logs; Stage 2 shuffles by intermediary to resolve hops; Stage 3 persists and enriches.
- Switched from gRPC to SSE (Server-Sent Events) to reduce serialization overhead.
- Replaced immutable data structures on the hotpath with mutable ones to slash GC pause times.
- Solves hot-node issues by spreading data across multiple stages of redistribution.
- Uses consistent hashing for dynamic scaling without central coordination.
- Implements time-travel queries via mutation-history tracking in the graph database.
Decoder
- eBPF: A technology for running sandboxed programs in the Linux kernel to monitor network and system performance.
- SSE (Server-Sent Events): A standard allowing servers to push real-time updates to web clients over a single HTTP connection.
- Hot Node: A distributed system instance that receives disproportionately more traffic or workload than others, causing performance degradation.
Original article
Full article content is not available for inline reading.
Explore all 53 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi