DEVOURED
Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned
Netflix manages massive scale by processing service topology through a three-stage reactive pipeline that redistributes flow logs to resolve network intermediaries.
What: Netflix engineers Parth Jain, Rakesh Sukumar, and others built a streaming topology system using eBPF, Kafka, and a three-stage pipeline to convert raw network hops into logical application-level dependencies. The system uses consistent hashing to redistribute traffic at each stage, preventing hot nodes during traffic spikes.
Why it matters: This architectural pattern demonstrates that scaling complex observability systems requires decoupling data resolution from enrichment to handle the power-law traffic patterns inherent in distributed microservices.
Takeaway: If your infrastructure pipeline is bottlenecked by 'hot nodes' during traffic spikes, introduce a redistribution stage (shuffling) between your initial ingestion and final aggregation.
Deep dive
- Use eBPF flow logs for network visibility, IPC metrics for application-level details, and tracing for actual request paths.
- Implement a three-stage pipeline: ingestion/aggregation (Stage 1), intermediary resolution/redistribution (Stage 2), and enrichment/persistence (Stage 3).
- Use Server-Sent Events (SSE) instead of gRPC for streaming internal pipeline data to reduce serialization overhead and improve backpressure handling.
- Leverage consistent hashing with dynamic instance membership to enable automatic scaling of processing nodes.
- Pragmatically use mutable data structures in hot paths to reduce GC pressure when handling millions of events per second.
- Use property-level mutation tracking to support 'time travel' queries, allowing historical reconstruction of service topologies.
Decoder
- eBPF: A kernel-level technology that allows developers to run sandboxed programs in the Linux kernel for networking and observability without changing kernel source code.
- Backpressure: A signaling mechanism in streaming systems that allows downstream consumers to tell upstream producers to slow down, preventing buffer overflows.
- Consistent Hashing: A hashing strategy that minimizes data remapping when scaling a cluster up or down.
Original article
Full article content is not available for inline reading.
Explore all 56 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi