DEVOURED
A tale of two Flink autoscalers
Netflix is abandoning its custom Flink autoscaler for the Apache Flink Autoscaler to gain operator-level state awareness and better resource utilization.
What: Netflix is migrating over 30,000 Apache Flink jobs from a legacy container-level autoscaler to the Apache Flink Autoscaler. The switch enables per-operator scaling and has delivered a 58% increase in compute efficiency for some teams compared to the legacy system.
Why it matters: This shift highlights the transition from generic container-based orchestration to workload-aware autoscaling that understands streaming job topology and state, which is critical for cost-efficient data pipeline management at scale.
Deep dive
- The legacy autoscaler scaled entire Flink clusters based on container metrics, causing coarse-grained, inefficient resource allocation.
- The Apache Flink Autoscaler operates inside the job, allowing for granular adjustments based on individual operator state.
- Telemetry gaps in the custom system were resolved by the new implementation.
- Annualized compute savings reached 58% for specific internal teams.
Decoder
- Apache Flink: A distributed processing engine for stateful computations over unbounded and bounded data streams.
- Autoscaling: The automatic adjustment of computational resources based on real-time load, aiming to balance performance with infrastructure costs.
Original article
Netflix is moving 30,000+ Flink jobs from a homegrown autoscaler toward the Apache Flink Autoscaler. The old system saved 25–45% by scaling whole clusters from container metrics, but missed operator-level state and suffered from telemetry gaps. The OSS autoscaler reasons from inside each job and produced 58% annualized compute savings for one team.
Explore all 66 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi