DEVOURED
The Evolution of Cassandra Data Movement at Netflix
Netflix revamped its Cassandra data movement engine, now processing 3 PB/day by reading backups directly from S3 and converting them to Spark DataFrames, enabling optimized connectors for various data abstractions.
What: Netflix moved from an older Cassandra-to-Iceberg movement engine to a new layered platform. This platform reads Cassandra backups directly from S3, converts them into Spark DataFrames, and allows different data abstractions to build optimized connectors. The system processes about 3 petabytes of data daily and was migrated with zero downtime using shadow validation and a Maestro Decider for fallback.
Why it matters: This evolution demonstrates how large-scale data platforms like Netflix iterate on foundational data infrastructure to improve efficiency, flexibility, and maintainability for critical data movement tasks, moving towards more direct data access and standardized processing frameworks like Spark.
Takeaway: If you need to move large volumes of data from NoSQL databases to analytical stores, consider processing backups directly from object storage rather than live database instances to reduce load and improve efficiency, leveraging frameworks like Spark for transformation.
Decoder
- Apache Cassandra: A free and open-source distributed wide-column NoSQL database management system designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure.
- Apache Iceberg: An open table format for huge analytic datasets, designed to improve on existing table formats by supporting schemas, hidden partitioning, and schema evolution.
- Spark DataFrames: A distributed collection of data organized into named columns, conceptually equivalent to a table in a relational database or a data frame in R/Python, offered by Apache Spark for structured data processing.
- Maestro Decider: Likely a custom Netflix service or component used for decision-making and fallback logic within their operational orchestration system, Maestro.
Original article
Netflix replaced its Cassandra-to-Iceberg movement engine with a layered platform that reads backups directly from S3, converts them to Spark DataFrames, and lets each data abstraction build its own optimized connector. The engine moves about 3 PB/day, migration uses shadow validation, enhanced observability, and a Maestro Decider fallback to the prior solution, enabling a transparent cutover with zero downstream code changes.
Explore all 58 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi