DEVOURED
Modern Data Warehousing at Scale: Arcesium's Migration to DuckDB and Iceberg
Arcesium migrated 170TB of financial data to an Apache Iceberg and DuckDB stack, cutting costs by 40% and ingestion runtimes by 80%.
What: Arcesium engineers built 'Swiftlake', an execution layer integrating DuckDB and Iceberg on S3, leveraging Kubernetes autoscaling via KEDA to handle a 15-trillion-record Profit & Loss warehouse.
Why it matters: This illustrates a broader shift toward 'modern data stack' architectures that prioritize decoupled, open-format storage like Iceberg over proprietary RDBMS engines to optimize for performance and egress costs.
Decoder
- Apache Iceberg: An open table format for huge analytic datasets that provides SQL-like reliability and performance on object storage.
- Egress: The transfer of data out of a cloud network, often incurring significant costs.
- KEDA (Kubernetes Event-driven Autoscaling): A tool that enables Kubernetes clusters to scale pods based on external event sources like SQS queues.
Original article
Arcesium migrated a 170TB, 15-trillion-record P&L warehouse from an RDBMS to Apache Iceberg on S3 with DuckDB. The new stack cut ingestion runtime by 80% for larger portfolios, reduced egress timeouts, and lowered infrastructure spend by about 40%. The key enablers were a homegrown, open-sourced DuckDB-Iceberg execution layer called Swiftlake, KEDA for Kubernetes autoscaling, and Iceberg time-travel recovery.
Explore all 70 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi