DEVOURED
How Tubular Labs reclaimed 50% of engineering capacity by rebuilding their 70TB pipeline on Apache Iceberg and Amazon S3 Tables
Tubular Labs cut infrastructure recovery times by over 95% by migrating a 70TB analytical pipeline to Amazon S3 Tables and Apache Iceberg.
What: Tubular Labs rebuilt their pipeline architecture using a Common Pipeline Runtime (CPR) built on Apache Spark and Apache Iceberg. This transition replaced fragile file-based storage with transactional S3 Tables, enabling idempotent retries and atomic writes that reduced infrastructure recovery time from up to 16 hours to 40 minutes.
Why it matters: Managing billion-row pipelines with custom file-based write logic introduces immense technical debt; leveraging managed table formats for ACID transactions offloads infrastructure complexity and frees engineering capacity.
Takeaway: If your Spark pipelines struggle with consistency during retries, investigate implementing an Iceberg-based Write-Audit-Publish (WAP) pattern to gain transactional atomicity.
Deep dive
- Tubular Labs’ pipeline processes 2 billion row updates daily across a 70TB dataset.
- The legacy system used non-idempotent custom file-merge logic that made partial failures difficult to recover from.
- CPR (Common Pipeline Runtime) separates business transformations from infrastructure-level concerns like Change Data Capture and atomic commit protocols.
- S3 Tables automatically handle compaction and snapshot maintenance, which were previously manual tasks.
- The Write-Audit-Publish (WAP) pattern allows the system to validate changes in isolation before promoting them to the main table.
- Engineering capacity increased by 50% as the team shifted focus from triaging data corruption incidents to building new product features.
Decoder
- Write-Audit-Publish (WAP): A pattern where data is written to an isolated branch or stage, validated for quality, and only then published to the production table.
- Idempotent: A property of an operation where applying it multiple times results in the same outcome as applying it once, crucial for reliable retries.
Original article
Full article content is not available for inline reading.
Explore all 55 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi