DEVOURED
What Zero-Copy Actually Costs
The term 'zero-copy' is a dangerously overloaded marketing label that hides four distinct costs: egress, repeated scans, source load, and metadata churn.
What: Data architect Alex Merced breaks down 'zero-copy' into six patterns: federated query, format virtualization, sharing protocols, catalog federation, mirroring, and caching/materialization. Three of these patterns actually create copies, and the 'correct' one depends entirely on read frequency and geographic location.
Why it matters: This exposes the 'no-ETL' industry narrative as a trade-off exercise, clarifying that data movement is often the most economical choice when query volume is high or systems are separated by regional boundaries.
Takeaway: Run the arithmetic for your specific use case: calculate bytes scanned per day × cost of egress. If the result is high, replicate the data locally.
Deep dive
- Federated Query: Queries live data via remote connections; dangerous for high-volume analytical workloads.
- Format Virtualization: Translates metadata (e.g., OneLake serving Delta as Iceberg) without rewriting data files.
- Sharing Protocols: Uses signed references (e.g., Delta Sharing) to access provider files; egress costs land on the provider.
- Catalog Federation: Metadata moves (table schemas), but actual data reads stay local to the storage.
- Pushdown: The critical performance metric; check query plans to ensure filters are running on the remote side, not locally.
- Agent Amplification: Autonomous agents query in loops, turning predictable human dashboards into massive, unpredictable egress bills.
Decoder
- Egress: The cost charged by cloud providers for data leaving a network region or availability zone.
- Pushdown: The optimization where the primary processing engine delegates parts of a query (filters, aggregations) to the source system to reduce data transfer.
- Reflection: A specialized, materialized view (often used by Dremio) designed to accelerate specific query shapes.
Original article
Full article content is not available for inline reading.
Explore all 65 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi