Back to Devoured - April 27, 2026
DEVOURED
HDFS Lost. How Object Storage and Table Formats Won the Data Lake

HDFS Lost. How Object Storage and Table Formats Won the Data Lake

Data infrastructurecloud podostack.com
Object storage plus table formats replaced HDFS for data lakes because network speeds caught up to disk speeds, eliminating the need for data locality and enabling independent scaling of storage and compute.
What: The article explains how the data engineering industry quietly abandoned Hadoop HDFS between 2018-2022 in favor of object storage like S3 combined with table formats like Apache Iceberg and Delta Lake, fundamentally changing how data lakes are built.
Why it matters: This shift happened because the hardware assumptions underlying HDFS became obsolete—modern cloud networks (25-100 Gbps) are now as fast or faster than local disk I/O, eliminating the performance benefit of data locality while the new architecture offers massive operational and cost savings through decoupled storage and compute.
Takeaway: If you're running Hadoop on cloud VMs, consider migrating to S3 (or equivalent object storage) plus Iceberg or Delta Lake with ephemeral compute to reduce costs and operational complexity.
Deep dive
  • The fundamental principle of Hadoop—"move code to data" through data locality—made sense when networks were 1 Gbps and reading from disk was 8x faster than network transfer, but modern cloud instances get 25-100 Gbps network throughput which matches or exceeds local disk speeds
  • HDFS architecturally couples storage and compute in the same DataNodes, forcing you to scale both together even when you only need one, leading to constant over-provisioning and paying 24/7 for idle compute attached to storage
  • Object storage breaks this coupling completely—data lives in S3, compute runs on ephemeral clusters that exist for minutes and disappear, scaling independently and paying only for seconds of actual usage
  • S3's initial "rename problem" (no atomic directory renames, only copy-and-delete of individual objects) seemed like a dealbreaker for Spark committers that relied on atomic directory operations
  • Table formats like Iceberg and Delta Lake solved this by maintaining explicit manifests of files rather than treating directories as implicit state, making commits atomic pointer swaps to new manifests
  • This wasn't just a workaround but an upgrade—table formats added ACID transactions, schema evolution, time travel, hidden partitioning, and efficient deletes that HDFS-based data lakes never had
  • The winning stack is S3 for storage, a table format for ACID and metadata, and ephemeral compute (Spark, Trino, DuckDB, Snowflake) reading directly from S3, with each layer independently swappable
  • Operational benefits include no NameNode to manage, no block balancer tuning, no specialized Hadoop administrators needed, with the cloud provider handling durability and replication
  • HDFS still makes sense in one scenario—on-premises hardware you already own, where you can't turn servers off anyway and HDFS maximizes utilization of both disks and CPUs
  • Running HDFS on cloud instances is an anti-pattern that pays for compute you can't turn off while fighting the ecosystem that assumes S3
Decoder
  • HDFS: Hadoop Distributed File System, a distributed filesystem that stores data across multiple machines with built-in replication
  • Data locality: The principle of running computation on the same physical machines where the data is stored to avoid network transfer
  • Table formats: Metadata layers like Apache Iceberg and Delta Lake that track which files belong to a table and enable ACID transactions on object storage
  • Object storage: Simple key-value storage systems like S3 that store immutable files without filesystem features like directories or atomic renames
  • ACID transactions: Atomicity, Consistency, Isolation, Durability guarantees that ensure database operations complete reliably
  • NameNode: The master server in HDFS that manages the filesystem namespace and metadata
  • DataNodes: Worker nodes in HDFS that both store data blocks and execute computation
  • Ephemeral compute: Short-lived compute clusters that spin up for a specific job and terminate when complete
  • Parquet: A columnar file format commonly used for analytics workloads
Original article

Data systems evolved to decouple storage and compute, making it cheaper and easier to scale.

Explore all 66 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi