Back to Devoured - May 14, 2026
DEVOURED
When 36,000 Tiny Files Break Your Spark Pipeline: A Deep Dive into S3 DNS Exhaustion and the Small File Problem

When 36,000 Tiny Files Break Your Spark Pipeline: A Deep Dive into S3 DNS Exhaustion and the Small File Problem

Data performanceawsspark Medium
36,000 tiny Parquet files on S3 can crash Spark pipelines with UnknownHostException by exhausting DNS and S3 API limits, revealing a critical small file problem.
What: Sanjeeb Meister details how a Spark pipeline failed to read 36,000 tiny Parquet files from S3, leading to `UnknownHostException` errors. The root causes were DNS exhaustion, overwhelming S3 LIST/GET requests, and excessive driver/task metadata overhead.
Why it matters: This highlights a common, insidious performance trap in large-scale data processing on cloud object storage, where seemingly simple data structures (many small files) can lead to complex distributed system failures that manifest misleadingly.
Takeaway: Consolidate small files using compaction or adopt table formats like Delta Lake or Iceberg if working with Spark on S3.
Deep dive
  • A Spark pipeline failed when trying to read 36,000 tiny Parquet files from S3, manifesting as UnknownHostException errors.
  • The error was misleading, as networking was fine; the issue stemmed from overwhelming DNS resolvers due to a high volume of S3 operations.
  • Each S3 object access involves a DNS lookup, and a large number of small files triggered too many lookups too quickly.
  • Beyond DNS, the pipeline also hit limits for S3 LIST and GET API calls, leading to throttling.
  • Spark's driver and task executors also struggled with the sheer volume of metadata for thousands of tiny files.
  • Temporary mitigations include tuning Spark's spark.sql.files.maxPartitionBytes to combine small files into fewer partitions and spark.hadoop.fs.s3a.maxConnections to limit concurrent S3 connections.
  • The fundamental solution is to reduce the number of small files through compaction, merging them into larger files.
  • Adopting modern table formats like Delta Lake, Apache Iceberg, or Apache Hudi provides built-in mechanisms for file compaction and metadata management, effectively solving the small file problem.
Decoder
  • Parquet: A columnar storage file format optimized for analytics, known for efficient data compression and encoding.
  • S3 (Amazon Simple Storage Service): Amazon's cloud object storage service, widely used for data lakes and large-scale data storage.
  • DNS (Domain Name System): A hierarchical and decentralized naming system for computers, services, or other resources connected to the Internet or a private network, translating domain names to IP addresses.
  • DNS exhaustion: A state where DNS resolvers are overwhelmed by too many requests, causing timeouts or failures in resolving domain names.
  • Delta Lake: An open-source storage layer that brings ACID transactions to Apache Spark and other big data workloads, enabling reliability for data lakes.
  • Apache Iceberg: An open table format for huge analytic datasets, providing SQL-like table capabilities (like schema evolution, hidden partitioning) on top of object storage.
  • Apache Hudi: An open-source data lake platform that enables streaming data ingestion and provides record-level updates and deletes on data stored in cloud storage.
Original article

Thousands of tiny Parquet files on S3 can break Spark reads with UnknownHostException, even when networking is fine, by overwhelming DNS, S3 LIST/GET calls, and driver/task metadata. Spark partition tuning can help stabilize reads, but the real fix is compaction and table formats like Delta Lake or Iceberg.

Explore all 40 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi