DEVOURED
How we extended Apache DataFusion to execute one query across many machines
Datadog open-sourced Distributed DataFusion, allowing developers to execute a single Apache DataFusion query across multiple machines to handle heavy analytical workloads.
What: The framework enables scaling DataFusion beyond a single node by introducing stages, tasks, and network shuffles while maintaining low-latency interactive performance. Datadog reports up to 10x speed improvements for heavy queries by distributing compute across an EC2 cluster.
Why it matters: By building a framework rather than a managed service, Datadog avoids locking users into specific infrastructure, allowing teams to implement their own data sources and optimization rules for distributed SQL execution.
Takeaway: If you are building an analytical query engine using Apache Arrow and DataFusion, try the Distributed DataFusion framework on GitHub to scale your query execution across multiple machines.
Deep dive
- Distributed DataFusion extends physical query plans across nodes rather than just local CPUs.
- Uses Apache Arrow Flight for low-latency network data transfer between workers.
- Implements stages to manage network boundaries and shuffles for group-by aggregations.
- Includes a 'NetworkCoalesceExec' node to gather results from distributed workers into a final single stream.
- Designed to avoid distribution overhead for lightweight queries by opting for local execution when stats suggest it is faster.
- The system is written in Rust and integrates with existing Substrait-based planning pipelines.
Decoder
- Apache DataFusion: An extensible, high-performance query engine written in Rust that uses Apache Arrow as its in-memory data format.
- Substrait: A standardized intermediate representation for relational algebra that allows different query engines to share the same logical query plans.
- Predicate pushdown: An optimization technique that moves filtering logic closer to the data source to reduce the amount of data processed.
Original article
Full article content is not available for inline reading.
Explore all 67 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi