DEVOURED
Apache DataFusion Comet 1.1.0 Release
Comet 1.1.0 improves Spark performance by moving Iceberg writes to native Rust and provides better memory tracking to prevent container crashes.
What: Apache DataFusion Comet 1.1.0 introduces experimental native Iceberg writes via 'iceberg-rust', native shuffle integration with Celeborn, and refined off-heap memory management to reduce 'ExecutorLostFailure' errors.
Why it matters: By moving operations from the JVM to native Rust and splitting write operators, Comet reduces overhead, though it creates a new operational burden of sizing 'spark.executor.memoryOverhead' for non-JVM memory.
Takeaway: Set 'spark.comet.memory.logInterval=1s' to analyze native memory usage peaks, and size your container's memory overhead accordingly.
Deep dive
- Native Iceberg write support using 'iceberg-rust' (experimental).
- Native shuffle implementation over Apache Celeborn.
- Improved memory pool accounting to prevent task panics on partial grants.
- Logs native memory usage on executors every 10 seconds.
- Fixes for various wrong-result issues in decimal arithmetic and dictionary encoding.
Decoder
- Native shuffle: Transferring data between stages in a distributed compute job using memory-efficient binary formats (Arrow) outside the Java Virtual Machine heap.
- Off-heap memory: Memory allocated outside the JVM's managed garbage collection process, often used by high-performance data systems to store large data buffers.
Original article
Full article content is not available for inline reading.
Explore all 59 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi