DEVOURED
Announcing Apache Fluss 1.0: Real-Time Data Foundation for AI
Apache Fluss 1.0 graduates to a top-level project, adding a Rust-based REST Gateway and deeper integration with Flink, Spark, Hudi, and Paimon.
What: Apache Fluss 1.0 introduces a stateless Rust HTTP Gateway for real-time metadata and batch writes, adds native Python and C++ client support, and improves data lifecycle management with row-level TTLs and predicate pushdown. The release also enables Paimon and Hudi integration for lakehouse architectures.
Why it matters: This release marks Fluss's evolution into a unified real-time data foundation that bridges streaming (Flink), batch (Spark), and agent-based AI workloads without requiring JVM overhead for every component.
Takeaway: If you need an HTTP entry point for real-time data, test the new Rust Gateway; see the Quickstart guide at the project website.
Deep dive
- New Gateway: Stateless Rust-based REST service for non-JVM metadata discovery, table management, and batch writes.
- Native Clients: Shared Rust core for Rust, Python, and C++ clients, enabling non-JVM context lookups.
- TTL Enhancements: Row-level and log-level TTLs allow for granular data expiration independent of partitions.
- Predicate Pushdown: TabletServer-level filtering for Arrow-format logs significantly reduces network I/O.
- Lakehouse Tiering: Adds Hudi integration and support for historical partition lookups in Paimon.
- Operational Improvements: Coordinator high-availability, multiple local disk support, and resource protection for disk/memory pressure.
- Flink/Spark: Flink SQL now supports RoaringBitmap aggregations, while Spark supports batch Union Read across hot and cold storage.
Decoder
- TabletServer: The component responsible for storing and serving data segments within the Fluss architecture.
- RoaringBitmap: A compressed bitmap index structure designed for high-performance set operations and cardinality estimation.
- Predicate Pushdown: An optimization technique where filters are applied as close to the data storage as possible to avoid loading irrelevant rows.
Original article
Full article content is not available for inline reading.
Explore all 53 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi