DEVOURED
mondayDB 3 – Solving HTAP for a Trillion-Table System
monday.com replaced its MySQL, Cassandra, and Redis fleet with mondayDB 3, a custom HTAP system powered by DuckDB, achieving 5-20x performance improvements and 40-60% cost reductions for its trillion-table system.
What: monday.com transitioned its board reads from a MySQL + JSON architecture to mondayDB 3, a CQRS-based Lambda architecture. This new system stores immutable snapshots in object storage, uses an external WAL for real-time changes, and a soft-stateful serving layer that syncs and queries local DuckDB files on every read. It resulted in 5x faster board loads, 20x faster large boards, 50x faster aggregations, and a 40-60% reduction in infrastructure costs.
Why it matters: monday.com's experience with a highly flexible, multi-tenant system highlights the limitations of general-purpose OLTP databases for analytical workloads with dynamic schemas, even when augmented with JSON. It demonstrates how purpose-built systems leveraging columnar engines like DuckDB, combined with CQRS and external WALs, can offer significant performance and cost advantages by aligning the data architecture with actual access patterns. This architecture is now being extended for AI contextual layers.
Takeaway: If you are dealing with multi-tenant systems, dynamic schemas, and analytical workloads on operational data, consider a CQRS-based architecture with an embedded columnar database like DuckDB and an external WAL to achieve better performance and cost efficiency.
Deep dive
- mondayDB 3 is an HTAP system designed to handle over a trillion dynamic tables with highly flexible schemas.
- The previous architecture used MySQL with JSON blobs for schema flexibility, plus Cassandra and Redis.
- This led to slow board loads (over 2 seconds for large boards), poor aggregation performance, and inefficient I/O due to row-oriented storage and multi-tenant shared indexes.
- The new architecture is a CQRS-based Lambda architecture with durable snapshots in object storage (S3), an external write-ahead log (WAL) for real-time mutations, and a soft-stateful serving layer.
- Serving nodes are Go processes on Kubernetes with local NVMe SSDs, acting as smart read-through caches.
- Each serving node maintains an LRU cache of over 200,000 DuckDB files, one per board.
- Before each query, the serving node loads the board's DuckDB file, syncs pending WAL entries, and executes the query in-process using DuckDB's vectorized C++ engine.
- DuckDB's native WAL is disabled; monday.com uses its own distributed external WAL for durability and real-time changes.
- Dynamic schema evolution is handled on the fly by issuing ALTER TABLE ADD COLUMN in DuckDB when a new column is referenced.
- A custom routing layer, Ranja, uses Weighted Rendezvous Hashing with capacity-aware weights and hedged requests to ensure cache affinity and resilience.
- The migration of over 1 million organizations took 18 months, using feature flags, dual-read validation, and per-account rollout with instant rollback capability.
- The architecture is designed as a multi-entity platform, supporting new entity types via JSON-driven plugins.
- monday.com plans to evolve mondayDB 3 into an AI contextual layer for text search, semantic retrieval, and RAG, leveraging its per-board file isolation and real-time freshness.
Decoder
- HTAP (Hybrid Transactional/Analytical Processing): A database system designed to efficiently handle both transactional (OLTP) and analytical (OLAP) workloads within a single platform.
- CQRS (Command Query Responsibility Segregation): An architectural pattern that separates the model for updating information (the command side) from the model for reading information (the query side).
- Lambda Architecture: A data processing architecture designed to handle massive quantities of data by combining batch processing with stream processing for real-time data.
- DuckDB: An in-process SQL OLAP database management system designed for analytical queries, often described as SQLite for analytics.
- WAL (Write-Ahead Log): A log of changes to data before they are applied, used to ensure data durability and atomicity.
- Soft-stateful: Refers to a system where state is maintained for performance but can be rebuilt from a durable source of truth if lost, meaning no critical data relies solely on that local state.
- Ranja (Weighted Rendezvous Hashing): A distributed hashing algorithm used for consistent load balancing and routing, ensuring that a given key (e.g., tenant ID) consistently maps to the same node for cache affinity.
- RAG (Retrieval-Augmented Generation): An AI technique that combines a language model with a retrieval system to fetch relevant information from an external knowledge base to improve the accuracy and relevance of generated responses.
Original article
Full article content is not available for inline reading.
Explore all 58 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi