DuckDB's co-creator Hannes Mühleisen unveiled Quack, a new client-server protocol enabling transaction-oriented workloads and 36X faster bulk inserts than Postgres.
Deep dive
- DuckDB co-creator Hannes Mühleisen introduced the "Quack" client-server protocol at the AICouncil keynote.
- Quack allows DuckDB to handle transaction-oriented workloads, including fast inserts, parallel writes, and read replica syncing, moving it beyond its embedded analytics origins.
- Benchmarks showed Quack completing a 60 million row bulk insert in 5 seconds, compared to 20 seconds for Postgres.
- For small inserts with 8 clients, Quack achieved 5000 transactions per second, slightly faster than Postgres.
- Quack provides a core primitive for building distributed DuckDB systems with an optimized remote procedure call protocol.
- The protocol's architecture was influenced by MotherDuck, a leading DuckDB-powered cloud database.
- Hannes Mühleisen and Mark Raasveldt had previously published a 2017 paper advocating for client protocol redesign due to existing inefficiencies.
- The DuckDB team continues to innovate with features like ergonomic SQL (GROUP BY ALL), a simpler data lake (DuckLake), and a faster SQL parser (PEG).
Decoder
- OLAP (Online Analytical Processing): A category of software tools that provide an enterprise with the ability to gain insight into its data from different points of view, typically involving complex queries over large datasets for analysis.
- OLTP (Online Transaction Processing): A category of software tools designed for transaction-oriented applications, typically involving many concurrent small transactions (inserts, updates, deletes) in mission-critical systems.
- Quack Protocol: A new client-server communication protocol for DuckDB, designed to enable distributed, transaction-oriented workloads beyond its traditional embedded analytics use case.
- Remote Procedure Call (RPC): A protocol that allows a program on one computer to execute code on a remote computer without explicitly coding the remote interaction.
Original article
Postgres has had a good 30-year run – is DuckDB coming for its crown? Yesterday at his #AICouncil keynote, DuckDB's co-creator Hannes Mühleisen unveiled a shot across the bow: a new client-server protocol called Quack. This is a foundational shift, enabling DuckDB to move beyond its roots as an in-process, single-node, analytics database towards more general purpose use cases. With Quack, DuckDB supports transaction-oriented workloads: fast inserts, parallel writes, and syncing read replicas. DuckDB's Quack is 36X faster than Postgres for bulk inserts, and rivals Postgres performance for transaction throughput on smaller inserts. [1] More broadly, Quack provides a core primitive for building distributed systems with DuckDB, now with a native, highly optimized remote procedure call protocol to coordinate across clusters of database nodes. MotherDuck, who operate the leading DuckDB-powered cloud database, has undoubtably inspired and influenced Quack's architecture. Hannes has been contemplating how a better protocol would work for years: he and DuckDB co-creator Mark Raasveldt authored a paper in 2017 titled "A Case for Client Protocol Redesign" where they mused about the inefficiencies of the then-current, and still-used approaches. Brick by brick, the DuckDB team members are rewriting the foundations of databases and data infrastructure for the modern era: more ergonomic SQL (c.f. GROUP BY ALL), a simpler data lake (DuckLake), a faster SQL parser (PEG not YACC), and much, much more. As Hannes said on stage yesterday, "Maybe it's time to use a database built after 2000." Ultimately, the world of developers will be the final judge. [1] For a 60M row bulk insert, DuckDB's Quack finished in 5 seconds, vs Postgres' 20 seconds. For small inserts with 8 clients, DuckDB's Quack supported 5000 transactions per second, slightly faster than Postgres.
A detailed overview of the Quack protocol, including a link to Hannes' talk yesterday at AICouncil, can be found here: https://duckdb.org/quack/ (Kudos to Gábor Szárnyas for contemporaneously launching this site with the full video already available).
Yeah... I'm sure it's going to replace Postgres for Postgres' most common use case... OLTP... (eye roll) - Quack is great; it solves a major problem with DuckDB... but there always seems to be a need to create some odd competitive angle where there isn't much of one, either for clicks, ego, or something.
I don't think DuckDB is trying to replace postgresql at all. Its intended for more data lake type scenarios, right? Not read/write working data.
Sai Krishna Srirampur Kaushik Iska
36 times faster than Postgres COPY command with binaries? Or just plain vanilla Postgres? Kind of an apples-to-oranges comparison, no? https://github.com/duckdb/duckdb-quack/blob/v1.5-variegata/benchmarks/benchmark-batch.py
Dying to know what was the prompt for this image
DuckDB doesn't handle concurrent read/writes. With Quack if you can do inserts into one database and use it to connect another where you do the reads would this simulate concurrency or am I conflating things here?
Hot take: I'd rather use Postgres with pg_duckdb. I'm not in a rush to lose a whole ecosystem for Duck.
Curious, if PostgreSQL develops a similar communication layer to duckdb and gets improved loads and writes…does this still apply? Quacking for a friend
Sounds like a lot of copium to me.
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi