DEVOURED
The Benchmarking Crisis
The ClickBench benchmark is losing its utility as developers increasingly optimize 'vibe-coded' database engines to game the benchmark's specific queries.
What: Database expert Curt Monash warns that newer database engines are hardcoding optimizations for ClickBench queries—such as hashing strings or skipping data based on known query patterns—rather than building generally robust systems.
Why it matters: This reflects a growing trend in the 'AI-generated' database ecosystem where speed in benchmarks is prioritized over standard reliability, leading to a disconnect between demo results and production performance.
Decoder
- Vibe-coded: A dismissive term for code written using AI tools where the author relies on the tool's output appearing correct ('the vibe') rather than understanding the underlying architecture or ensuring logical robustness.
- ClickBench: A popular open-source benchmark for analytical database performance that has become a primary marketing target for new database startups.
Original article
AI code generation is helping database engines game ClickBench. One, pgrust, groups strings by hash value alone, caches predicates the benchmark repeats, and on one query skips most of the data. Any benchmark stops being useful once systems optimise for it, and vibe-coded engines have no customers to keep them honest.
Explore all 58 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi