DEVOURED
QuestDB 10.1: Fresher Data, Less I/O, and More Analysis in SQL
QuestDB 10.1 introduces merge-append partitions to slash disk I/O for out-of-order writes, significantly accelerating ingestion throughput.
What: The 10.1 update adds optional merge-append partitions, exact percentile functions (percentile_disc/cont), SQL subsampling, and internal metrics history in a 'sys.metrics' table.
Why it matters: This release prioritizes handling high-volume, out-of-order time-series data which traditionally caused heavy write amplification due to constant partition rewrites.
Takeaway: Enable merge-append partitions via the cairo.o3.partition.merge.append.enabled=true configuration setting.
Deep dive
- Merge-append partitions: Reduces write amplification by appending late data to partition slices instead of rewriting the entire partition.
- SUBSAMPLE: A SQL clause for thinning results while preserving line shape (useful for charts/archives) using methods like sdt, lttb, and m4.
- Exact Percentiles: New percentile_disc() and percentile_cont() functions that handle negative values and array inputs.
- Parallel GROUP BY: Aggregation over joins now uses all query workers instead of a single thread.
- Monitoring: QuestDB now persists internal metrics to a sys.metrics table, eliminating the need for separate dashboard servers.
Decoder
- Write Amplification: A phenomenon where the amount of data written to the storage media is a multiple of the data intended to be written, reducing disk life and performance.
- Markout: A quantitative finance technique comparing trade prices to subsequent mid-prices to evaluate the quality of a trade execution.
- Wal (Write-Ahead Log): A standard method for ensuring data integrity by logging changes before applying them to the main database structures.
Original article
Full article content is not available for inline reading.
Explore all 69 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi