DEVOURED
Encoding or Compression: Why not both?
Encoding and compression should be treated as complementary tools: use encoding to accelerate query execution, and layer compression only when storage costs require it.
What: This analysis contrasts data-aware encodings (dictionary, frame-of-reference) which allow vectorized, in-place query operations, with general-purpose compression (zstd) which is opaque to the query engine.
Why it matters: Modern databases benefit from keeping data permanently encoded to stay queryable, applying heavy compression only as a secondary layer for cold storage to avoid the CPU penalty of frequent decompression.
Takeaway: Avoid decompressing data in the middle of your hot query paths; always keep a lightweight, encoded representation in memory for SIMD-accelerated filtering.
Deep dive
- Dictionary encoding allows string comparisons to be reduced to cheap integer equality checks.
- Frame-of-Reference (FOR) encoding enables range pruning by allowing the engine to check block min/max headers.
- General-purpose compression (zstd) beats encoding on raw bytes but prevents random access and predicate pushing.
- Layering zstd on top of an already-dense, fixed-width integer array yields higher compression ratios than compressing raw string data.
- Encoding application is significantly faster than compression/decompression, often by several orders of magnitude.
Decoder
- SIMD (Single Instruction, Multiple Data): A CPU instruction set that processes multiple data points in a single operation, critical for high-speed analytical query performance.
- Frame-of-Reference Encoding: A numeric compression technique that stores values as small deltas from a local reference minimum, reducing byte-width requirements.
Original article
Full article content is not available for inline reading.
Explore all 58 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi