DEVOURED
Hitting a billion tokens per minute on one GPU by combining a query planner and an inference engine
The QUery-Aware Inference Layer (Quail) boosts inference throughput to over a billion tokens per minute on a single H100 by tightly integrating SQL query planning with LLM execution.
What: Developed by Modal and CMU's Full Stack Data Lab, Quail optimizes 'AI-SQL' workloads by treating inference as a SQL execution problem. It achieves 10x the speed of vLLM by eliminating decode phases, using structured query plans to optimize KV cache management, and fusing kernels.
Why it matters: It shows that inference performance for structured data tasks can be drastically improved by abandoning the 'chatbot' inference model in favor of query-aware execution.
Takeaway: If running high-volume AI-SQL workloads, try Quail on Modal using the `quail-engine` Python package to see immediate performance gains.
Deep dive
- Quail combines a SQL query planner with a specialized inference engine.
- The architecture removes the need for standard 'decode' phases by focusing on prefill-only classification tasks (e.g., AI.IF).
- It uses a KV-aware join-order search to optimize memory usage and cache reuse.
- Custom Triton kernels perform operator fusion, significantly reducing host overhead.
- The system is designed for 'AI-SQL' where prompts are generated programmatically from database rows, not natural language chats.
Decoder
- KV Cache: A memory buffer in LLM inference that stores Key and Value states to avoid recomputing previous tokens, crucial for long-sequence performance.
- Kernel Fusion: A technique in GPU programming that combines multiple operations (e.g., add + normalization) into a single GPU instruction to reduce memory traffic.
- TPM (Tokens Per Minute): A standard metric for measuring the throughput of an LLM inference system.
- Arithmetic Intensity: The ratio of compute operations to memory access operations, defining how well a workload utilizes GPU resources.
Original article
Full article content is not available for inline reading.
Explore all 67 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi