DEVOURED
Unlocking asynchronicity in continuous batching
Hugging Face demonstrates how asynchronous batching using CUDA streams and events can boost LLM inference GPU utilization by 22%, dramatically speeding up generation.
What: Hugging Face detailed a method to achieve asynchronous batching, published May 14, 2026, that allows the CPU to prepare batch N+1 while the GPU computes batch N, eliminating idle time. This technique, which leverages CUDA streams and events, improved GPU utilization from 76% to 99.4% and reduced total generation time by 22% (from 300.6s to 234.5s) for an 8B model generating 8K tokens with a batch size of 32.
Why it matters: This deep dive into GPU-CPU parallelization reveals a significant, "free" performance gain for LLM inference without requiring model or kernel changes, highlighting that hardware coordination is a crucial, often overlooked, aspect of maximizing AI accelerator efficiency.
Takeaway: If you are optimizing LLM inference, consider implementing asynchronous continuous batching using CUDA streams and events, as detailed in the `transformers` library, to significantly improve GPU utilization and throughput.
Deep dive
- Synchronous batching in LLM inference leads to CPU and GPU idling in turns, wasting up to a quarter of total runtime.
- Asynchronous batching aims to parallelize CPU batch preparation for batch N+1 with GPU computation for batch N.
- This method, implemented in Hugging Face's
transformerslibrary, achieved a 22% speedup (from 300.6s to 234.5s) and boosted GPU utilization from 76% to 99.4%. - The core technique uses CUDA streams to allow concurrent GPU operations and CUDA events to enforce dependencies between operations across streams.
- Non-default CUDA streams are crucial as the default stream is synchronizing, blocking CPU and other streams.
- Three dedicated streams are used: Host-to-Device (H2D) for input transfer, Compute for model forward pass, and Device-to-Host (D2H) for output transfer.
- CUDA events (e.g.,
h2d_done,compute_done) are recorded in one stream and waited upon by another to ensure correct execution order. - To prevent race conditions, two sets of input/output tensors are used and alternated between (slot A and slot B).
- CUDA graphs, usually specific to memory addresses, can be used with a memory pool to avoid doubling VRAM usage for dual slots.
- A "carry-over" mechanism replaces placeholder tokens in batch N+1's inputs with actual tokens from batch N's outputs after batch N completes.
- The final result is a pipeline where the CPU quickly dispatches all GPU work, then prepares the next batch in parallel, only blocking at the very end to process outputs.
Decoder
- Continuous batching: An optimization technique for LLM inference where multiple ongoing requests are batched together dynamically, ensuring the GPU is always busy and reducing padding.
- CUDA streams: Ordered queues of GPU operations (kernel launches, memory copies) that execute sequentially within the stream but can run concurrently with operations in other streams.
- CUDA events: Markers that can be recorded into a CUDA stream; when the GPU reaches the marker, the event is set, and other streams can be instructed to wait for that event before proceeding.
- KV cache: "Key-Value cache," a memory optimization for LLMs where previously computed keys and values for attention layers are stored to avoid redundant computation during token generation.
- FlashAttention: An optimized attention algorithm that speeds up and reduces memory usage for attention computations in transformers, especially for long sequences.
- CUDA graphs: A feature that allows a sequence of CUDA operations to be recorded once and then replayed efficiently, reducing CPU overhead for repeated patterns.
Original article
Full article content is not available for inline reading.
Explore all 57 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi