DEVOURED
NVIDIA DevTech co-developed TokenSpeed, an inference engine specialized for coding agents that beats their general-purpose TensorRT-LLM by 11% on real Claude Code and Cursor production traces.
What: TokenSpeed, built by NVIDIA DevTech, AMD Triton, Qwen Inference, Together AI, and others starting March 2026, delivers 9% lower latency and 11% higher throughput than TensorRT-LLM on NVIDIA Blackwell for coding agent workloads. Custom MLA kernels nearly halve decode latency and have been adopted by vLLM. Benchmarked on SWE-smith traces targeting 70+ tokens/second per user.
Why it matters: This signals a fork in LLM serving architecture. General-purpose engines optimized for chatbots don't fit coding agents, which routinely exceed 50K token contexts across dozens of turns. As data centers scale to tens of gigawatts, workload-specific engines that squeeze 10% more throughput per GPU translate to hundreds of millions in capacity savings.
Takeaway: If you operate LLM inference for coding agents on NVIDIA Blackwell, test vLLM's TokenSpeed MLA kernels for potential 11% throughput gains.
Deep dive
- TokenSpeed is a from-scratch LLM inference engine optimized for agentic coding workloads where contexts exceed 50K tokens and conversations span dozens of turns, unlike chatbot workloads most engines target
- Began development mid-March 2026 with production hardening planned over next month; many PRs still landing as of May 2026
- Benchmarked against SWE-smith traces (real production coding-agent traffic) targeting maximum TPM per GPU while maintaining 70+ TPS per user floor for responsive UX
- On Kimi K2.5 with Attention TP4 + MoE TP4 configuration, TokenSpeed beats TensorRT-LLM by 9% on minimum latency (batch size 1) and 11% on throughput around 100 TPS per user
- TokenSpeed MLA kernels outperform TensorRT-LLM MLA across all five typical prefill workloads for coding agents (prefill with long prefix KV cache)
- Custom decode kernel groups query sequence length and attention heads to fully utilize Tensor Cores when head count is small, nearly halving latency versus TensorRT-LLM on batch sizes 4, 8, 16 with long prefix KV cache
- Architecture uses local SPMD design with I/O placement annotations at module boundaries; lightweight static compiler auto-generates collective operations during model construction
- Scheduler decouples control plane (C++ finite-state machine for compile-time safety of KV cache and resources) from execution plane (Python for fast iteration and lower cognitive load)
- Kernel layer is modular and treated as first-class subsystem with portable public API, centralized registry, plugin mechanism for heterogeneous accelerators (NVIDIA, AMD), and unified infrastructure
- TokenSpeed MLA already adopted by vLLM; developed in collaboration with NVIDIA DevTech, AMD Triton, Qwen Inference, Together AI, Mooncake, LongCat, FluentLLM, EvalScope, NVIDIA Dynamo
- Blog focuses on single (non-disaggregated) deployment; PD disaggregation support undergoing cleanup for dedicated follow-up
Decoder
- MLA (Multi-head Latent Attention): Attention mechanism variant used in models like DeepSeek V3 and Kimi K2.5 that compresses key-value cache using latent representations to reduce memory bandwidth during inference
- SPMD (Single Program, Multiple Data): Parallel programming model where identical code runs across multiple processors operating on different data partitions
- TPM/TPS (Tokens Per Minute/Second): TPM measures aggregate throughput (total tokens per minute across all users per GPU); TPS measures per-user latency (tokens per second for individual user experience)
- KV cache: Key-Value cache storing attention computation results from previous tokens to avoid recomputation during autoregressive generation
- Tensor Cores: Specialized matrix multiplication hardware in NVIDIA GPUs; TokenSpeed groups query sequence length with attention heads to better utilize these units when head count is small
- Prefill vs decode: Prefill processes initial prompt to populate KV cache; decode generates one token at a time reusing cached context, each requiring different kernel optimizations
- TP (Tensor Parallelism): Model parallelism technique splitting individual layer weights across multiple GPUs; TP4 means 4-way split
- PD disaggregation: Prefix-Decode disaggregation, architecture that separates prefill and decode phases into different GPU pools for better resource utilization
Original article
TokenSpeed, a high-performance LLM inference engine, optimizes agentic workloads with speed-of-light efficiency, leveraging a compiler-backed modeling mechanism and a high-performance scheduler. It delivers faster throughput than TensorRT-LLM for coding agents, with optimizations like TokenSpeed MLA to enhance Nvidia Blackwell's performance. Developed with NVIDIA DevTech and other collaborators, TokenSpeed significantly reduces latency and increases throughput in typical agentic workloads.
Explore all 73 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi