DEVOURED
TPU Inference Externalization Full Steam Ahead
Google's TPUv7 Ironwood chips demonstrate up to 50% better performance-per-dollar than NVIDIA's B200/B300, marking Google's pivot to selling TPU capacity externally.
What: The InferenceX preview shows TPUv7 Ironwood outperforming NVIDIA's latest Blackwell silicon in aggregated inference workloads. Google is shifting from a JAX-heavy backend (TorchAX) to a native PyTorch backend (TorchTPU) to make integration into standard tools like vLLM and SGLang significantly easier.
Why it matters: By externalizing its TPU software stack and selling hardware, Google aims to break NVIDIA's datacenter hegemony using superior system-level integration and custom networking.
Takeaway: Watch for the open-sourcing of the native TorchTPU backend around October 2026, which will simplify TPU support in PyTorch-based inference engines.
Deep dive
- Ironwood utilizes a two-die architecture with independent logical devices per chip.
- TPU-specific optimizations (DP attention, SparseCore offloading, MoE kernel fusion) are required to hit peak performance.
- Native TorchTPU replaces the translation-heavy TorchAX, allowing direct PyTorch execution with XLA/StableHLO compilation.
- TPU 8i 'Boardfly' topology reduces network hops by over 50% vs. 3D torus architectures.
- Speculative decoding and prefill-decode disaggregation remain the primary roadmaps for further efficiency gains.
Decoder
- Systolic Array: A matrix multiplication architecture where data flows through a grid of cells, accumulating results without frequent memory access.
- StableHLO: A standardized high-level dialect for machine learning operations that XLA compilers use to generate optimized hardware code.
Original article
Full article content is not available for inline reading.
Explore all 46 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi