DEVOURED
Portability Is a Myth: Why the Best AI Stacks Will Never Be Hardware-Agnostic
Patrick Toulme argues AI kernel portability is impossible: MoE matmul ships as 282 Pallas lines on TPU versus 4 million CUDA lines on Blackwell.
What: Patrick Toulme argues that TPU's Pallas, NVIDIA's CuTile/CUTLASS, AWS's NKI, AMD's FlyDSL, and Tenstorrent's tt-Metalium each expose hardware-specific primitives that no universal DSL can unify. Evidence: MaxText's MoE grouped matmul is 282 Pallas lines on TPU versus flashinfer's 4 million CUDA lines on Blackwell, with zero shared code.
Why it matters: Hardware-specific kernel optimization creates vendor lock-in deeper than APIs: switching from NVIDIA to Google TPUs or AWS Trainium would require rewriting core ML operations, not just changing cloud providers. This suggests the ML infrastructure market will remain fragmented, with little cross-vendor code reuse.
Deep dive
- Patrick Toulme claims AI kernel portability is structurally impossible because vendor kernel DSLs expose fundamentally different hardware primitives
- Competing DSLs: Google's Pallas (TPU), NVIDIA's CuTile/CUTLASS (CUDA), AWS NKI (Trainium), AMD FlyDSL, Tenstorrent tt-Metalium
- Evidence: MaxText MoE grouped matmul is 282 Pallas lines on TPU versus 4 million CUDA lines on Blackwell, zero shared code
- The algorithms themselves diverge across hardware, not just the syntax - each vendor's primitives demand different decomposition strategies
- This suggests high-level frameworks (PyTorch, JAX) can provide portability only by leaving performance on the table
- Teams needing peak performance must maintain hardware-specific kernel implementations, multiplying engineering cost and creating deep vendor lock-in
- Implication: the ML infrastructure ecosystem will remain fragmented, with NVIDIA-optimized stacks incompatible with TPU/Trainium equivalents
Decoder
- Pallas: Google's kernel DSL for TPU hardware
- CUTLASS: NVIDIA C++ template library for GPU matrix operations
- NKI: AWS Neuron Kernel Interface for Trainium/Inferentia chips
- MoE: Mixture of Experts, neural network architecture with specialized sub-models
- Blackwell: NVIDIA's 2024 GPU architecture, successor to Hopper
Original article
AI kernel portability is structurally impossible because TPU's Pallas, NVIDIA's CuTile and CUTLASS, AWS's NKI, AMD's FlyDSL, and Tenstorrent's tt-Metalium each expose hardware-specific concepts that no universal DSL can unify. The evidence: MaxText's MoE grouped matmul ships as 282 lines of Pallas on TPU while flashinfer's equivalent for Blackwell SM100 takes 4 million lines of generated CUDA, with zero shared code because the algorithms themselves diverge across hardware.
Explore all 67 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi