DEVOURED
Don't Write Kernels, Derive Them
Tang replaces hand-written operator kernels with an IR that derives and fuses kernels on-the-fly, allowing performance parity with expert-tuned engines.
What: Tang's tensor IR uses views and comprehensions to derive fused kernels at runtime. It matches the performance of hand-tuned engines like those for 80B MoE models on H200s by replacing operator-specific code with algebraic rewrites.
Why it matters: The shift from manual kernel writing to derived kernels moves the burden of optimization from humans to compilers, enabling better performance across diverse hardware without specialized library maintenance.
Deep dive
- IR constructs: Views (affine maps) and Comprehensions (reductions over a monoid)
- Derived kernels outperform hand-written ones in specific regimes by eliminating materialized intermediate buffers
- Fusion is treated as a single substitution rewrite rule; flash attention is derived, not hardcoded
- Backward pass is generated automatically via reverse-mode differentiation of comprehensions
- Supports multiple targets: CUDA, WGSL, Metal, C, OpenCL, and GLSL
- Validated on M4 Max, RTX 3090, H200, and Android-based humanoid robot SoC
Decoder
- Monoid: A mathematical structure used here to define how reduction operations (like sum or max) are combined.
- Affine Map: A transformation used in views to express memory access patterns like transposes or slices without explicit pointer arithmetic.
- FlashAttention: A memory-efficient attention mechanism that computes attention scores in blocks to minimize memory traffic.
Original article
Full article content is not available for inline reading.
Explore all 61 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi