DEVOURED
Last year we built megakernels. This year we deleted them: The road to Together Compiler
Together AI discovered that replacing complex, hand-written 'megakernels' with intelligent agent-driven optimization and standard primitives yields 28% higher performance.
What: Together AI built Together Compiler (TKCC) to replace hand-written 'megakernels' by using AI agents to optimize and fuse GPU kernels. The compiler uses CUDA Graphs and Programmatic Dependent Launch (PDL) to achieve 1,205 tokens per second on Llama 3.2 1B.
Why it matters: The industry is moving away from rigid, monolithic GPU kernels toward flexible, agent-orchestrated optimization pipelines that can adapt to specific model architectures and hardware limitations.
Deep dive
- Megakernels: Executing entire model forward passes in a single monolithic GPU kernel, which proved rigid and difficult to productionize.
- Compiler Limitations: Found that greedy fusion heuristics in Torch Inductor often fail, and isolated micro-benchmarking leads to suboptimal kernel selection.
- Key Primitives: Reached 98% of megakernel performance using just CUDA Graphs and PDL to reduce launch overhead and overlap compute/memory tasks.
- Agentic Optimization: Replaced rule-based compilers with an agent-centric approach that directly modifies forward-pass source code.
- Performance: TKCC achieves up to 1.7x higher throughput than baseline megakernels by adapting to specific hardware (Hopper, Blackwell) and architectural needs.
Decoder
- Kernel fusion: Combining multiple smaller GPU operations into a single kernel to reduce memory traffic and improve hardware utilization.
- SM (Streaming Multiprocessor): The fundamental compute unit of a GPU.
- CTA (Cooperative Thread Array): A grouping of threads that share memory and execute a kernel on the GPU.
- PTX: NVIDIA's low-level intermediate representation for GPU code.
- FlashAttention: An algorithm to compute attention in Transformers more efficiently by reducing memory read/write cycles.
Original article
Full article content is not available for inline reading.
Explore all 56 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi