DEVOURED
First Steps Toward Automated AI Research
Recursive's autonomous research system outperformed human-led open-source communities by discovering novel architectural and kernel optimizations for language model training.
What: The Recursive team built an automated research agent that autonomously proposes, implements, and validates training experiments across three benchmarks: NanoChat, NanoGPT Speedrun, and SOL-ExecBench. By iteratively testing architectural changes like hashed n-gram embeddings and custom FP8 Triton kernels, the system improved training efficiency on H100 and B200 GPUs, often beating established community records.
Why it matters: This demonstrates a shift where AI is used to optimize its own foundational training stack, moving beyond simple hyperparameter tuning to structural code generation and kernel-level performance engineering.
Deep dive
- The system uses an automated loop: propose, implement, run, validate, and repeat.
- NanoChat: Achieved 0.9109 BPB, improving upon the community-led autoresearch@home result by 0.0263 BPB.
- NanoGPT: Reduced training time from 79.7s to 77.5s using FP8 attention projections and cautious Adam updates.
- Kernel Engineering: Achieved a 0.754 SOL-ExecBench score, an 18% improvement over the previous baseline.
- Key innovations included: Hashed bigram/trigram embedding tables mixed into attention paths, causal token shifting, and fused Triton kernels that avoid redundant memory trips.
- The system handles reward hacking by using a dual-loop design where an automated evaluator constantly hardens against loopholes.
- Discovered solutions often combined disparate techniques that human researchers had implemented in isolation.
Decoder
- BPB (Bits Per Byte): A common metric for evaluating language models where lower values indicate better compression and higher prediction accuracy.
- SOL (Speed-of-Light): A relative performance metric comparing a custom kernel implementation against the estimated theoretical maximum hardware performance (1.0).
- Triton: A language and compiler developed by OpenAI that allows developers to write efficient GPU kernels in Python without needing low-level CUDA knowledge.
- Reward Hacking: When an AI agent discovers a technical loophole to achieve a high benchmark score that doesn't actually reflect the intended performance improvement.
Original article
Full article content is not available for inline reading.
Explore all 50 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi