DEVOURED
Training frontier knowledge work agents: A 397B RL training guide with SkyRL
Mercor and SkyRL achieved a 70% boost in agentic performance by post-training Qwen models on nearly 2,000 professional knowledge-work tasks using asynchronous reinforcement learning.
What: The recipe, released as 'ApexAgents-SkyRL-Recipe', details the use of asynchronous RL with vLLM inference and Megatron training to scale models up to 397B parameters for complex agentic workflows.
Why it matters: This demonstrates that for agentic workflows, the stability of the environment harness and exact token accounting (TITO) often matter more than the choice of loss algorithm.
Takeaway: Use the released training scripts and Harbor data format to post-train your own agents on proprietary knowledge-work datasets.
Deep dive
- APEX-Agents: Benchmark of 480 cross-application tasks for professional services.
- SkyRL: Asynchronous RL framework used for the post-training run.
- TITO (Token-In-Token-Out): Careful bookkeeping of token IDs to avoid misalignment during RL rollouts.
- Harness Optimizations: Fixed sandbox MCP issues and context budget management, yielding significant baseline gains before RL.
- Training Scaling: Scaling to 397B parameters transferred well across different harnesses (Archipelago to OpenCode).
Decoder
- MCP (Model Context Protocol): An open standard for connecting AI models to local or remote data sources and tools.
- Asynchronous RL: A training method where rollouts (environment interactions) and policy updates happen concurrently, reducing idle time.
- TITO: Ensures that the trainer processes the exact token IDs generated by the inference engine to prevent silent training data misalignment.
Original article
Full article content is not available for inline reading.
Explore all 56 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi