DEVOURED
Mo' turns, mo' problems
Long-running agentic RL sessions present massive infrastructure challenges, including rollout-training mismatches and the complexity of serialization for tree-structured agent logs.
What: Ian Barber details the operational burden of agentic post-training, where long-running sessions, parallel tool calls, and inconsistent inference/training environments create significant 'off-policy' noise.
Why it matters: As training becomes agentic, the complexity of the harness and sandbox environment matters as much as the model weights themselves.
Deep dive
- Agentic RL is prone to off-policy issues where the rollout trajectory deviates from the model's current weights.
- Long sessions (up to 4+ hours) make harness consistency difficult, leading to different rollout behaviors.
- Tree-structured agent rollouts require complex serialization methods to apply gradient updates.
- Mismatch between latency-optimized inference kernels (SGLang) and throughput-optimized training kernels (Megatron) causes bitwise inconsistency.
- Suggested fixes involve clipping ratio updates or enforcing deterministic accumulation, though both have performance costs.
Decoder
- Rollout: A single generation sequence or episode produced by an agent that is subsequently used for RL training.
- Off-policy: Training on data generated by a previous version of a model (or different sampling parameters), leading to update drift if not corrected.
- KV Cache: The cached key/value states for transformer attention, essential for efficient long-session generation.
Original article
Full article content is not available for inline reading.
Explore all 51 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi