DEVOURED
Reinforcing Recursive Language Models
RL fine-tuning closes the gap: Daniel Kim and Rehaan Ahmad's 4B Qwen matches Claude Sonnet 4.6 on evidence selection, 8x faster.
What: Daniel Kim and Rehaan Ahmad RL fine-tuned Qwen3.5-4B as a recursive language model (RLM) using a single shared policy for parent and child agents. After cold-start SFT on Qwen3.5-397B traces and GRPO training on 1000 evidence-selection queries, the model approached Claude Sonnet 4.6's 0.607 rubric score while completing queries in 7 seconds versus 60+ for Sonnet.
Why it matters: This demonstrates that task-specific RL fine-tuning can enable small models to match frontier model performance on specialized inference strategies, suggesting a path toward deployable, cost-efficient agentic systems that don't require massive models for production use.
Takeaway: Code, training scripts, and RLM scaffold implementation are available on SkyRL for experimenting with RL fine-tuning your own task-specific RLMs.
Deep dive
- Uses GRPO (Group Relative Policy Optimization) where child RLM rollouts inherit the advantage of parent rollouts, enabling single-policy training instead of separate parent/child models
- Training pipeline requires cold-start SFT on teacher traces from Qwen3.5-397B-A17B before RL because small models cannot bootstrap RLM behavior from prompting alone (0 pass@16 without SFT)
- Step-wise training treats each turn as a separate sample since the RLM scaffold rewrites the user prompt per turn rather than accumulating chat history, so an N-turn rollout produces N training samples
- Rubric-based LLM judges for reward assignment proved more robust than F1 span-overlap metrics, which were too noisy when multiple valid answer spans exist for the same question
- REPL environment exposes built-in functions (list_papers, search, extract_section, get_paper_abstract) and RLM-specific calls (FINAL, rlm_query, rlm_query_batched for parallel sub-agent spawning)
- Evidence selection task: given a question and up to 10 arXiv papers, return variable-length verbatim text spans that answer the question (RAG unsuitable due to dynamic span requirements)
- Training objective includes normalization term (1/k_g) when summing child contributions to prevent gradient domination when many children spawned, keeping contributions balanced across RLM tree depth
- Dataset consists of 1000 synthetically generated queries over paper groups, with a frontier model generating questions and selecting relevant paragraphs as ground truth from semantically similar high-upvoted papers
- Models see noisy PDF-parsed text at test time (not clean OCR) to mimic production settings where running OCR on every new document is prohibitively expensive
- Ablation with reduced prompt (200 vs 1500 tokens describing task strategy) converges slightly lower and less stably, suggesting current models still need explicit strategy guidance but future RLM-native models could discover strategies autonomously
- Training performed on single 8xH200 node with batch size 16 and 8 samples per prompt, supporting up to 512 concurrent rollouts across parent and child RLMs
- Recursive extension supports arbitrary RLM depth using recursive subtree loss formulation where each node contributes its own loss plus averaged contributions from all children
Decoder
- RLM (Recursive Language Model): Inference strategy where a language model operates in a programmatic environment (typically a REPL) and can spawn copies of itself as sub-agents to decompose long-context or complex tasks, recursively calling itself with different prompts and contexts
- REPL (Read-Eval-Print-Loop): In RLM context, a Python environment where the model writes code each turn, the system executes it, and results are returned as the next user message—code becomes the primary interface for inspecting and transforming data, not just another tool
- GRPO (Group Relative Policy Optimization): RL algorithm that computes advantages for each rollout relative to a group of sampled rollouts (group statistics) rather than using a learned value function baseline
- Advantage inheritance: Training technique where child RLM rollouts inherit the sequence-level advantage computed for their parent rollout, enabling single-policy training across the entire RLM tree depth without separate reward signals for each child trajectory
- Step-wise training: Each turn in a multi-turn RLM rollout becomes a separate training sample (necessary when turns don't share prefixes because user prompt is rewritten per turn), with the final turn's advantage broadcast backward to previous turns
Original article
Full article content is not available for inline reading.
Explore all 50 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi