SpyRL: Self-PlaY Reinforcement Learning

From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement

Self-supervised learning, but for RLVR. We transform an open-ended task into a multi-agent game whose rules automatically generate fully verifiable rewards, enabling scalable LLM self-improvement beyond math and code.

Overview

Drawing on the principle of self-supervised learning, which constructs pretext tasks to derive supervision from the data itself, we propose Reinforcement Learning with Self-Verifiable Rewards (RLSVR), a task-transformation-based training paradigm for extending RLVR to open-ended tasks. RLSVR transforms open-ended tasks into verifiable proxy environments whose internal rules and interaction outcomes automatically generate reward signals.

We instantiate RLSVR with SpyRL, a multi-agent self-play environment inspired by Who Is the Spy?. Agents receive asymmetric information, complete the same target task, and vote to identify a designated spy. Because the spy identity is predetermined, voting outcomes provide fully verifiable rewards, while successful identification remains closely related to output quality.

Key Features

• Self-play beyond verifiable domains: Prior self-play frameworks (R-Zero, Absolute Zero) are proposer–solver loops that need a verifiable solver signal to steer difficulty. SpyRL needs none: the reward comes from the rules of the game, so it extends to summarization and creative writing where no ground truth exists.
• Verifiable rewards without a verifier: The environment assigns the spy identity, so reward is checkable by construction. The performing reward is a zero-sum function of the vote counts, keeping the spy and the civilians in genuine competition.
• Collective, not pointwise, judgement: Every player votes, and rewards are normalized within the group. One detector's misjudgement is outvoted rather than becoming the whole learning signal — far more robust than single-verifier pipelines.
• Cheap document-level data: The performing stage only needs raw documents — reports, story prompts, math-heavy web text. No question-level supervision, no labels, no preference pairs.

How the Game Works

Each task instantiates the same two-stage game with a different information-degradation operator and a different performing objective:

• Summarization: Summarize a government report in one paragraph; the spy has a 20% contiguous span of the report masked.
• Creative writing: Write a story from a prompt; the spy has a 20% contiguous span of the prompt masked.
• Math reasoning: Design and solve a problem grounded in a document; the spy has a 40% contiguous span of the document masked.

The two rewards

Detection (verifiable): The reward is assigned based on whether the detector identifies the correct spy within the group.

Performing (zero-sum): The reward is driven entirely by the vote counts. The first term keeps the spy and the civilians in direct competition; the second penalizes any civilian who drew more suspicion than its peers, so "be good" concretely means "be better than the others who saw the same thing."

Alternating optimization

The two stages (clue/performing and decision/detection) are trained in alternation, driven by the training phase configuration.

Repository Structure

Everything specific to this paper lives in three places:

• spyrl/: Launch scripts.
• verl/experimental/agent_loop/: Game logic.
• verl/utils/dataset/: Environments, corpora, roles, and prompts.

Training

Every run writes checkpoints to the output directory and a human-readable transcript of one full game per step. This file contains every player's prompt and output, the detector prompt, each vote, whether it was correct, and the resulting rewards.

Evaluation

Training produces standard Hugging Face checkpoints, so evaluation runs with the usual tooling. Reasoning tasks use existing benchmarks like GSM8K and Math500, while summarization and creative writing use GPT-4o A/B win rates against the untrained base model.