Introduction

Reinforcement Learning (RL) is widely used in post-training language models to improve their capabilities on complex reasoning and long-horizon agentic tasks.

In a typical setup, the model makes repeated attempts at a task, receives an outcome reward for each, and learns to favor actions taken in higher-reward attempts. However, a single outcome reward provides a sparse learning signal that does not distinguish which intermediate actions helped or hurt the outcome. This sparsity limits sample efficiency, especially on difficult agentic tasks, where attempts are long and successes are rare.

One popular approach to addressing this sparsity is On-Policy Distillation (OPD), which uses a stronger teacher model to give token by token guidance to the student’s own attempts, providing much denser rewards. However, teaching is a skill in itself. A model that is good at solving a task is not necessarily good at teaching it, as its feedback may be beyond the student’s current capabilities. This motivates training not only the student to imitate the teacher but also the teacher to provide better-aligned guidance to the student.

We propose one such method, Joint On-policy Learning and Teaching (JOLT). Building on On-Policy Self-Distillation (OPSD), we use a single model in two roles: a teacher with access to privileged information, such as verified solutions, and a student without such access. We train the two roles jointly. The student follows the standard distillation objective. The teacher’s objective pushes its feedback to move the student in the same direction as RL on the student's own reward. Together, these objectives train the teacher to meet the student where they are and help the student learn from guidance within their reach.

Our experiments demonstrate improvements in both model performance and training efficiency across a range of mathematical reasoning, coding, and agentic tasks.

From sparse rewards to dense feedback

In its most basic form, outcome-reward RL uses the expected return objective: JRL(θ) = Eτ∼πθ[R(τ)]

Here, τ is a trajectory following a policy πθ, and R(τ) is the return for that trajectory. The corresponding policy gradient is: gRL(θ) = ∇θ JRL(θ) = Eτ∼πθ[∑t R(τ) ∇θ log πθ(at|st)]

where st and at represent the state and action, respectively, at timestep t. To reduce the variance of sample-based estimates, the gradient is often expressed as: gRL(θ) = Eτ∼πθ[∑t Aπθ(st, at) ∇θ log πθ(at|st)]

where the advantage Aπθ(st, at) measures the expected excess return of action at relative to the current policy at state st. One approach, Group Relative Policy Optimization (GRPO), estimates the advantage by comparing each trajectory's return to the mean return of the group sampled for the same prompt.

However, even with these adjustments, outcome-reward reinforcement learning suffers from sparse supervision, particularly on difficult, long-horizon tasks. A growing body of work addresses this by drawing on additional sources of information to provide a denser learning signal.

On-Policy Distillation (OPD) is one such approach. Rather than learning only from a final reward, OPD samples trajectories from the student and evaluates each sampled action under a stronger teacher. For a teacher policy πT and a fixed snapshot θ̄ of the student parameters used to generate the trajectories, the local gradient evaluated at θ = θ̄ is: gOPD(θ̄; πT) = Eτ∼πθ̄[∑t (log πT(at|st) - log πθ̄(at|st)) ∇θ̄ log πθ̄(at|st)]

Here, JOPD(θ; θ̄, πT) is the negative reverse-KL distillation objective. The teacher policy and sampled prefixes are held fixed during differentiation. The teacher-student log-probability ratio provides a dense learning signal at every visited state, replacing the advantage as the action-level weight.

On-Policy Self-Distillation (OPSD) further removes the need for a separate teacher model. The student policy serves as its own teacher given privileged information, such as a verified solution, feedback, or a reference trace. However, OPSD only trains the student to imitate the teacher; there’s no mechanism to ensure that the teacher provides useful signal to the student.

Joint on-policy learning and teaching

JOLT extends OPSD by training a shared model as both the teacher that provides better-aligned guidance and the student that learns from it. The better-informed teacher learns from outcome rewards with KL regularization toward the student, while the unprivileged student learns from the teacher’s dense distillation signal. The teacher update uses teacher-generated attempts, while the student update uses student-generated attempts scored by the teacher. JOLT+ further supplements the student’s distillation signal with outcome rewards.

Training the teacher

Define what a teacher should do

Better teacher performance doesn’t necessarily translate to useful updates to the student. A teacher may use privileged information to exploit shortcuts unavailable to the student, assign high probability to trajectories that are unlikely under the student, or provide unreliable guidance after errors in student-generated prefixes. Teacher quality should therefore be determined by the learning signal provided to the student, rather than by the teacher’s own task reward.

We call a teacher reward-consistent if the distillation update it induces in the student matches the policy gradient of a student’s expected task reward, up to positive scaling. Specifically, a reward-consistent teacher πT⋆ should satisfy: gOPD(θ̄; πT⋆) = 1/βT gRL(θ̄)

where βT is a positive scalar. As in the previous section, gOPD is the OPD local gradient, θ̄ is a fixed snapshot of the student model parameters θ, and gRL is the policy gradient of the expected return objective.

One teacher that satisfies this condition is: πT⋆(a|s,c) = 1/Z(s) πθ̄(a|s) exp(1/βT Aπθ̄(s,a))

Here, Aπθ̄(s, a) represent student continuation advantages and Z(s) is a normalizing constant to ensure a proper probability distribution.

From the ideal target to a teacher objective

We can train a teacher toward the ideal target πT⋆ by minimizing its KL divergence to it. For a fixed student and state s, this gives: min πT KL(πT(·|s,c) ∥ πT⋆(·|s,c))

Substituting the expression for πT⋆ gives an equivalent maximization problem: max πT {Ea∼πT(·|s,c)[Aπθ̄(s,a)] - βT KL(πT(·|s,c) ∥ πθ̄(·|s))}

This objective encourages the teacher to choose actions with higher student advantage while staying close to the student’s action distribution. The parameter βT controls the strength of this penalty: larger values keep the teacher closer to the student’s current behavior, while smaller values allow it to favor high-advantage actions more aggressively.

A practical surrogate objective

This objective depends on how well the student would continue after each action, but those student continuation advantages are generally unavailable. We therefore use centered returns from the teacher’s own trajectories as a practical surrogate. The reward term encourages the teacher to solve the task, while the KL term discourages departures from the student’s current action distribution. Although the KL term keeps the teacher close to the student, this substitution is not exact, as the teacher’s returns need not match the student’s continuation advantages.

JT(θ; θ̄) = Eτ∼πθ̄(·|c)[∑t (sg[ÂT(τ)] log πθ(at|st,c) - βT KL(πθ(·|st,c) ∥ sg[πθ̄(·|st)]))]

Here, ÂT(τ) is the teacher attempt’s return minus the mean return of the teacher-generated attempts in its group. The operator sg denotes stop-gradient: its argument is treated as fixed during differentiation, as are the sampled prefixes and student reference.

Training the student

The student update is a standard OPSD update. The student generates trajectories from the task. The same model then scores each sampled token with the privileged information in context. The student objective penalizes the reverse KL divergence between the student and this privileged teacher at every state the student visits: JOPSD(θ; θ̄) = - Eτ∼πθ̄[∑t KL(πθ(·|st) ∥ sg[πθ̄(·|st,c)])]

Therefore, tokens the teacher prefers are reinforced, while tokens the teacher considers less likely than the student does are suppressed.

Putting it together

Combining the teacher and student objectives gives the joint training objective: JJOLT(θ; θ̄) = JT(θ; θ̄) + βS JOPSD(θ; θ̄)

βS is a scalar that sets how much the student distillation loss counts in the combined update.

The reward-consistency result characterizes the student’s distillation update for a given teacher. JOLT uses this principle to guide joint training, combining the teacher and student objectives to update their shared parameters.

JOLT+ also adds the student's own outcome reward to its update. The centered return of each student trajectory is added to the distillation signal, combining dense teacher supervision with direct outcome feedback on student trajectories. The teacher objective is unchanged, so we can write: JJOLT+(θ; θ̄) = JT(θ; θ̄) + βS JOPSD(θ; θ̄) + Eτ∼πθ̄[∑t sg[ÂS(τ)] log πθ(at|st)]

Here, ÂS(τ) is the student trajectory’s return minus the mean return of its group of student-generated trajectories.

Experiments

We evaluate JOLT and JOLT+ on benchmarks spanning mathematical reasoning, code generation, tool use, and long-horizon terminal tasks, using different base models and different forms of privileged information.

• GSM8K: We train DeepSeek-V2-Lite on GSM8K. The teacher receives the verified answer to each problem.
• MATH: We train Qwen3-0.6B on MATH. The teacher receives worked reference solutions.
• LCBv6: We train Qwen3-8B on 131 coding problems from LiveCodeBench-V6. The teacher receives a model-generated solution that passed all training-visible tests, or execution feedback when no attempt passed them.
• AppWorld: We train Qwen3-4B-Instruct-2507 on AppWorld. The teacher receives summaries of previous attempts.
• Terminal-Bench: We train Qwen3-30B-A3B-Instruct-2507 on TMax, a set of training tasks for terminal agents, and evaluate on Terminal-Bench 2.1. The teacher receives private verifier information.

Accuracy

JOLT+ has the highest mean score in every setting we evaluate. Compared with GRPO, the strongest baseline tested, it improves GSM8K accuracy by 1.5 percentage points (pp), MATH accuracy by 4.2pp, LCBv6 accuracy by 3.0pp, and the AppWorld state-test pass rate by 9.9pp.

Without any direct student reward, JOLT's mean accuracy is higher than GRPO's on GSM8K, MATH, and LCBv6. However, on AppWorld, it stays below both GRPO and π-Distill.

Terminal-Bench 2.1 is the longest-horizon setting. JOLT+ has the highest pass@k at every reported k, from 10.96% at pass@1 to 34.83% at pass@32. JOLT improves on the base model and both distillation baselines at every k, but stays below GRPO. OPSD scores below the base model at every k.

Token efficiency and scaling

JOLT exhibits early sample efficiency while achieving the highest accuracy of the three methods, leveling off near 72%. OPSD improves quickly at small budgets, but plateaus at about 62% accuracy. GRPO learns more slowly at first, but continues improving for longer, climbing to about 70%. At a matched accuracy of 63.44%, GRPO's observed checkpoint uses 8.40 million tokens, while JOLT's fitted curve gives 1.30 million, 6.5× fewer tokens.

We also measure cost using completion tokens, which count student completions for GRPO and OPSD, and both teacher and student completions for JOLT, but omit prompts, privileged context, and the extra optimization passes. Comparison based on completion tokens rather than processed training tokens widens the efficiency gap to about 13× at the same accuracy level. Over a range of target accuracies, JOLT uses 9.5–14.4× fewer completion tokens and 4.3–7.4× fewer processed training tokens.

Conclusion

Privileged information can make a teacher better at solving a task without making it better at teaching a student to solve the same task without that information. JOLT addresses this distinction by training the teacher relative to the student’s current behavior.

This joint training improves both accuracy and efficiency. With student outcome rewards, JOLT+ achieves absolute gains over GRPO of 1.5pp on GSM8K, 4.2pp on MATH, 3.0pp on LCBv6, 9.9pp on AppWorld, and 2.7pp on Terminal-Bench 2.1 pass@1. On MATH, JOLT uses 13× fewer completion tokens and 6.5× fewer processed training tokens than GRPO at matched accuracy, without direct student reward.

JOLT allows the teacher guidance to evolve alongside the student. This motivates exploring whether the teacher can also learn to generate useful training tasks. An interesting direction for future work is to jointly train a task-generating teacher and a problem-solving student, using the student’s learning progress to improve the generated tasks in a curriculum.