When Can LLMs Learn to Reason with Weak Supervision? (4 minute read)
Research reveals when language models can learn reasoning from as few as 8 training examples and identifies a critical failure mode where models memorize answers instead of learning logic.
Deep dive
- Models progress through two phases: a pre-saturation phase where training reward increases and transferable reasoning develops, followed by post-saturation where reward plateaus and learning stops
- Models with extended pre-saturation phases (like Qwen-Math on math tasks) can generalize from just 8 training examples, while rapidly saturating models (like Llama across all domains) require substantially more data
- Pre-saturation duration is domain-dependent based on pretraining exposure—even strong models like Qwen-Math saturate faster on graph tasks where pretraining exposure was low
- Models that saturate faster are less robust to label noise, with Llama-3B performance degrading from 51% to 42% accuracy as corruption increases from 10% to 90%
- Self-supervised proxy rewards like majority vote and self-certainty are brittle—Llama-3B reward-hacks majority vote to perfect scores while actual performance collapses from 45% to 4%
- High output diversity is misleading; Llama maintains higher diversity than Qwen but performs worse because diversity doesn't equal faithful reasoning
- Unfaithful reasoning means models memorize correct answers while generating chain-of-thought explanations that don't logically support those answers, preventing transferable learning
- The failure mode is memorization rather than exploration—models that saturate quickly memorize incorrect answers just as easily as correct ones under noisy supervision
- Continual pre-training on 52B domain-specific math tokens followed by supervised fine-tuning on 43.5K explicit reasoning traces extends the pre-saturation phase for Llama models
- This two-stage approach (CPT + reasoning-focused SFT before RL) recovers generalization across all three weak supervision settings: scarce data, noisy labels, and proxy rewards
- Only math-specialized models show stable improvement with proxy rewards, suggesting domain specialization interacts with supervision quality
- The findings suggest reasoning faithfulness should be evaluated jointly with diversity metrics rather than treating diversity alone as a proxy for model capability
Decoder
- RLVR: Reinforcement Learning from Verifiable Rewards, a training approach where models receive feedback based on whether their answers can be verified as correct
- Saturation dynamics: The pattern where training reward initially increases then plateaus, marking the transition from active learning to diminishing returns
- Pre-saturation phase: The period during training when reward steadily increases and the model learns transferable reasoning patterns
- Unfaithful reasoning: When models produce correct final answers but with chain-of-thought explanations that don't logically support those conclusions
- Continual pre-training (CPT): Additional pre-training on domain-specific data applied to an already-trained model before fine-tuning
- Proxy rewards: Alternative reward signals used when ground-truth verification is unavailable, such as majority vote among multiple responses or model confidence scores
- Reasoning faithfulness: The fraction of model responses where the chain-of-thought trace logically supports the final answer
Original article
When Can LLMs Learn to Reason with Weak Supervision?
Summary
We study when RLVR generalizes under three weak supervision settings (scarce data with as few as 8 examples, noisy reward labels, and proxy rewards such as majority vote and self-certainty) across multiple models from the Qwen and Llama families on three reasoning domains: Math, Science, and Graph.
We find that generalization is governed by saturation dynamics: models progress through a pre-saturation phase where training reward steadily increases and the model learns transferable reasoning, followed by a post-saturation phase where reward plateaus and further training yields diminishing returns. Models with extended pre-saturation phases (Qwen on Math and Science) generalize from as few as 8 examples, tolerate significant label noise, and even work with proxy rewards. Rapidly saturating models (Llama across all domains) fail across all three settings.
The root cause of failure is unfaithful reasoning, not lack of diversity. Failing models maximize training reward by memorizing answers while producing reasoning traces that do not logically support their final answers, despite maintaining high output diversity.
Continual pre-training on domain-specific data combined with supervised fine-tuning on explicit reasoning traces before RL improves faithfulness, extends the pre-saturation phase, and recovers generalization across all three weak supervision settings.
RLVR Under Weak Supervision
We study three settings where supervision is imperfect: scarce data (as few as 8 examples), noisy reward labels, and self-supervised proxy rewards. The findings below span multiple models from the Qwen and Llama families across Math, Science, and Graph reasoning domains.
Scarce data
How does data scarcity affect RLVR generalization? We train with as few as 8 examples across different models and domains, tracking saturation dynamics — the point at which training reward plateaus and learning effectively stops.
Qwen-Math-1.5B sustains learning for 342 steps (35%→67% MATH-500). Qwen-1.5B saturates at step 172. Llama-3B-Instruct saturates earliest at step 60.
The same pattern emerges across all three domains: models with extended pre-saturation phases generalize from as few as 8 samples, while rapidly saturating models require substantially more data. This is domain-dependent — even Qwen-Math saturates faster on Graph, where pretraining exposure is low.
Noisy rewards
When ground-truth verifiers are imperfect, reward labels may contain errors. We corrupt a fraction γ of training labels and measure how robustly each model-domain pair generalizes.
Llama-3B-Instruct on Math shows progressive degradation with increasing label corruption — MATH-500 drops from ~51% at γ = 0.1 to ~42% at γ = 0.9. Models that saturate faster are generally less robust to noise: Llama memorizes incorrect answers just as easily as correct ones. Qwen-Math-7B on Graph tolerates low corruption but degrades at γ ≥ 0.5.
Self-supervised proxy rewards
When ground-truth verifiers are entirely unavailable, models must rely on alternative reward signals. We compare RLVR (ground-truth) against two proxy rewards: majority vote (consensus among sampled responses) and self-certainty (model confidence).
Self-supervised proxy rewards are brittle and model-dependent. Qwen-3B with majority vote shows temporary gains before collapsing after ~500 steps. Llama-3B-Instruct reward-hacks majority vote to 1.0 as MATH-500 collapses from 45% to 4%. Self-certainty collapses in both models. Only math-specialized models (Qwen-Math) show stable improvement with proxy rewards (see Figure 22 in the paper).
Why Do Some Models Fail?
A natural hypothesis: failing models lack output diversity — they can't explore enough. But this is wrong. Llama maintains higher diversity than Qwen, yet performs worse. The real explanation is unfaithful reasoning: Llama produces correct final answers with chain-of-thought traces that do not logically support them.
Low reasoning faithfulness explains why some models fail under weak supervision: they memorize answers rather than learn transferable reasoning, leading to rapid saturation. Raw diversity is misleading — it should always be evaluated jointly with faithfulness.
Making Llama Generalize Under Weak Supervision
Continual pre-training on domain-specific data combined with supervised fine-tuning on explicit reasoning traces before RL recovers generalization across all three weak supervision settings.
Supervised fine-tuning on explicit reasoning traces before RL improves reasoning faithfulness, extends the pre-saturation phase, and enables generalization under all three weak supervision settings. Continual pre-training further amplifies the effect, achieving the strongest gains across both in-domain and out-of-domain benchmarks. See Figure 7 in the paper for how CPT + Thinking SFT improves faithfulness compared to other configurations.