DEVOURED
Reward Models Can Be Too Sensitive
Meta researchers found that reward models often overreact to minor differences in equally good responses, leading to harmful 'reward hacking'.
What: Vijay Viswanathan and a team from Meta propose using Monte Carlo dropout to create discrete reward clusters, which reduces the model's sensitivity and prevents the reinforcement learning policy from collapsing.
Why it matters: As reward models become more central to fine-tuning, their sensitivity has become a bottleneck; discretizing these outputs provides a more stable training signal that prevents models from gaming the evaluation metric.
Deep dive
- Reward models often assign different scores to responses that are semantically identical.
- Oversensitivity causes reinforcement learning policies to prioritize 'hacking' the reward model rather than improving performance.
- New metrics of 'discriminative ability' and 'specificity' are proposed to replace traditional accuracy.
- Monte Carlo dropout is applied at inference time to identify reward clusters.
- Experiments show that using discrete reward signals consistently results in higher-quality policies.
Decoder
- Reward Hacking: When an agent exploits unintended patterns in a reward model to maximize its score without actually performing the desired task correctly.
- Monte Carlo Dropout: A technique using dropout during inference to approximate the uncertainty of a neural network's predictions.
Original article
Discretizing Reward Models
Despite their widespread use, the role of reward models in shaping reinforcement learning is poorly understood. Reward models offer a tempting promise: they automatically estimate response quality in the absence of verifiers or human judges. Unlike "verifiable rewards" which typically produce binary scores, reward models typically produce continuous scores, allowing them to be sensitive to fine-grained differences in responses. However, we show this apparent strength is a serious weakness: many popular reward models are oversensitive, assigning different scores to equally good responses. Theoretically, we show that seemingly perfect reward models can be highly oversensitive; empirically, this oversensitivity can lead to bad policies. In place of existing notions of "reward model accuracy," we propose evaluating reward models using distinct measures of "discriminative ability" and "specificity" (the complement of oversensitivity). As a solution, we describe a training-free algorithm that uses Monte Carlo dropout on any neural reward model to produce discrete reward clusters. Theoretically, we prove there exist discretizations that reduce oversensitivity at minimal expense of discriminative ability; empirically we show, in both controlled and natural RL settings, that discretizing rewards leads to less reward hacking and better policies than training on the original rewards.
Explore all 71 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi