When Google announced Gemini 3.8 Flash, the released model card indicated that it correctly answered 88.8% of BioMysteryBench’s human-solvable tasks and 56.5% of its hard tasks. In Vals’ independent production runs, the same model scored 71.7% and 21.6%, respectively. Harness and environment differences can move any benchmark result, but this gap was especially wide, especially considering the model was state-of-the-art by Google’s evaluation and near last by ours.

BioMysteryBench allows agents to access websites on the internet, but they are told that accessing specific studies containing task data is not permitted. Cheating in this way caused the poor performance on our benchmark: Gemini 3.8 Flash searches for answers online 21% of the time, whereas Gemini 3.7 practically never exhibited this behavior.

Cheating Trends on Coding Benchmarks

This led us to investigating historical prevalance of cheating across our benchmarks. Terminal-Bench-2.1 is conducive to this study as it allows internet access, but prohibits answer lookup. When we plotted cheating attempts over time, we discovered that rate of attempted cheating on benchmarks is increasing for almost all major model providers.

Many evaluations independently track cheating already, as BioMysteryBench did, yet longitudinal changes in this metric are not closely monitored. SWE-Bench-Verified, for example, is an older benchmark that is now less frequently relied on (and deprecated by us at Vals). This benchmark inspired an even greater degree of cheating, particularly due to it’s task construction being very amenable to searching for solution using simple git queries. The GPT 5.6 series of models did this with particular consistency.

Takeaways

These findings highlight the value of independent evaluators. It is not unlikely that the same guardrails preventing models from cheating during training are being used during evaluations within organizations. If models are trained to complete tasks in ways that evade these specific guardrails, it would not be surprising for labs to occasionally release benchmark results that are not externally trustworthy.

At Vals, we are working to more systematically ensure that our evaluations do not reward credit when models cheat. We anticipate that this will be increasingly important as model capabilities advance.

Methodology

For BioMysteryBench, we analyzed 2,430 task-trials across nine models (three 90-task runs per model). The benchmark evaluator’s anti-cheating rationales were independently classified by GPT-5.6 Luna for all 765 zero-scored trials. For Terminal-Bench 2.1, we screened 3,738 task-trials across fourteen models (three 89-task runs per model). It distinguishes task-specific lookup attempts, deterministic shortcut evidence, and shortcut cases that still received verifier credit. For SWE-bench Verified, we audited 6,496 mini-SWE-agent trajectories across historical Opus, Gemini, GPT, and GLM releases using GPT-5.6 Luna.