DEVOURED
Contra Labs evaluated AI models across 15,000 professional creative judgments and found no model leads all three workflow phases in any domain, with GPT 5.3 Codex and Grok Imagine Video each starting last in ideation but reaching first place by refinement.
What: Contra Labs ran a study where professional creatives from their network (1.5M+ members, $250M+ earned) evaluated outputs from Claude Opus 4.6, Gemini 3.1 Pro, GPT 5.3 Codex, Veo 3.1, Grok Imagine Video, and others across landing pages, desktop apps, ad images, brand assets, and product videos. They separated evaluation into convergence (objective criteria like typography and layout where evaluators agree) and divergence (taste-based preferences where disagreement is signal, not noise). Claude and Veo dominated ideation but dropped by refinement. Gemini excelled at mockup when design systems were introduced but struggled with iteration. GPT 5.3 Codex and Grok Imagine Video started weakest and climbed to first by refinement.
Why it matters: This exposes that creative AI workflows are multi-model problems, not single-model problems. The industry default of asking 'is this output good?' assumes quality is a single axis, but professional creative judgment operates on two orthogonal dimensions: verifiable best practices (where evaluators should converge) and aesthetic direction (where disagreement reflects legitimate taste differences). A model can excel at following briefs and still produce generic, undifferentiated work because best-practice adherence and taste flexibility are independent capabilities.
Takeaway: If building creative tools, design UX for phase transitions and multi-model workflows rather than forcing users into one model. If using AI creatively, match model to stage: Claude Opus 4.6 or Veo 3.1 for exploration, Gemini 3.1 Pro for design system execution, GPT 5.3 Codex or Grok Imagine Video for refinement edits.
Deep dive
- Contra Labs developed the Human Creativity Benchmark (HCB) to separate convergence (evaluator agreement on objective criteria) from divergence (evaluator disagreement reflecting taste), arguing most AI benchmarks incorrectly treat creative disagreement as noise when it's actually signal
- Study drew from Contra's network of 1.5M+ independent creatives who've earned $250M+, evaluating outputs across three workflow phases: ideation (exploration), mockup (actualizing direction), and refinement (production-ready tweaks)
- 93 prompts across 5 domains (landing pages, desktop apps, ad images, brand assets, product videos) produced 15,000 judgments using pairwise ranking, scalar ratings (prompt adherence, usability, visual appeal), and qualitative feedback
- No model led all three phases in any domain: Claude Opus 4.6 and Veo 3.1 dominated ideation then dropped; Gemini 3.1 Pro excelled at mockup when design systems were introduced; GPT 5.3 Codex and Grok Imagine Video started last/third and climbed to first by refinement
- Veo 3.1 was the only model that degraded across all phases on every dimension, with evaluators noting it introduces new creative elements rather than applying targeted edits, making it excellent at generation but unreliable for refinement
- Ad images showed clearest convergence arc with evaluator agreement rising from 0.345 to 0.549 across phases as criteria shifted to verifiable typography, CTA placement, and contrast; landing pages ran counter (0.484 to 0.333) as outputs became acceptable and personal judgment took over
- Evaluator agreement varies by dimension: high on prompt adherence (shared, checkable criteria), low on visual appeal (personal, distributed criteria), confirming the two-signal separation is working as designed
- In ad images, usability functioned as hard gate: outputs scoring 5 on usability finished top-two 84% of the time vs 10% for score-1 outputs, regardless of visual quality; high visual appeal could not rescue low prompt adherence
- Desktop apps evaluation surfaced 15 core themes spanning prompt adherence, usability, layout, visual hierarchy, readability; epistemic network analysis revealed Claude Opus 4.6's usability tightly bound to prompt adherence, while Gemini 3.1 Pro showed weaker coupling
- Evaluator attention shifts predictably: ideation focuses on layout and structure, mockup shifts to design system fidelity (color, typography), refinement narrows to production details like typography legibility and spacing
- Study proposes best-practice adherence and taste flexibility are orthogonal axes, not a single quality dimension: models can be strong on defaults (reliable outputs), strong on steerability (responds to creative direction), both, or neither
- Limitations: select evaluator group, prompts not externally validated, phased structure compresses real creative workflows where designers iterate fluidly and revisit stages; future research will explore longer, less constrained creative arcs and multi-model workflows
Decoder
- Bradley-Terry model: Statistical method for aggregating pairwise comparison data (A vs B, B vs C, etc.) into a global ranking, commonly used in sports rankings and preference studies
- Kendall's W: Coefficient of concordance measuring agreement among multiple raters, ranging from 0 (no agreement) to 1 (complete agreement)
- Epistemic network analysis (ENA): Method for analyzing how themes or concepts co-occur in qualitative data, revealing patterns in how evaluators connect ideas when explaining their judgments
- Mode collapse: Phenomenon where generative AI models converge on safe, averaged outputs rather than producing diverse, distinctive variations, particularly problematic in creative domains
Original article
Most AI benchmarks treat taste-based disagreement as noise to be resolved, but the Human Creativity Benchmark separates objective criteria from subjective preferences, since creative work lacks ground truth.
Explore all 66 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi