DEVOURED
The AI Superforecasters Are Here
AI agents scaffolded for deep research now rival top human superforecasters in prediction accuracy, potentially automating the 'opinion layer' of decision-making.
What: Startups like FutureSearch and Preseen are using multi-agent scaffolds to automate research-heavy forecasting, achieving results competitive with human experts on platforms like Metaculus and Kalshi. These systems use subagents to perform exhaustive web research and analysis, often completing in minutes what takes humans weeks.
Why it matters: This signals a transition where AI moves from mere information retrieval to synthesizing subjective probabilities, which could redefine how businesses and governments allocate capital and form policy.
Takeaway: Try prompting an LLM with 'Think like a superforecaster and answer with a probability' for complex questions to gain a structured probabilistic analysis of an issue.
Deep dive
- AI superforecasters use 'scaffolds'—orchestration layers that manage subagents to research, analyze, and synthesize data.
- Current performance is roughly parity with top human superforecasters, with models improving at approximately 0.9 Metaculus Elo points per month.
- Models excel in high-data, well-contained domains like finance but struggle with complex, long-horizon societal questions.
- The methodology shifts forecasting from a rare, manual luxury to an automated utility.
- Prediction markets may evolve from being the primary source of truth to a mechanism for aggregating the opinions of AI agents.
Decoder
- Superforecaster: A person or system with demonstrated high accuracy in predicting the likelihood of specific future events.
- Scaffold: A software wrapper around an LLM that structures its reasoning process, manages agent loops, and enforces tool usage to improve output quality.
- Metaculus: A popular forecasting platform that tracks and scores the accuracy of human and algorithmic predictions on global events.
- Conjunctive chain: A sequence of events that all must happen consecutively for a specific outcome to occur; each link in the chain multiplies the uncertainty.
Original article
Full article content is not available for inline reading.
Explore all 67 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi