DEVOURED
AI is doing something weird to Science
AI is transforming scientific discovery by excelling as a "proposer" of ideas, but human roles as "poser," "verifier," and "curator" remain indispensable, as shown by Donald Knuth's "Claude’s Cycles."
What: The article by Alejandro Piad Morffis details how AI, specifically LLMs like Claude Opus, act as powerful "proposers" in scientific research, generating novel candidates. Mathematician Donald Knuth validated a pattern, "Claude’s Cycles," proposed by Claude Opus, highlighting AI's role. This changes the loop of science but doesn't replace human roles of posing questions, verifying results (often with non-AI systems like Lean's type-checker or physical experiments), and curating discoveries.
Why it matters: This analysis shifts the conversation around AI in science from replacement vs. dismissal to understanding AI's specific role in a multi-component discovery loop. It emphasizes the critical, often neglected, role of robust verifiers and human-led question posing and curation, suggesting a need for institutional changes to adapt to this new dynamic.
Takeaway: When evaluating AI-in-science claims, specifically ask: "What was the verifier, and who built it?" If the verifier is another LLM or relies on plausibility, be skeptical; if it's a formal system or physical experiment, the results are more trustworthy.
Deep dive
- The scientific discovery process is broken down into four distinct, non-interchangeable roles: poser (human), proposer (AI/model), verifier (formal system/physical world), and curator (human).
- Donald Knuth validated a mathematical pattern, "Claude’s Cycles," proposed by Claude Opus, showing AI's ability to generate novel insights.
- Examples like Terence Tao using LLMs with Lean's type-checker, AlphaFold for protein structures, and Google DeepMind’s GNoME with UC Berkeley’s A-Lab for materials discovery, all demonstrate the "loop" where AI proposes, and an independent, reliable system verifies.
- The key change since 2022 is that the "proposer" slot is now increasingly occupied by general-purpose large language models, making candidate generation cheaper and more widely applicable across domains.
- What hasn't changed is that humans still pose the questions, verifiers are typically non-AI systems, and humans curate which findings are important.
- The article argues that relying solely on an AI proposer without a strong, independent verifier leads to "confident nonsense at scale," as seen with Meta's retracted Galactica model.
- The "verifier is the one that matters" slogan highlights that robust verification is crucial for valid science, even with weak proposers.
- Goodhart’s Law is invoked, suggesting that if institutions continue to optimize for paper count (a measure becoming a target), AI proposers will accelerate the production of low-quality research.
- The author suggests that the most valuable skills in this new paradigm will be posing the right questions and building strong verifiers, which are currently underfunded compared to AI proposer development.
- The piece advocates for thinking of AI as an "AI lab member" – indispensable, capable, sometimes surprising, but not a replacement for a principal investigator or an entity that bears accountability.
Decoder
- Poser: The role in scientific discovery that defines the questions worth asking.
- Proposer: The role in scientific discovery that generates candidate solutions, hypotheses, or patterns.
- Verifier: The role in scientific discovery that rigorously tests and confirms or refutes proposals, often using formal systems, physical experiments, or established scientific methods.
- Curator: The role in scientific discovery that evaluates which verified findings are significant, publishable, and worth pursuing further.
- Claude Opus: A large language model developed by Anthropic.
- Lean: A proof assistant and programming language used for formal verification in mathematics.
- AlphaFold: A deep learning system developed by DeepMind that predicts protein 3D structures from amino acid sequences.
- GNoME (Graph Networks for Materials Exploration): A Google DeepMind system that generates candidate stable crystal structures.
- A-Lab: An autonomous laboratory at UC Berkeley that robotically synthesizes and verifies novel materials.
- Galactica: A large language model by Meta, trained on scientific literature, that was quickly retracted due to generating plausible-sounding but fabricated information.
- Goodhart’s Law: An adage stating that "when a measure becomes a target, it ceases to be a good measure."
Original article
Full article content is not available for inline reading.
Explore all 40 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi