DEVOURED
How Anthropic enables self-service data analytics with Claude
Anthropic reports that 95% of its internal business analytics queries are now automated via Claude by prioritizing data governance over raw SQL generation.
What: Anthropic's data team achieved 95% accuracy by treating analytics as a verification and context problem rather than a coding one. They utilize a strictly governed semantic layer, mandatory adversarial review sub-agents, and a 'skill' system where markdown-based reference docs guide Claude's reasoning.
Why it matters: This validates a shift toward 'semantic-layer-first' architectures, where LLMs act as thin agents over a highly structured, human-maintained data foundation rather than just querying raw warehouse tables.
Takeaway: If you are struggling with agent hallucination in data tasks, stop trying to fix the prompt and start building a 'skill' system: move canonical logic into a semantic layer and use a CI-hook to ensure documentation is updated alongside your data models.
Deep dive
- 95% of business analytics queries at Anthropic are now automated using Claude.
- Accuracy is primarily a context and verification challenge, not a code generation one.
- Three main failure modes were identified: concept/entity ambiguity, data staleness, and retrieval failure.
- The stack uses a 'data foundations' layer (canonical models) to solve ambiguity.
- 'Skills' are implemented as folders of markdown that provide procedural knowledge to the agent.
- Offline evals are treated as versioned telemetry to monitor model performance drift.
- Negative results from ablations—such as the failure of raw SQL corpus retrieval—guided the team to focus on structure over volume.
- Adversarial review sub-agents increased accuracy by 6% but increased latency and token usage.
- Corrections from stakeholders are harvested via an automated agent to suggest PRs that update documentation.
Decoder
- Semantic layer: A conceptual model that defines business metrics and dimensions consistently across an organization, sitting between the raw data and BI tools.
- Ablation: A technique where components of a system are selectively removed to determine their individual contribution to performance.
- MCP (Model Context Protocol): An open standard for connecting AI assistants to data sources and development tools.
Original article
Full article content is not available for inline reading.
Explore all 68 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi