DEVOURED
The Cache Is the Price
While headline rates for models appear identical, cache pricing differences make OpenAI's GPT-6 Astra significantly more expensive for context-heavy agentic tasks than Claude Fable 5.1.
What: Julien Simon analyzes how prompt cache read/write costs create a large pricing delta despite similar headline token rates. Because caching is tied to specific models, routing work between models often results in expensive cache invalidation and rebuilds.
Why it matters: This reveals that developers often optimize for the wrong metrics (headline rates) rather than the 'hidden' costs of context management and cache performance.
Takeaway: Check your actual 'cache share' and 'escalation rate' before implementing routing systems, as poor routing can cost significantly more than sticking to one model.
Deep dive
- Headline rates hide the true cost of agentic workloads.
- OpenAI charges $1.00 per million tokens for cache hits, while Anthropic charges $0.25.
- Prompt caches are model-specific; switching models invalidates the cache, forcing an expensive rebuild.
- Escalation from a small model to a frontier model often requires rebuilding the entire prefix context.
- 'Routing' is frequently less efficient than 'selection' (picking the right model for the job upfront).
- The cost of handovers between models is the most overlooked factor in API billing.
- Cache hits usually serve the majority of traffic in long-session agentic workflows.
Decoder
- Prompt Caching: A feature where models store pre-processed prefix tokens, allowing providers to charge less for subsequent calls that use the same initial context.
- Routing: The practice of directing tasks to cheaper models first and only escalating to more expensive 'frontier' models when necessary.
- Frontier Model: The most capable and typically most expensive models produced by labs at any given time.
Original article
Full article content is not available for inline reading.
Explore all 54 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi