DEVOURED
Accelerating LLM Inference with Prompt Caching for Open‑Source Models on Databricks
Databricks now offers automatic prompt caching for open-source LLMs like Llama and Mistral hosted on its platform, significantly boosting inference performance without configuration.
What: Databricks has extended its automatic prompt caching feature to open-source LLMs, including GPT-OSS, Llama 3.1, Gemma 3, Mistral, and DBRX models, on its Foundation Model APIs. This capability reuses identical prompt prefixes to reduce redundant processing, resulting in a 2.5x increase in throughput and a 3x reduction in P50 latency for GPT-OSS in production, and requires no customer setup.
Why it matters: This move makes serving open-source LLMs on Databricks more competitive and efficient, directly addressing the common issue of repetitive prompt processing that wastes compute resources and increases costs, especially for applications using long system prompts or frequent identical requests.
Takeaway: If you're running open-source LLMs on Databricks, expect automatic performance improvements in inference speed and cost-efficiency due to the new prompt caching feature.
Deep dive
- Databricks has rolled out automatic prompt caching for open-source LLMs hosted on its Foundation Model APIs (FMAPIs).* This feature applies to batch inference, pay-per-token, and provisioned-throughput workloads.* Supported open-source models include GPT-OSS (20B, 120B), Gemma 3 12B, Fine-tuned Llama 3.1 8B, Llama 3.1 8B, and 3.3 70B.* Prompt caching works by reusing repeated prompt prefixes, skipping the "prefill" stage of LLM inference.* This significantly reduces latency and increases throughput.* In real-world production testing on GPT-OSS, it led to a 2.5x increase in per-replica input-token throughput and a 3x reduction in P50 latency, with a 30% cache hit ratio.* The caching is entirely automatic; customers do not need to configure anything.* Prompt caches are isolated, reside only in volatile memory, and are never persisted, ensuring data security.* Databricks previously offered this feature for proprietary models (GPT, Gemini, Claude).
Decoder
- LLM (Large Language Model): A type of artificial intelligence model trained on vast amounts of text data to understand, generate, and process human language.* Inference: The process of using a trained machine learning model to make predictions or generate outputs on new, unseen data.* Prompt caching: A technique in LLM inference where the intermediate computational results (specifically, the key-value cache or KV cache) for repeated prompt prefixes are stored and reused, avoiding re-computation and speeding up subsequent requests with the same prefix.* Prefill stage: The initial phase of LLM inference where the input prompt tokens are processed to generate the first set of internal representations (KV cache) before the model starts generating output tokens one by one.* P50 latency: The 50th percentile latency, meaning 50% of requests are completed within this time or faster. It's a measure of typical performance.
Original article
Full article content is not available for inline reading.
Explore all 62 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi