DEVOURED
MinIO's MemKV promises 95% better GPU utilization by ending AI recompute tax
MinIO's MemKV moves context from NVMe over 800GbE RDMA to GPUs, claiming 95% better utilization by eliminating recompute tax
What: MinIO launched MemKV on May 13, 2026, a petabyte-scale context memory store that moves data from NVMe to GPUs over 800GbE RDMA. Benchmarks show 95%+ better GPU utilization and ~50% lower cost per token by eliminating 'recompute tax' where GPUs repeat work after losing context.
Why it matters: This signals AI infrastructure competition moving from model performance to token economics and GPU efficiency. As agentic AI scales, the bottleneck isn't compute power but context management—memory layers that preserve state across distributed GPU clusters without forcing recalculation.
Deep dive
- MinIO launched MemKV on May 13, 2026, a petabyte-scale context memory store for AI inference designed to eliminate "recompute tax"—the inefficiency when GPUs repeat calculations after losing context
- Claims 95%+ better GPU utilization and ~50% lower cost per token on benchmark workloads by improving TTFT (Time to First Token) and TPOT (Time Per Output Token)
- Moves context directly from NVMe to GPUs over 800 Gigabit Ethernet RDMA with no HTTP overhead or file system translation, unlike retrofitted file-storage approaches
- Enables stateless serving layers where any GPU replica can resume any conversation mid-flight by pulling cached context in microseconds, eliminating sticky sessions
- Allows per-region deployment instead of global context mirroring, treating geographic placement as a performance choice rather than correctness requirement
- CTO Ugur Tigli frames it as "context-as-a-service"—durable, addressable state like database rows rather than throwaway cache entries
- Security concerns raised by ArmorCode's Karthik Swarnam about securing the memory layer: provenance, access control, and preventing manipulation of contextual data
- MemKV joins AIStor (object storage) as MinIO's second AI data foundation product, specifically purpose-built for the inference data path
Decoder
- Recompute tax: The inefficiency when GPUs repeat calculations they've already performed because context/memory was lost, wasting time and energy on redundant work
- RDMA (Remote Direct Memory Access): Network protocol allowing direct memory access from one computer to another without involving the operating system, enabling microsecond-latency data transfer
- TTFT (Time to First Token): Latency metric measuring how long it takes an AI model to generate its first token of output after receiving a prompt
- TPOT (Time Per Output Token): Metric measuring the time it takes to generate each subsequent token after the first one
- KV cache: Key-value cache storing intermediate computation results (attention keys and values) during AI inference to avoid recalculating them for subsequent tokens
- NVMe (Non-Volatile Memory Express): High-speed interface protocol for SSDs that provides much faster data transfer than traditional storage interfaces
Original article
Full article content is not available for inline reading.
Explore all 67 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi