DEVOURED
A comprehensive technical guide dissects agent memory architectures from naive FIFO buffers to production multi-tenant systems with episodic, semantic, procedural, and working memory stores.
What: An educational deep-dive into how AI agents implement memory systems. Covers the full stack: context windows and FIFO eviction, the four-way taxonomy (episodic/semantic/procedural/working memory borrowed from cognitive science), vector embeddings and cosine similarity for semantic search, the RAG retrieval loop (embed query → vector search → retrieve top-k → compose prompt → generate → govern → update), production retrieval pipelines (HyDE, Reciprocal Rank Fusion, dense + sparse + graph retrievers running in parallel), governance for supersession and PII redaction, multi-agent memory topologies with scoped permissions, and reference architecture for multi-tenant deployments with hot/warm/cold storage tiers and latency budgets.
Why it matters: This signals the maturation of agent memory from a vector-DB demo into a full system discipline. The shift from 'just append to context' to governed lifecycles (write, age, supersede, redact, forget) mirrors how production databases evolved from append-only logs. The multi-agent topology section reveals the real frontier: when agents share memory across scopes, the failure modes aren't forgetting anymore, they're cross-user leakage and poison propagation, which means memory is now an authz and data-governance problem, not just a retrieval problem.
Deep dive
- Stateless LLMs vs stateful agents: Language models forget everything after each response; agents use an orchestration loop that carries context forward, and memory is the mechanism that decides what goes into the prompt each turn
- Context window ceiling: The simplest memory is stuffing the entire transcript back into the prompt, but every model has a fixed context window; naive FIFO (drop oldest first) breaks when the dropped turn contains critical facts like the user's name
- Working vs long-term memory: Agents juggle live conversation (recent turns, pending tool calls, active reasoning) in working memory and persistent facts in long-term memory; the handoff between them determines what gets encoded, when, and at what granularity
- Memory lifecycle, not storage: Production memory is about write, age, supersede, redact, forget; naive append leaks PII and produces contradictions, naive overwrite loses temporal context, governance preserves history explicitly and marks old facts as superseded while redacting sensitive tokens before they enter the store
- Vector embeddings: An embedding is a list of numbers (typically 1,536–3,072 dimensions for OpenAI models) that places similar meanings near each other in geometric space; retrieval uses cosine similarity to find the closest memories to a query embedding
- Four-way taxonomy from cognitive science: Episodic (time-stamped events, 'what happened when'), Semantic (facts and relations, like a knowledge graph), Procedural (learned skills and tool schemas), Working (the current scratchpad with system prompt + retrieved facts + tool results)
- RAG loop: Receive user message → embed query → vector search → retrieve top-k → compose prompt → LLM generates → govern new info (PII filter, supersession check) → update memory; each stage is a place to spend engineering budget
- HyDE (Hypothetical Document Embeddings): A question and its answer have different shapes; embedding the question directly often misses documents containing the answer, so HyDE asks the LLM to produce a hypothetical answer first, embeds that fake answer, then searches (the answer doesn't need to be correct, just shaped like the retrieval target)
- Hybrid retrieval pipeline: Production systems run dense (embeddings), sparse (BM25 for exact IDs and rare tokens), and graph walk retrievers in parallel, then fuse rankings with Reciprocal Rank Fusion (RRF), apply permission and temporal filters, rerank, and pack survivors into the prompt
- Six architectures compared: Simple buffer (rolling FIFO), Rolling summary (compress old turns), Vector store (embed everything, retrieve top-k), Knowledge graph (structured relations), Hierarchical/MemGPT (paging system), Self-editing/Letta (agent edits its own store); vector + graph + governance is the common production pattern
- Multi-agent memory is a graph: When agents share memory, the question becomes which agent remembers which fact on whose behalf and who else can read it; memory becomes a graph of stores with scopes, permissions, and propagation rules (default should be private memory, shared memory explicit)
- Six multi-agent failure modes: Cross-user leakage (one user's preferences land in another's session), over-sharing (private research notes propagate to project channel), poison propagation (one agent's bad write infects others), conflicting decisions, stale playbook, attribution loss
- Production is a system, not a feature flag: Real agent memory needs an API surface (POST /memory/events, POST /memory/search, DELETE /memory/{id}), separate read and write paths, background extractors for summarization and re-embedding, multi-tenant isolation (namespace per tenant, single collection + filter, or tiered hybrid), latency budgets (target 800ms p95), and hot/warm/cold storage tiers (KV cache for top facts, vector DB for recent episodes, object storage for audit logs)
- Multi-tenancy patterns: Small tenants share a collection with payload filters; enterprise tenants get dedicated namespaces; tiered hybrid lets you sell isolation as a product feature while keeping costs efficient at the long tail
- Latency optimizations: Skip retrieval entirely on queries that don't need it (need detection), cache top-N facts per user in KV, run dense/sparse/graph retrievers in parallel instead of sequentially, each stage (query rewrite 80ms, dense 60ms, BM25 30ms, graph 50ms, rerank 250ms) compounds into p95
Decoder
- RAG (Retrieval-Augmented Generation): Architecture pattern where an LLM's prompt is dynamically augmented with facts retrieved from an external memory store; the agent searches for relevant context before generating each response rather than relying solely on what fits in the context window
- Cosine similarity: Geometric measure of how similar two vectors are, calculated as the cosine of the angle between them; ranges from -1 (opposite) to 1 (identical); used in vector search to rank memories by semantic closeness to a query
- RRF (Reciprocal Rank Fusion): Method for merging rankings from multiple retrievers (dense, sparse, graph) into a single ranked list; each retriever contributes inversely to its rank position, balancing results without tuning weights
- HyDE (Hypothetical Document Embeddings): Retrieval technique that embeds a fake answer to the user's question rather than embedding the question itself, because the shape of an answer is closer to the shape of documents that contain answers
- BM25: Sparse retrieval algorithm (like TF-IDF) that scores documents by exact keyword match and term rarity; complements dense embeddings by catching exact IDs, rare tokens, and queries where lexical match matters more than semantic similarity
- MemGPT: Hierarchical memory architecture that pages context in and out like an operating system's virtual memory, explicitly managing what sits in the LLM's context window vs what lives in external storage
- Letta (formerly MemGPT): Self-editing memory system where the agent can programmatically update, delete, or reorganize its own long-term memory store via function calls, rather than just appending new entries
- Tenant isolation: Multi-tenancy pattern that prevents one customer's data from leaking into another's; implemented via separate namespaces (dedicated collection per tenant), payload filters (shared collection with tenant_id on every query), or tiered hybrid (small tenants share, enterprise gets dedicated)
- p95 latency: The 95th percentile response time; 95% of requests complete faster than this threshold; used as an SLA metric because mean/median can hide tail latency that users actually feel
Original article
Language language models forget everything the moment they finish replying. Memory systems help them 'remember' things so they can have conversations. Agent memory systems are a part of the loop that carries information forward. This article looks at different ideas on what information should be passed on in each loop.
Explore all 73 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi