Agent Memory Systems
AI agent memory architecture: working, session, episodic, and long-term memory for LLMs, with write rules, forgetting, privacy controls, and production evals.
- Read time
- 16 min
- Level
- Advanced
- Updated
- 2026-08-25
- Components
- Working context · Session store · Episodic log
An agent memory system is a set of stores — working context, session state, episodic logs, and long-term semantic memory — with explicit write rules, retrieval at turn time, and a forgetting policy so an LLM agent remembers what it should and drops what it must.
Key takeaways
- 01
Memory is not "the context window." Split working, session, episodic, and semantic stores; each has a different lifetime and trust level.
- 02
Write rules matter more than embedding choice: what gets persisted, by whom, and with which ACL.
- 03
Unbounded long-term memory poisons future turns with stale preferences, secrets, and wrong facts — forgetting is a feature.
- 04
Retrieve memory with the same permission filter you use for RAG; a remembered document is still a document.
- 05
Evaluate recall and non-recall: the agent should remember the account tier and forget the one-time password.
AI agent memory architecture: four stores
AI agent memory architecture is how state survives a turn. Working memory is the current context window: the prompt, recent turns, tool results. Session memory is durable state for this conversation or ticket (goal, entities, pending confirmation). Episodic memory is a log of past episodes — what happened, when, with which user. Long-term memory for an LLM is usually a semantic store of distilled facts and preferences, retrieved by similarity plus filters.
Do not collapse those into one vector bucket. Working memory is ephemeral and high-trust (you just produced it). Semantic memory is durable, low-trust until verified, and shared across sessions. Mixing them is how a joke in Tuesday's chat becomes Friday's policy.
- Working: context window, tool scratchpad, this turn
- Session: ticket or thread state, TTL in hours to days
- Episodic: append-only event log with timestamps
- Semantic: distilled facts, preferences, with provenance
Write rules, provenance, and privacy
Every write needs a rule: who may persist (user, agent, human reviewer), what type (preference, fact, credential — credentials never), and what provenance (span ID, source quote). Prefer explicit memory tools ("save this preference") over silently embedding every turn. Silent writes maximize recall of garbage.
Privacy is architectural. Memory records inherit the user's ACL and retention class. Do not write another employee's conversation into a shared semantic index. Redact PII at write time if the store is used for retrieval across users. Client-owned stores in the client's VPC are the default for anything that is not public knowledge.
Read path: what to inject, what to fetch
On each turn, load session state always (it is small and precise). Retrieve semantic and episodic memory with a query, a k limit, and a recency prior — not the entire biography. Inject memories as labeled data, not as new system instructions, so a poisoned memory cannot rewrite the agent.
Cap tokens reserved for memory. If retrieved memories crowd out the user question and the tools, you have built a worse RAG system. Rank by relevance, recency, and confidence; drop low-confidence facts rather than hedging them into the prompt.
Forgetting, correction, and decay
Forgetting policy is mandatory. TTL on session state, decay on unused semantic facts, hard delete on user request, and supersede-on-correction ("the office moved") rather than accumulating contradictions. Episodic logs can stay append-only if semantic memory is the only thing retrieved for generation.
Expose a way for operators and users to see and delete what was stored. A memory system you cannot inspect will store the wrong thing, and you will find out in a complaint, not a dashboard.
Failure modes and when not to add long-term memory
Failures: stale facts treated as current, cross-user leakage, secrets persisted because the model "thought it would help," context stuffing from over-retrieval, and write loops that memorize the agent's own mistakes. Confabulated memories — the model saving a fact it invented — need a verifier or a human confirm on writes to the semantic store.
Do not add long-term memory to a single-turn FAQ bot, a system that must not retain user content, or a workflow whose source of truth is already a CRM. Read the CRM. Memory is for state the systems of record do not hold: preferences, in-progress plans, distilled episode summaries. If a database already has the field, query it.
- Stale or contradictory semantic facts
- Cross-user or over-broad ACL on the memory index
- Silent writes that persist secrets or jokes
- Invented facts saved without verification
Evaluating memory
Build cases for recall ("user's timezone is Paris" still true next week), update ("timezone changed to Tokyo"), and non-recall (OTP, card number, another user's ticket). Score retrieval hit rate on the memory index separately from generation. Trace which memory IDs were injected. Operate with retention reports and a delete audit — memory that cannot be forgotten is a compliance bug.