How to Add Memory to an AI Agent
How to add memory to an AI agent or chatbot: working, session, and long-term stores, write policies, privacy, and evals for recall that do not leak PII in 2026.
To add memory to an AI agent, split working context (this call), session memory (this conversation), and long-term memory (durable facts across sessions); write only what a policy allows; retrieve with query-time ACLs; and score recall on a golden set. Long-term memory is a permissioned database, not a bigger prompt.
What you’ll build
- Separate working, session, and long-term memory with explicit TTLs
- A write policy that records only durable facts, not every token of chitchat
- Query-time ACLs so user A cannot retrieve user B’s memories
- Forget/export APIs that actually delete vectors and rows, not only the UI flag
- Recall precision/recall on a golden set, plus a leakage test for PII
Before you start
- 01An agent or chatbot with traces of turns, tools, and user ids from your IdP
- 02A session store (Redis or equivalent) with TTL and encryption at rest
- 03A long-term store: SQL for structured facts, hybrid index for notes, or both
- 04A written retention and PII policy for what the agent may remember
- 05A recall golden set: questions that depend on earlier turns or prior sessions
Key takeaways
- 01
Working, session, and long-term memory have different TTLs, stores, and failure modes — do not collapse them into one vector index.
- 02
Write policies matter more than embedding models: remember preferences and decisions, not every utterance.
- 03
Session vs episodic vs semantic memory is a design choice: episodes are events, semantic rows are facts; retrieve both with hybrid search when needed.
- 04
Query-time ACLs and a real delete path are mandatory; memory is PII.
- 05
Eval recall (did we surface the right fact) and leakage (did we surface someone else’s) on every pin change.
Working, session, and long-term memory
How to add memory to a chatbot starts with not calling everything memory. Working memory is the context window for this inference. Session memory is the thread: what we are doing now. Long-term memory is what should still be true next Tuesday: preferred language, account number the user asked you to keep, the project alias.
Episodic memory is “what happened” (a support call last Thursday). Semantic memory is “what is true” (the user is on plan Pro). Many agents need both: retrieve episodes with hybrid search, read semantic facts as rows. Mixing them in one undifferentiated vector store is why the agent recalls a joke and forgets the plan.
Write policies: what gets remembered
Unconstrained writes turn the index into a second, worse log. Use an explicit writer: a tool `remember_fact` with a schema (key, value, scope, expiry), or a classifier that only promotes certain intents. Prefer user-confirmed writes for durable personal data. Do not remember passwords, full payment numbers, or health data unless the product is built for that and the store is approved.
Every write needs source turn id, timestamp, and expiry. Facts should be updatable: a new “preferred name” supersedes the old row rather than accumulating contradictions. Conflict policy is part of the design; the model should not vote on two addresses without asking.
Retrieving memory without drowning the prompt
At turn start, load session slots, then retrieve k long-term items with hybrid search (vector + keyword) over that user’s memories. Re-rank. Cap tokens. Put memories in a labeled untrusted-data block so a stored note cannot jailbreak the agent (indirect injection into memory is real).
Do not retrieve the whole user history. Recency boosts and type filters (preference vs episode) keep the window small. If nothing retrieves, proceed; do not invent a memory. Citations (“as you said on 12 May”) should point at a stored item id.
Privacy, query-time ACLs, and forgetting
Memories are PII. Encrypt at rest, scope by tenant and user, and filter at query time — never bake “this is Alice’s” only into the embedding text. Operators should not have a wide-open UI over all memories. Retention: session TTL in hours; long-term with a documented max age unless the user refreshes the fact.
Forget means delete rows, vectors, and caches. Export means a file the user can read. Test both in CI with a fixture user. If legal holds apply, that is a flag on the row, not a second shadow index you forget to filter.
Evals for recall, noise, and leakage
A recall golden set is a scripted dialogue: turn 1 states a fact, turn N asks it back, possibly after a session break. Score: correct recall, refusal when the fact was never stored, and no recall of another user’s fixture. Also score whether the agent over-recalls (dragging an old project into a new question).
Shadow mode for write policies: log what would have been stored without persisting, until precision of writes is acceptable. A four-week build can ship session memory in week 2, long-term with ACLs in week 3, and evals plus forget APIs in week 4. The client owns the stores, evals, and runbooks.
- Working: capped turns + grounded summary, not the whole log
- Session: TTL store with structured slots
- Long-term: policy-gated writes, hybrid retrieve, query-time ACLs
- Eval: recall, noise, cross-user leakage, real delete
When not to add long-term memory
If the agent is a one-shot form filler, session TTL is enough. If the facts live in CRM or IAM, read those APIs instead of copying them into a vector store that will drift. Duplicate stores create conflicting truth.
If you cannot resource a forget path and ACL tests, do not persist. A chatbot that remembers nothing is safer than one that remembers everyone.
Step-by-step build
- 1
Name the three memories and their TTLs
Working: the current prompt window (system, latest turns, tool results). Session: the conversation state with a TTL (hours). Long-term: durable facts keyed by user/tenant that survive sessions. Write these into the design doc so nobody “just adds Redis.”
- 2
Implement working context with a cap
Keep the last N turns plus pinned instructions and the active tool results. When over budget, summarize older session turns into a bounded blob that cites turn IDs. Never drop the current user message or the latest tool error.
- 3
Add session state
Store conversation id, user id, rolling summary, and structured slots (ticket id, cart, language) in a TTL store. Load on each turn. Session memory should die when the TTL expires unless a write policy promotes a fact to long-term.
- 4
Define long-term write and retrieve policies
Promote only tagged facts: preferences, decisions, identifiers, and explicit “remember this.” Store structured facts in SQL; store free-text notes in a hybrid index with user and tenant metadata. Retrieve at turn start with hybrid search, filtered by the asking user.
- 5
Enforce privacy: ACLs, redaction, forget
Filter memories at query time. Redact secrets before write. Provide export and delete that cascade to vectors. Do not log raw memories to a third-party tracer without a residency review. Treat retrieved memories as untrusted data for prompt-injection purposes.
- 6
Evaluate recall and leakage
Build dialogues where the answer depends on a prior fact (session and cross-session). Score recall, extra-memory noise, and cross-user leakage. Shadow a new write policy on live traffic before it can persist. Fail CI if leakage is non-zero.
Common pitfalls
The mistakes that show up in real deployments — each one costs a week if you learn it the hard way.
Stuffing the entire chat into every call
That is a log, not memory. It blows tokens, repeats stale instructions, and still forgets last month. Cap working context and summarize or retrieve the rest.
Writing every turn into the vector store
You will retrieve jokes, secrets, and outdated preferences with equal confidence. Write only when a policy says the fact is durable (preference, decision, account id).
One global memory index
Without query-time user (and tenant) filters, long-term memory is a cross-user leak. Filter before the model sees hits. Empty-for-other-user must look like a miss.
Summaries that invent facts
A rolling summary is a new hallucination surface. Ground summaries in quoted turns, keep a revision, and eval summary faithfulness the same way you eval RAG.
No forget path
GDPR-style deletion that only hides a row while embeddings remain is not deletion. Cascade deletes to session, SQL, and vectors, and test it.