Sanket's Tutorials

11 - Memory for LLM Apps

11 min read

Scope

LLMs are stateless — each call sees only what’s in the prompt. “Memory” is everything you do to make an app act like it remembers: what to keep in the context window now (short-term), what to persist in an external store and retrieve later (long-term), and — the 2025-26 frontier — letting the agent itself decide what to write, update, and forget. Covers conversation-memory strategies, the episodic/semantic/procedural taxonomy, vector + temporal-graph backends, the frameworks (Mem0, Letta/MemGPT, Zep, LangMem), retrieval + forgetting/decay, evaluation (LongMemEval, LoCoMo), and the risk surface (memory bloat, stale facts, PII/GDPR, memory poisoning). For the agent loop that uses memory, see 08 - Agents & Agentic Systems; for the attack surface, see 14 - Safety, Security & Risk.

Related: Gen AI Engineer, 08 - Agents & Agentic Systems, 14 - Safety, Security & Risk, LangGraph Senior Prep

1. The concept: the model has no memory — you build it

A chat model is a pure function: tokens in, tokens out. It has no state between calls. The “memory” a user perceives is an illusion you engineer by deciding what text lands back in the next prompt. Two regimes:

Short-term memoryLong-term memory
Whereinside the context window (the prompt itself)external store (DB / vector index / graph)
Lifetimeone conversation / threadacross sessions, days, users
Mechanismyou re-send prior turnsyou retrieve relevant past facts and inject them
Bounded bycontext length + cost/latencystorage + retrieval quality
Failure modetruncation, “lost in the middle”retrieval miss, stale/contradictory facts

Mental model

Short-term memory = the model’s working set (RAM). Long-term memory = disk you page in on demand. The whole discipline is deciding what’s worth promoting to RAM right now, and what’s worth writing to disk for later. Memory is not RAG over documents — it’s RAG over the interaction history and learned facts about the user/agent.

2. Conversation-memory strategies (short-term)

The classic four ways to fit a growing conversation into a finite window. (Pattern names from the LangChain era; the patterns outlive the deprecated classes — see §8.)

StrategyWhat it keepsProsCons
Bufferevery turn, verbatimlossless, simpleoverflows the window; cost grows every turn
Sliding window (buffer-window, last k)last k turnsbounded cost, recency-correctsilently forgets anything older than k
Summarya running LLM-written summary of the whole chatvery compact, unbounded historylossy; summarization drift; extra LLM call per update
Summary-buffer (hybrid)summary of old turns + verbatim recent turnsbest of both — detail near, gist farmost moving parts; tune the cutoff

Default that ages well

Summary-buffer is the pragmatic default for assistants: verbatim last few turns (precision where it matters) plus a rolling summary (cheap long tail). It also sets up the consolidation pattern that defends against memory poisoning (§9).

Summarization is lossy and can hallucinate

A summary can drop the one detail the user later asks about, or invent one. Never summarize a tool-call result, a number, an ID, or a legal/medical fact — keep those verbatim or push them to long-term store. Summaries are for narrative, not for facts you’ll be held to.

3. Vector / semantic memory (long-term retrieval)

The standard long-term backend: embed each memory (a fact, a turn, a summary) into a vector, store it, and at query time embed the current input and pull the top-k most similar memories into the prompt. It’s 07 - RAG machinery pointed at interaction history instead of documents.

  • Granularity matters. Storing raw turns gives precision but bloats fast and retrieves noise. Storing extracted facts (“user is vegetarian”, “prefers metric units”) retrieves far cleaner — this extraction step is what Mem0 and LangMem automate.
  • Scoping is mandatory. Every memory must carry a user_id (and often agent_id / thread_id) so retrieval is filtered per-user. Without it you leak one user’s memories into another’s prompt — a privacy incident (§9).
  • Hybrid retrieval (vector + metadata filter + recency) beats pure cosine similarity; relevance alone surfaces old, superseded facts.

4. The memory taxonomy: episodic / semantic / procedural

Borrowed from cognitive science, now the standard vocabulary (LangMem, Letta, and the research literature all use it). Knowing the three types — and that they need different stores and write policies — is a senior signal.

TypeWhat it holdsExampleTypical storeWrite cadence
Episodicspecific past events/experiences (“what happened”)“Last Tuesday the user asked to cancel order #441”vector store / event logappend per event
Semanticdistilled facts & knowledge (“what is true”)“User is allergic to penicillin”vector store / KV / graphextract + dedupe + update
Proceduralhow to behave (“how to do it”)the agent’s own system prompt / few-shot rules it has learnedprompt / rules blockrewrite/optimize, rarely

Procedural memory = the agent editing its own instructions

The most modern and least obvious type: the agent rewrites its own system prompt or few-shot examples based on feedback (“the user hates verbose replies → tighten the persona”). LangMem supports this explicitly via prompt optimization. It’s how an agent learns a behavior rather than just recalling a fact.

Entity memory is a focused slice of semantic memory: track a structured record per entity (person, project, company) and update its fields as new turns arrive — closer to a CRM row than a free-text fact. Good when the domain has stable entities you keep referring to.

5. The frameworks (2025-26)

You rarely build memory from scratch. The current landscape:

FrameworkCore ideaBackendDistinctive strength
Mem0LLM extracts facts from each turn, then a decision step issues ADD/UPDATE/DELETE/NOOP to keep the store consistentvector (+ optional graph variant, Mem0^g)production-oriented; reports large token/latency savings vs stuffing full history
Letta (was MemGPT)LLM-as-OS: agent self-edits in-context memory blocks; pages between tiers via tool callscore (in-context) / recall / archival tiersself-editing memory blocks; stateful-agent server
Zep (engine: Graphiti)temporal knowledge graph — facts are edges with validity intervals, so superseded facts are invalidated, not deletedtemporal KG (bi-temporal)tracks how facts change over time; non-lossy history
LangMemSDK over LangGraph’s store; supports semantic and procedural (prompt-optimizing) memoryLangGraph BaseStorenative to LangGraph; the only one that rewrites the agent’s own prompt

How to choose

  • Need to know when a fact was true / track changes over time (e.g. “what was the user’s plan last month”) → Zep/Graphiti temporal graph.
  • Want a drop-in extract-and-consolidate layer with low token cost → Mem0.
  • Building stateful agents that manage their own memory tiers → Letta.
  • Already on LangGraph and want procedural/semantic memory native → LangMem.

Latency varies wildly

LLM-driven memory (extraction, self-edit, prompt optimization) costs inference on the write path. LangMem’s procedural optimization has been measured in the tens of seconds at p95 — fine for a background/batch job, unusable inline in a chat turn. Push heavy memory writes off the request path.

6. Agentic memory: agents that self-edit (the frontier)

The 2025-26 shift mirrors the reasoning-model shift in 02 - Decoding, Sampling & Reasoning Controls: from you hand-coding “summarize after N turns” to the agent deciding what to remember.

  • The agent has memory tools — add_memory, update_memory, delete_memory, search_memory — and calls them inside its reasoning loop when it judges something worth keeping.
  • MemGPT/Letta pioneered this: the model itself decides when to write to core memory or page archival memory in/out. Mem0’s full-agentic mode lets the LLM choose what to store and how to phrase the retrieval query. A-Mem (2025) generates linked, self-organizing memory notes.
  • Upside: adaptive, no brittle hand-tuned rules. Downside: memory quality = the model’s judgment, every operation costs tokens, and a bad write persists and contaminates future turns.

Self-editing memory is also the attack surface

The same loop that lets an agent write its own memory lets an attacker trick it into writing a poisoned memory (§9). Autonomy on the write path and security on the write path are the same design problem.

7. Retrieval over memory + forgetting / decay

A store that only grows is a store that gets slower, costlier, and noisier. Mature memory systems forget on purpose.

Retrieval scoring — don’t rank by similarity alone. The well-known composite (from generative-agents work) blends three signals:

score = w_sim · relevance(query, mem)     # semantic match
      + w_rec · recency(now − last_access) # exponential time decay
      + w_imp · importance(mem)            # LLM-rated salience 1–10

Forgetting strategies:

StrategyHowWhen
TTL / expirydrop memories older than Xephemeral context (today’s task)
Importance-weighted decaylow-importance + old → evict firstgeneral long-term store
Consolidationevery N turns, summarize raw turns into facts, then delete the rawbound growth + shrink attack window
Supersessionmark old fact invalid when a newer one contradicts (temporal graphs do this natively)facts that change (“moved to Berlin”)

Forgetting is a feature, not a bug

Importance-weighted decay keeps the store small enough that retrieval stays sharp, and limits how long a poisoned or stale entry can live. “Remember everything” is the wrong default.

8. ⚠️ LangChain’s classic memory classes are deprecated

A common interview trap and a real migration in live codebases.

ConversationXMemory is dead — use LangGraph persistence + LangMem

The old ConversationBufferMemory, ConversationBufferWindowMemory, ConversationSummaryMemory, ConversationSummaryBufferMemory classes were deprecated in LangChain v0.3.1 and slated for removal at v1.0. They had no real persistence, no per-conversation isolation, no production story.

The replacement is LangGraph persistence, with two primitives:

  • Checkpointer — in-thread (one thread_id) conversation history; this is short-term memory done right (durable, resumable).
  • BaseStore — cross-thread, user-scoped long-term facts.

Layer LangMem on top for LLM-driven extraction and procedural (prompt-optimizing) memory.

The strategies in §2 (buffer/window/summary/summary-buffer) are still correct — you now implement them over the checkpointer/store instead of the retired classes.

9. Caveats & risk mitigation

  • Memory bloat → retrieval noise. An ever-growing store buries the relevant memory under near-duplicates; top-k starts returning junk and the model gets worse. Mitigate: extract facts not raw turns; consolidate every N turns then delete the raw; importance-weighted decay; dedupe on write.
  • Stale / contradictory facts. “User lives in Paris” and “User moved to Berlin” both retrieved → the model picks one at random. Mitigate: supersession on write (UPDATE/invalidate, not just ADD — Mem0’s update step, Zep’s temporal invalidation), and store a timestamp so newer wins.
  • PII retention + GDPR right-to-deletion. A long-term store quietly accumulates personal data, and Article 17 (right to erasure) means a user can demand it all be deleted — including copies baked into summaries and graph edges. Mitigate: scrub/redact PII before write, strict per-user scoping (user_id on every record), documented retention/TTL, and a real deletion API that purges the raw store and derived artifacts. Treat “we can’t find all their data” as a compliance failure, not a tech detail. See 14 - Safety, Security & Risk.
  • Prompt injection writing poisoned memories. This is the sharpest agentic-memory risk. An attacker embeds instructions in ordinary-looking input; the agent’s own self-edit loop stores them, and they fire on future turns — even for other users of a shared store. The MINJA attack (NeurIPS 2025) achieved this via query-only interaction with reported >95% injection success. A poisoned memory in a multi-agent system can propagate downstream fast. Mitigate:
    • Validate writes — schema/policy-check and (LLM- or rule-) classify candidate memories before they persist; reject instruction-like or out-of-policy content.
    • Separate read and write paths — agent reads from a validated snapshot; writes land in a staging area and only promote after review, so in-session poisoning can’t immediately steer behavior.
    • Consolidate-then-delete shrinks the window a raw poisoned turn can survive.
    • Per-user scoping stops cross-user contamination; read-only audit snapshots give you a clean baseline to diff against.
  • Self-editing = trusting model judgment. Agentic writes are only as good as the model; a confidently-wrong extraction persists. Mitigate: keep a human-/eval-in-the-loop on high-stakes memories; log every write with provenance so you can trace and roll back.
  • Cost & latency on the write path. Extraction, consolidation, and prompt optimization are extra LLM calls. Mitigate: do them async/batched off the request, not inline — tie to 15 - Cost, Latency & Performance.

10. In practice

# LangGraph short-term (checkpointer) + cross-thread long-term (store) + LangMem
from langgraph.checkpoint.postgres import PostgresSaver
from langgraph.store.postgres import PostgresStore
 
checkpointer = PostgresSaver(conn)   # in-thread conversation history (short-term)
store = PostgresStore(conn)          # cross-thread, user-scoped facts (long-term)
 
# Always namespace long-term memory PER USER (scoping = privacy + isolation)
namespace = ("memories", user_id)
store.put(namespace, key="diet", value={"fact": "vegetarian"})
hits = store.search(namespace, query="what does the user eat", limit=5)
# Mem0: extract-and-consolidate layer (ADD/UPDATE/DELETE decided by the LLM)
from mem0 import Memory
 
m = Memory()
# write path: facts get extracted from the turn; updates supersede, not duplicate
m.add("I switched from Paris to Berlin last month", user_id="u_123")
# read path: scoped, semantic retrieval into the prompt
context = m.search("where does the user live", user_id="u_123")
# Defensive write: validate before persisting (anti-poisoning, §9)
def safe_write(candidate: str, user_id: str):
    if looks_like_instruction(candidate) or contains_pii(candidate):
        return  # reject / scrub — don't store attacker-controlled directives
    staging.put(("staging", user_id), value=candidate)  # promote after review

11. Sources

Backlinks

Graph View