Your agent runs great in a demo. Then a real user hits it across three sessions, references something from last week, and it has no idea what they're talking about. That's a memory problem — and it's the next thing most production AI teams need to solve.
There's a specific moment every team building AI agents eventually hits. The demo works. The first few users try it and it feels like magic. Then the feedback starts coming in: "It forgot what I told it yesterday." "I had to re-explain my whole situation from scratch." "It doesn't know anything about me." The agent isn't broken. It just has no memory.
This isn't a new problem in computing — operating systems have been managing memory hierarchies for decades. But in AI, we've spent two years optimizing for single-session performance and mostly ignored what happens across sessions, across users, and across time. That tab is now coming due.
It helps to think about agent memory the same way an OS thinks about storage: you have fast, expensive, limited memory for active work, and slower, cheaper, persistent memory for everything else.
In-context memory is what the model actively "sees" — everything in the current prompt and conversation. It's instant, perfectly accurate, and gone the moment the session ends. This is what most agents run on entirely. At low session counts, it's fine. At scale, it means every user starts cold every time.
External memory is anything you pull in from outside the context — a user profile, a database of past interactions, relevant documents retrieved by a vector search. This is the architecture Issue 002 covered for RAG pipelines, and it applies just as directly here. The difference is that for agents, the "documents" are previous conversations, extracted facts, and user preferences rather than business knowledge.
Learned memory is what's baked into the model weights through fine-tuning — the stuff the model "knows" without needing to look anything up. As Issue 006 covered, this is genuinely good for behavior patterns and task shapes, but it can't store user-specific facts reliably and it doesn't update without another training run.
Most production memory problems are actually "we're using in-context memory for a job that needs external memory." The fix isn't more context window — it's a write operation at the end of each session and a read operation at the start of the next one.
The pattern that works: after each meaningful agent interaction, extract and store the durable facts — user preferences, key decisions, open tasks, important context — into a persistent store. At the start of the next session, retrieve the relevant subset and inject it into context before the user says anything. The user experiences the agent as remembering them. The agent is actually doing a targeted read from a database.
The extraction step is where most teams get it wrong. The instinct is to store the whole conversation transcript. The transcript is full of noise — pleasantries, clarifying questions, intermediate reasoning steps that don't matter next session. What you want is the distilled version: the facts, preferences, and decisions that would actually be useful to a new agent instance starting cold.
A prompt that says "Given this conversation, extract the key facts about this user that would help a future session — preferences, constraints, open tasks, important decisions made" does that extraction reliably. It's a tiny, cheap call at the end of each session that pays for itself immediately.
Once you have persistent memory, you get a failure mode that Issue 003 didn't fully cover: memory poisoning. A user gives the agent incorrect information in session one. That incorrect information gets extracted, stored, and injected back as fact into every future session. The agent is now confidently wrong, indefinitely.
The mitigation: timestamp everything you store, surface the source when you use it ("Based on what you told me last Tuesday..."), and build a way for users to correct stored facts explicitly. You also want a confidence decay — facts from six months ago should be held more loosely than facts from yesterday, especially for anything time-sensitive like preferences or project status.
Agent memory done well is what makes the difference between a tool users return to and one they abandon after two sessions. The demo works without memory. The product doesn't.
What it is: Zep is an open-source memory layer purpose-built for AI agents and assistants. It handles the full memory lifecycle: storing conversation history, extracting structured facts (entities, preferences, summaries), building a user knowledge graph, and serving relevant context back on retrieval — without you building any of that plumbing yourself.
Why it's worth a look: Most teams reaching for a memory solution either roll their own (fast to start, expensive to maintain) or try to shoehorn a general-purpose vector DB into the job (works, but misses the extraction and graph layers). Zep sits in between: it handles the agent-specific parts — session management, entity extraction, temporal fact storage — while exposing a clean API. It's also self-hostable, which matters if your use case has data residency requirements.
Best for: Teams that have shipped an agent, hit the "it forgot me" wall, and want to add persistent memory in days rather than weeks. The hosted version gets you running in an afternoon; the self-hosted path is straightforward if you need it.
Packer et al., UC Berkeley, 2023
This paper is the cleanest articulation of the memory problem I've read. The core idea: LLMs are constrained by a fixed context window the same way early computers were constrained by limited RAM — and the solution is the same one OS designers found in the 1960s. Build a virtual memory system that manages what's "in RAM" (context) and what's on "disk" (external storage), and let the model itself decide what to page in and out.
MemGPT implements this by giving the model explicit memory management functions — tools it can call to write facts to storage, retrieve past context, and manage what's in its active working window. The model becomes an active participant in its own memory management rather than a passive consumer of whatever you stuff into the prompt.
Why it matters practically: You don't need to implement MemGPT directly to learn from it. The insight that agents should have explicit read/write operations on their own memory — rather than relying on context window size alone — is the design principle behind every good agent memory system being built today, including Zep above. Read the abstract and section 2; the rest is implementation detail you can skim.
Drop this at the end of any agent session to extract a clean memory payload for the next one. Pipe the output into your storage layer directly.
You are a memory extraction assistant. Given the conversation transcript below, extract the durable facts that would be useful to a future agent session with this same user. Return a JSON object with these fields: - "user_facts": key things about the user (role, preferences, constraints, communication style) - "open_tasks": things the user is working on or waiting for that weren't resolved this session - "decisions_made": important choices or conclusions reached in this session - "corrections": anything the user explicitly said was wrong or needed to change from prior context - "expires_soon": any time-sensitive facts that will go stale in the next few days Rules: - Be ruthlessly concise. Each item should be one sentence or less. - Omit pleasantries, clarifying questions, and intermediate reasoning that won't help next time. - If a fact is uncertain, prefix it with "Possibly:" - If nothing meaningful was said on a field, return an empty array for that field. Transcript: [PASTE SESSION TRANSCRIPT]
The output slots directly into a user profile store. On the next session open, retrieve the user's record and inject it into the system prompt before the first user message — something like "Here's what you know about this user from past sessions: [JSON]." Two API calls per session; the agent feels like it actually knows the user.
New issue every weekday at 7am. Free forever.
No spam. Unsubscribe in one click.