Memory systems for AI agents

Most language models are stateless between requests. The model sees the current prompt and whatever context the application puts inside it. Conversation history can provide continuity, but sending an ever-growing transcript is expensive, noisy, and eventually limited by the context window.

Memory is the application layer that decides what is worth keeping, how it should be represented, and when it should be brought back. It is closer to a curated, searchable record than to a magical second brain.

What memory means for an agent

A useful memory system has three jobs: write information from an interaction, organize it so it can be found later, and read only the pieces that help with the current task. The loop is simple:

This pattern combines ideas from retrieval-augmented generation, which pairs a model with an explicit non-parametric knowledge store,^[1] and agent architectures that store experiences, form higher-level reflections, and retrieve them for planning.^[2]

Types of memory

Short-term or working memory

The active context for the current task: recent messages, the current plan, tool outputs, and constraints. It is fast and directly visible to the model, but limited in size and easy to pollute.

Episodic memory

Records of events and experiences: what happened, when it happened, and what the agent or user did. Episodes are useful when the answer depends on sequence, time, or prior attempts.

Semantic memory

Durable facts and preferences extracted from episodes. “The user prefers concise answers” is semantic memory; “the user asked for a concise answer on Tuesday” is an episode.

Procedural memory

Reusable ways of doing things: workflows, tool instructions, policies, or successful strategies. Procedures should be versioned and tested because a stale instruction can cause repeated errors.

These categories are useful boundaries, not mandatory database tables. A small agent may store them together with clear metadata. A larger system may use separate stores and retrieval policies.

A practical memory layer

Start with an append-only event log. Preserve the original interaction, timestamp, source, and task or session identifier. Then run a memory writer that extracts candidate facts or episodes. Each candidate should carry evidence pointing back to the source event.

memory = {
  "content": "User prefers concise technical explanations",
  "kind": "semantic",
  "source_event": "evt_1842",
  "created_at": "2026-09-15T10:30:00Z",
  "confidence": 0.91,
  "status": "active"
}

Store candidates in a form that supports both exact filters and approximate search. A vector index can find semantically related memories, while ordinary fields handle user identity, time ranges, type, permissions, and status. Retrieval-augmented systems benefit from explicit external memory because it can be updated and inspected independently of model parameters.^[1]

At read time, create a query from the current task, retrieve a broad set, apply filters, rerank the survivors, and compress them into a small context block. The final prompt should contain the memory content and its source or timestamp when that helps the model judge relevance.

Key design principles

• Do not save everything. Repetition, transient chatter, and guesses create retrieval noise.
• Keep provenance. Evidence makes memories auditable and gives the system a way to explain or correct itself.
• Model change explicitly. New information can update, supersede, or contradict an old fact. Do not silently keep both as equally true.
• Use time carefully. A preference from last year may be less useful than a recent one. Recency should be a signal, not an automatic override.
• Give users control. Memory needs inspection, correction, deletion, and clear retention rules, especially when it contains personal data.
• Protect the context budget. More retrieved text is not always better. A few relevant memories usually beat a long dump.

Evaluate the system

Test memory as a system, not just as a similarity search. Include questions that require remembering a fact, reasoning across sessions, tracking changes over time, and abstaining when the answer was never stored. LongMemEval uses 500 questions to test abilities including information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.^[3]

Track at least retrieval recall, answer correctness, unsupported-memory rate, update accuracy, latency, and token cost. Inspect failures by stage: was the memory not written, badly represented, missed by retrieval, or misused by generation?

A tiered design can also help with scale. MemGPT describes moving information between memory tiers so an agent can work beyond the model’s immediate context window.^[4] In practice, that might mean recent context in a fast store, durable memories in a searchable store, and cold history in cheaper archival storage.

The goal is useful continuity

Agent memory is a product and systems problem: representation, retrieval, updates, privacy, and evaluation all matter. The best first version is usually small and observable. Save fewer things, attach evidence, make retrieval measurable, and add complexity only when a real failure requires it.

← Back to all writing