Agent Memory & Persistence

A coding agent can finish a task, pass its tests, and still forget the work when the developer returns. In the previous chapter, billing_event_78291 stayed coherent because one run carried a task contract, a review branch, pnpm test --run billing, before-and-after replay receipts, and a human merge decision. Four days later, that packet isn't in the model's next prompt unless the runtime rebuilds it.

The developer opens a new chat: "Same duplicate-invoice failure. Continue." The application may still hold last week's transcript, but the large language model (LLM) sees only the context assembled for this call. If the event ID, branch agent/billing-event-78291, and verifier aren't in that prompt, the agent asks which test was failing, or worse, invents one.

That gap is the persistence problem. An AI agent (an application that uses an LLM to choose or propose steps) can't carry task state across sessions on its own. Its runtime has to store state outside the model, then select a small, scoped packet for the next prompt. The billing task makes the storage jobs visible: a short-term buffer for the live turn, a pinned core for must-keep facts, and long-term stores for events, knowledge, and approved procedures.

Memory also adds a trust boundary. A recalled note can tailor a reply, but it can't authorize a merge. A transcript may contain private data or malicious instructions, so storage and retrieval need tenant scope, provenance, retention rules, and the same action checks you'd require without memory.

Keep two questions separate as we follow the billing replay: what should the model see, and what may the runtime still do?

What problem does agent memory solve that a bigger model alone doesn't solve?

Answer

It stores task-specific history outside the model weights. A bigger model may reason better, but it still doesn't know this repository's current event ID, branch constraints, preferred verification command, or last review outcome unless the runtime retrieves and passes that information into the prompt.

Why every API call starts from scratch

What does the model actually receive when a new request arrives? A response is computed from the input assembled for that inference. The runtime might replay prior messages from a transcript store, but if relevant history isn't included in the later request, the model can't use it. That's statelessness at the model boundary: the model doesn't remember prior requests by itself.

Past information needed during one inference must arrive inside the context window, the bounded token sequence the model processes for that request. Current systems range from modest windows to one million or more tokens, but the window is still temporary and finite.^{[1]Reference 1Long contexthttps://ai.google.dev/gemini-api/docs/long-context} Longer inputs generally increase time to first token and billed input, even when caching lowers repeated-prefix cost. Replay everything and the prompt becomes slow, noisy, and eventually too large. Replay too little and the agent loses continuity.

At inference time, serving systems also maintain a key-value (KV) cache: tensors of attention keys and values for tokens the model has already processed. That cache makes multi-turn chats faster and cheaper to serve, but it isn't persistent agent memory. It lives inside the inference engine and can be evicted at any time. It can't answer a semantic question such as "what verifier did this repo use last month?"

Systems such as PagedAttention and RadixAttention make KV reuse more efficient, but they don't replace an external memory store ^{[2]Reference 2Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180[3]Reference 3SGLang: Efficient Execution of Structured Language Model Programshttps://arxiv.org/abs/2312.07104}. The cache speeds up work the serving path has already done; a memory store decides which past facts deserve retrieval.

⚠️ Common mistake: Treating the context window as long-term memory. It's volatile, costly to refill, and limited. Models with 1M+ token windows exist, but position and irrelevant context still matter. "Lost in the Middle" found weaker retrieval when relevant information appeared between the beginning and end of long inputs ^{[4]Reference 4Lost in the Middle: How Language Models Use Long Contextshttps://aclanthology.org/2024.tacl-1.9/}. Broader controlled tests also report performance declines as input length grows, even when the needed fact remains present, often called context rot ^{[5]Reference 5Context Rot: How Increasing Input Tokens Impacts LLM Performancehttps://research.trychroma.com/context-rot}. External memory lets the runtime select a small evidence packet instead of replaying every past token.

A coding agent has a KV cache hit on the last conversation prefix. Does that mean it has remembered the developer's preferred verification command from last month?

Answer

No. The KV cache is an inference optimization for tokens already processed in the current serving path. It can speed up reuse of a prefix, but it doesn't search past sessions or store durable facts. Last month's verification preference needs external memory, such as a project profile, event log, vector store, or relational record.

Why isn't a very long context window a complete memory strategy?

Answer

Long context lets you pass more text into one call, but it doesn't decide what should be saved, retrieve facts across sessions, isolate users, resolve stale facts, or keep costs bounded. Memory is a storage and retrieval design problem, bigger than context size.

Once context and KV cache are separate, memory design has one practical question: which facts deserve a place in the prompt, and which should stay outside it?

The three tiers of agent memory

Agent architectures often borrow names from human memory, using a design vocabulary organized by CoALA (Cognitive Architectures for Language Agents) ^{[6]Reference 6Cognitive Architectures for Language Agentshttps://arxiv.org/abs/2309.02427}. The mapping isn't a claim that model memory works like a brain. It separates engineering jobs: some state must stay in the live prompt, some history should be recalled on demand, and some knowledge needs a reviewed durable record.

Those jobs become storage layers you can build. Pinned facts and current work share the prompt. Broader records stay outside it. Merge authority never comes from either store.

Working memory: the live desk

Working memory is the context window itself, the agent's immediate processing buffer. Everything currently "in mind" lives here. In engineering terms, it's short-term memory because it matches what the model receives in one inference request.

• Current conversation messages
• Retrieved documents
• Tool call results
• System instructions

For the billing fix, working memory holds turn-by-turn details: the latest developer message, the result of pnpm test --run billing, or the two invoice IDs the replay just returned. It's also transient. When the conversation grows long, older messages must be dropped or summarized to make room.

Core memory: the pinned note

Core memory sits between working and long-term memory. It holds a small set of sourced facts needed in nearly every live turn. Unlike episodic stores that grow over time, core memory stays deliberately small and high-value. Teams usually implement it as a scoped profile projection injected on every turn.

For the returning billing task, core memory might store:

• Event ID and target branch (billing_event_78291, agent/billing-event-78291)
• Preferred verification command (pnpm test --run billing)
• Current request (finish the webhook idempotency patch; merge approval not established)

These facts are too important to let them scroll out of the context window. Pin {"verification_command": "pnpm test --run billing"} into core memory and the agent keeps it in every prompt, instead of forgetting the verifier after ten messages.

What kind of fact should be promoted from working memory into core memory?

Answer

A small fact that's high-value, stable enough to reuse, and needed in nearly every turn. Event ID, preferred verification command, active branch, and current objective fit. A one-off stack trace usually doesn't; it belongs in working or episodic memory.

Long-term stores: the filing cabinets

The remaining three memory types live in external storage and are retrieved on demand. Each one answers a different question.

Episodic memory records specific past interactions. The agent might recall that last week's replay of billing_event_78291 created two invoices because the handler allocated a new invoice row on every POST. That event suggests a hypothesis. It doesn't prove the current failure has the same cause.

Semantic memory stores distilled facts and concepts. The application might preserve a sourced repository preference, or retrieve a versioned rule stating that a webhook retry must return the original invoice. Developer chat isn't a safe source for inventing engineering policy. Vector similarity helps with fuzzy recall; exact identifiers, current status, and relationship queries are usually safer through authoritative relational or graph lookups, sometimes combined with embeddings ^{[7]Reference 7Unifying Large Language Models and Knowledge Graphs: A Roadmap.https://arxiv.org/abs/2306.08302}.

Procedural memory catalogs evaluated and approved strategies, such as running the named billing verifier and lint before proposing a merge. Observed outcomes can produce candidates for evaluation; they shouldn't silently rewrite instructions or grant the agent permission to merge.

Classify these memories for the billing-agent: current replay result, repository owner team, last week's PR transcript, "webhook retries must return the original invoice," and the approved merge playbook.

Answer

The current replay result is working memory while it's active in the live turn. A repository owner team loaded from repo metadata can be projected into core memory because it's small and regularly relevant. Last week's PR transcript is episodic memory. The versioned retry rule is semantic memory. An approved merge playbook is procedural memory.

When should a memory be stored in a relational table instead of only a vector store?

Answer

Use relational storage for exact facts and access-controlled records: user ID, repository ID, owner team, branch name, issue or PR ID, approval status, timestamps, and source-of-truth links. Use vector search for fuzzy recall. Exact operational facts shouldn't depend on nearest-neighbor similarity alone.

The live desk still fills up. Once a conversation no longer fits, the runtime needs a policy for what to keep, compress, or retrieve.

Pattern 1: sliding window with summary

The first pattern answers a narrow question: what can stay in the prompt when a single conversation grows? Keep the most recent messages raw and continuously summarize older ones. Immediate context stays precise while older context is compressed.

After the billing thread grows past a handful of turns, the agent doesn't need the full text of "Can you rerun the billing tests?" It does need the result: "Replay of billing_event_78291 returned two invoices." A sliding window keeps the last few messages raw and folds everything older into a rolling summary.

Here is the smallest working version. The class stores recent messages plus one summary. When add_message pushes the list past window_size, it takes the overflow, asks a summarizer to keep FACT: payloads, and retains only the recent window. get_context returns those fields separately, so prompt assembly can label the summary as untrusted historical data instead of promoting generated text into a trusted system instruction.

With a window size of 3, five messages leave the summary holding the overflow, including the event ID, while the recent list holds messages 3-5. Its shorter form is also lossy. A detail that looked minor in an early turn can vanish unless the summarizer is told to keep it. Exact transcripts should remain retrievable when precision matters.

Why is a rolling summary safer than dropping old messages outright, and why is it still risky?

Answer

It's safer because the agent keeps a compressed record of older facts instead of forgetting them completely. It's still risky because summarization is lossy. Details that look unimportant during compression can become important later, so exact transcripts should remain retrievable when precision matters.

When is sliding-window memory enough?

Answer

It's enough for short sessions where continuity matters only inside the current chat and old details don't need exact recall. It's not enough for cross-session personalization, audit trails, regulatory records, or tasks where a small old fact can become important later.

Pattern 2: retrieval-augmented memory

The window pattern protects the current turn, but it eventually loses older information. For memories worth recalling fuzzily, an application can index scoped projections as embeddings (vector representations that capture semantic meaning) and retrieve candidates per query. It shouldn't blindly embed every transcript or treat a vector index as the source of truth for repository state, approval, merge eligibility, or production changes.

Exact or sensitive records belong in access-controlled stores with source IDs, lifecycle controls, and an optional search projection. This applies Retrieval-Augmented Generation (RAG) ^{[8]Reference 8Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.https://arxiv.org/abs/2005.11401} to memory without confusing retrieval with authority.

Selective recall can beat replaying the whole transcript every turn, but published comparisons are setup-specific. In Mem0's own LoCoMo writeup, a full-context baseline that stuffed the whole conversation (about 26k tokens in that dataset) still scored highest on its LLM-as-a-Judge metric. Mem0 used far fewer tokens and reported much lower p95 latency, about 1.44s versus about 17s in that setup ^{[9]Reference 9Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memoryhttps://arxiv.org/abs/2504.19413}.

Treat those numbers as one vendor-reported comparison, not a universal ranking. The narrower lesson is useful: a few scoped candidates can cut prompt size, but accuracy and safety still depend on write policy, filtering, provenance, and evaluation.

A coding agent that handles hundreds of repositories can't scan every transcript when a new message arrives: "This looks like the same billing replay failure." Instead, it encodes the query, searches a scoped store, and retrieves the most relevant past experiences.

Generative Agents retrieve from a memory stream by combining relevance, recency, and importance after min-max normalization. The original implementation used equal weights and an exponential recency decay of 0.995 per sandbox hour ^{[10]Reference 10Generative Agents: Interactive Simulacra of Human Behavior.https://arxiv.org/abs/2304.03442}. Production rankers often use the same three signals with different weights and a calendar half-life.

The next example uses weights 0.5 / 0.3 / 0.2 and a one-week half-life. Those constants are application knobs, not canonical values. Similarity and importance must already sit on [0, 1] before they're added.

Now check the numbers by hand. A memory with similarity 0.90, age 24 hours, and importance 0.8 has recency exp(-ln(2) * 24 / 168) ≈ 0.91. Its combined score is 0.5 * 0.90 + 0.3 * 0.91 + 0.2 * 0.8 = 0.45 + 0.273 + 0.16 = 0.883.

Compare a memory with similarity 0.95, age 336 hours (two weeks), and importance 0.3. Its recency is exp(-ln(2) * 336 / 168) = 0.25, and its combined score is 0.5 * 0.95 + 0.3 * 0.25 + 0.2 * 0.3 = 0.475 + 0.075 + 0.06 = 0.61. The newer, more important memory wins despite lower semantic similarity.

Why does the newer memory win in the concrete scoring example even though its semantic similarity is lower?

Answer

The scoring function isn't pure similarity. It combines similarity, recency, and importance. The newer memory has strong recency and higher importance, so its combined score beats the older memory that only has a slightly better semantic match.

The score answers which eligible memory is more useful. It doesn't answer whether that memory may be seen. Tenant scope must be enforced first. The in-memory index below computes cosine similarity, but it ranks only rows that already match the tenant filter.

⚠️ Common mistake: Ranking on similarity before source, scope, and supersession checks. An agent might retrieve an older similar retry over yesterday's billing_event_78291 replay because the phrasing matches the query better. Filter ineligible and superseded records before reranking. A composite score can't resolve provenance or decide authorization.

The output proves only that the billing tenant survived the filter. Write, recall, and execution are still three different paths. Follow the replay failure across all three, and keep merge authorization outside each path.

Recall provides context, not permission

Memory records may be stale, incorrectly extracted, or derived from untrusted user text. The write path needs a schema and source record. The read path needs tenant filtering, supersession handling, and retrieval limits. A separate policy or system-of-record lookup decides whether an action is allowed.

Ask a narrow question first: which verification command is current? The next example resolves it only from current, directly sourced facts. Inferred memories remain useful as review hints, not profile updates.

A verification command helps choose a check, but it still doesn't authorize a merge. The executor must read a strictly typed approval record and use an idempotency key before creating an external effect. A string such as "approved" is truthy in Python, but it isn't an approval decision.

Why should the write path and read path be designed separately?

Answer

Writing decides what deserves storage, how to classify it, and which metadata to attach. Reading decides what's relevant to the current task, how to rerank candidates, and how much context to inject. If you merge the paths, agents tend to either save everything noisily or retrieve everything wastefully.

Pattern 3: MemGPT and hierarchical memory management

The first two patterns keep context small or find relevant records. MemGPT adds an explicit paging policy for moving information among those spaces.

MemGPT ^{[11]Reference 11MemGPT: Towards LLMs as Operating Systemshttps://arxiv.org/abs/2310.08560} adapts the operating-system idea of virtual memory. Its main context contains read-only system instructions, a fixed-size writable working block, and a first-in, first-out (FIFO) queue of recent messages. The first slot in that queue holds a recursive summary of turns already evicted. Outside the bounded prompt, recall storage retains the full message log and archival storage holds searchable long-term text objects such as documents:

When prompt tokens cross a warning threshold (the paper's example is about 70% of the window), the queue manager inserts a memory-pressure warning. The model can then copy important facts into the working block or archival storage before eviction. At a flush threshold, the manager evicts a slice of the queue, writes a new recursive summary into slot 0, and leaves the evicted messages in recall storage ^{[11]Reference 11MemGPT: Towards LLMs as Operating Systemshttps://arxiv.org/abs/2310.08560}.

The model requests those moves through function calls, which makes memory management an agentic capability. In a production implementation, the host must still enforce tenant scope, accepted sources, size limits, and write policy; the model isn't its own access-control system.

The open-source MemGPT project now ships as Letta, which documents editable in-context memory blocks (label, description, value, and a character limit) alongside recall and archival memory ^{[12]Reference 12Agent Memory: How to Build Agents that Learn and Rememberhttps://www.letta.com/blog/agent-memory/}. LangGraph makes a similar split: thread-scoped short-term memory backed by a checkpointer, and long-term memory stored under namespaces ^{[13]Reference 13LangGraph Memory Overviewhttps://docs.langchain.com/oss/python/concepts/memory}. The Mem0 paper describes an extraction, update, and retrieval memory layer ^{[9]Reference 9Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memoryhttps://arxiv.org/abs/2504.19413}.

These architectures describe memory mechanisms, not your application's policy. Tenant scope, retention, privacy, and action authorization still belong in the host.

Anthropic's client-side memory tool exposes file operations (view, create, str_replace, insert, delete) for a /memories directory while leaving storage implementation to the application. Its documentation calls out size bounds, expiration, sensitive-data validation, and path-traversal protection ^{[14]Reference 14Memory toolhttps://platform.claude.com/docs/en/agents-and-tools/tool-use/memory-tool}. A file-backed boundary still needs lifecycle and access controls; a writable directory alone isn't persistent recall you can trust.

What's the key difference between a sliding window and MemGPT-style memory paging?

Answer

A sliding window is mostly a fixed system policy: keep recent messages and summarize older ones. MemGPT-style paging gives the agent tools to move information between working context, recall storage, and archival storage when it decides the task needs it.

What's the risk of giving the agent tools that edit its own core memory?

Answer

Bad writes can persist bad assumptions. The agent might overwrite a verified preference with a hallucinated one, persist prompt-injection text, or append sensitive data that shouldn't be pinned. Core-memory writes need schemas, source attribution, validation, access controls, retention, and sometimes human approval.

Core memory updates

Core memory stores a small set of high-value facts in the prompt. In MemGPT-style designs, writable memory is distinct from pinned system instructions. The host should admit updates only to permitted fields from accepted sources. The model may propose a write, but it shouldn't append arbitrary profile or authorization text.

Write admission isn't only for core memory. Archival inserts and entity extraction need the same kind of gate: allowed fields or content classes, accepted sources, max length, a blocklist of authorization-like fields, tenant scope, and a retention TTL. Reuse one host-side helper for every tier so "store whatever the model extracted" never ships by accident.

The next helper makes that boundary explicit. It admits a small preference, rejects an authorization field, and applies the same tenant and retention checks to an archival write.

Archival memory

The admitted archival write now has somewhere to go. Archival memory provides long-term storage that's much larger than the context window and searched on demand. Insert only after admit_memory_write (or an equivalent host gate) accepts the payload. The store below is an in-memory stand-in; a production archival index would add embeddings and access control, but the admission check stays in the host.

If those stores are persisted, the same agent can maintain cross-session continuity by reloading user facts and prior events. That solves recall. Crash recovery and exact step-by-step resume are separate runtime concerns: they come from a checkpointing layer around the agent loop, not from MemGPT's memory hierarchy itself ^{[15]Reference 15LangGraph Persistencehttps://docs.langchain.com/oss/python/langgraph/persistence[16]Reference 16Temporal Workflow Execution Overviewhttps://docs.temporal.io/workflow-execution}.

Retrieval memory vs. workflow state

Long-running agents need more than facts to retrieve later. Retrieval memory answers "what does the agent know?" Checkpointed workflow state answers "where exactly should it resume after a crash?" Durable agent runtimes persist the current graph node, pending tool calls, and intermediate outputs after each step. LangGraph checkpoints graph state into threads, while Temporal persists workflow execution state and replays from recorded event history after failures ^{[15]Reference 15LangGraph Persistencehttps://docs.langchain.com/oss/python/langgraph/persistence[16]Reference 16Temporal Workflow Execution Overviewhttps://docs.temporal.io/workflow-execution}.

The split matters at the exact moment a run fails. A recalled "merge approved" note can inform a proposal. Resume still needs the checkpoint, a current approval check, and side-effect reconciliation.

An agent crashes after storing a note that claims "merge approved" but before sending the merge tool call. Which store answers what it recalls, and which store answers where to resume?

Answer

Retrieval memory returns the note as evidence, not as current approval. Checkpointed workflow state answers where to resume: current graph node, pending tool call, intermediate outputs, and whether the merge side effect actually happened. The resume decision should use workflow state plus current approval and side-effect reconciliation, not memory alone.

Why is "merge approved" in memory not enough to safely resume after a crash?

Answer

It describes an intended or inferred fact, not necessarily the committed side effect. The runtime must check workflow state and the merge system of record to know whether the merge call happened, whether it should retry, and whether the retry is idempotent.

Compressing memories so they fit

Recall and checkpoints answer different questions, but both still feed a bounded prompt. As conversations grow, raw history becomes impractical. Two compression strategies are especially useful.

Progressive summarization

As raw message history accumulates, it can be chunked and summarized in hierarchical layers to conserve tokens. The helper below takes a list of raw messages and a summarizer, then iteratively chunks the context and generates summaries at multiple compression levels. historical_summary is an internal data label, not a trusted chat role.

The useful question is what each compression step costs: fewer tokens, but less access to exact wording.

This creates a pyramid: four raw messages become one summary, those summaries become a meta-summary, and so on. Prompt assembly must keep historical_summary subordinate to system instructions. Each step is lossy. A detail that seems minor now ("rollout must remain canary-only") might be the key to a later release decision.

When should the agent retrieve the raw transcript instead of trusting a summary?

Answer

When exact wording, commands, diff hunks, stack traces, approvals, dates, policy exceptions, or incident evidence matters. Summaries are useful for orientation, but source text should remain available for decisions that affect production state, account access, developer trust, or auditability.

Entity-based extraction

Extracting structured entities converts candidate facts from conversation text into records that an application can validate and query. Extraction output isn't yet a trusted knowledge graph: it needs tenant scope, source turns, timestamps, supersession handling, and an admission policy before it changes a profile or affects an action.

The output contains both a preference and a merge request. Only the preference is pinnable. Extraction produces candidates; admission is a separate host decision.

Why extract structured entities if you already have summaries?

Answer

Summaries are good for narrative continuity. Structured candidate facts are good for precise queries, filters, validation, and conflict detection. A summary may say "project prefers pnpm test --run billing," while an admitted entity record can store key, value, source turn, timestamp, confidence, tenant scope, and supersession status.

Production memory design

A vector database solves only one piece of the problem. When a single-user prototype becomes a system serving thousands of concurrent users, memory also needs tenant isolation, minimized retention, deletion propagation, provenance, prompt-injection defenses, latency budgets, and consistency across stores.

Multi-user isolation and namespacing

In a multi-tenant production system, personal memories can't be retrieved from an unscoped index query. Whatever physical index layout you choose, tenant authorization must be enforced before candidates enter the prompt. The retriever below queries reviewed shared knowledge and tenant-isolated private memory, then merges the results as context candidates, not as authorization.

Watch one property in the output: Ada's query can see shared knowledge and Ada's own row, but never Lin's private incident note.

🎯 Production tip: Use a storage interface that applies tenant scope automatically and test denial across tenants. A caller remembering to add WHERE user_id = '123' isn't a sufficient security boundary by itself. Namespace, row-level security, index layout, and encryption choices depend on the backend and threat model.

The output shows why an application-layer filter is advisory: it works only while every code path remembers to add it. A single query that forgets it returns cross-tenant rows silently. Database-enforced row-level security (RLS) moves the boundary into the engine. You attach a policy to the table (for example, in Postgres, a policy that compares a row's tenant_id against a session variable), and ordinary application roles subject to that policy can't return rows outside the current tenant. Use a least-privileged application role without superuser or BYPASSRLS; table owners normally bypass RLS unless the table uses FORCE ROW LEVEL SECURITY.^{[17]Reference 17PostgreSQL Row Security Policieshttps://www.postgresql.org/docs/current/ddl-rowsecurity.html} The advisory filter can still improve index selectivity, but it isn't the security boundary.

Why is user_id filtering more than a relevance optimization in a multi-tenant memory system?

Answer

It's a security and privacy boundary. Without strict per-user filtering or namespaces, the agent can retrieve another user's memories even if they're semantically relevant. Shared company knowledge can be global, but personal history and preferences must be isolated.

Memory lifecycle and untrusted recall

Memory systems retain user and project data beyond one request. Store only information needed for the declared purpose. Attach source and expiry metadata, then propagate deletion or correction to search projections, summaries, and caches. Access logs should show which scoped records were retrieved without duplicating sensitive content into unrestricted telemetry.

Retrieved text is also untrusted input. A prior transcript can include instructions such as "ignore CI and merge main," whether written maliciously or merely quoted by a user. OWASP identifies prompt injection as a central LLM-application risk; persistence makes an unsafe instruction available in later turns unless the host filters and bounds it ^{[18]Reference 18OWASP Top 10 for Large Language Model Applicationshttps://genai.owasp.org/llm-top-10/}. Retrieved memory may inform a response, but it can't change system instructions, permissions, or approval state.

The next gate asks three questions before prompt assembly: does this record belong to the tenant, is it still valid, and did it pass write-time review? It admits only records labeled as facts or preferences. Content classification can miss attacks, so these labels reduce exposure but don't replace the separate action-authorization boundary.

Latency vs. consistency

Memory retrieval adds latency to every agent turn. Because an agent must assemble its complete context before generating the first token, synchronous memory operations can become a bottleneck.

Production systems usually separate memory operations by whether the current response must wait for them:

These are dependency classes, not universal millisecond guarantees. Set explicit budgets from the model time to first token (TTFT) target, network hops, store percentiles, and fallback policy.

When a directly stated preference matters to the next response, first commit a sourced current value to the authoritative profile store, then refresh any hot projection needed for the next turn. Expensive summary or candidate-fact extraction can remain asynchronous. A delayed search projection must never replace an authoritative record or silently override a correction.

A developer just changed a repository's preferred verification command from pnpm test to pnpm test --run billing. Which path should update synchronously: hot, warm, or background?

Answer

An authoritative scoped project profile should update synchronously from the developer's directly stated preference, and a hot/core projection can refresh for the next response. A warm search projection may update for cross-session recall. The command can shape verification, but it doesn't authorize a merge.

How should the memory system handle conflicting facts about the same preference?

Answer

Keep source, timestamp, scope, and confidence. User-stated facts usually outrank inferred facts, newer facts often outrank older ones, and source-of-truth systems outrank free-text memories. If conflict remains, the agent should surface the uncertainty instead of silently choosing.

Common mistakes and how to fix them

Memory failures become easier to diagnose when each symptom points to one broken boundary.

Which mistake is most likely if an agent retrieves too many memories per turn?

Answer

Memory noise. The model sees irrelevant, stale, or conflicting context and may overfit to the wrong memory. Tight top-k limits, metadata filters, reranking, and source-aware conflict handling are as important as storing the memories.

Design exercise: place each record

Use the same billing service to test the boundaries before choosing a database. You can complete this exercise without writing a full system.

• Scenario: You're building a coding agent for the billing webhook service. A developer returns every few days with the same billing_event_78291 class of issues. The agent needs to remember three categories of information:

1. Repository owner team and preferred verification command.
2. History of each specific PR or migration attempt (commits, test failures, review comments, outcomes).
3. General patterns the agent has noticed, such as a webhook handler that allocates a new invoice row on every POST.

Question: For each of the three categories above, which memory tier (working, core, episodic, semantic, procedural) is the best fit, and why?

Solution sketch

1. Repository ownership belongs in an authoritative metadata lookup and may be projected into core memory when needed. A stated verification command can be scoped, sourced, and pinned for response tailoring, but it doesn't authorize a merge.
2. The history of each PR or replay attempt is a specific access-controlled event record. It belongs in episodic memory and is retrieved only inside the repository or user scope when relevant.
3. A recurring failure pattern is a candidate aggregate insight. It belongs in reviewed semantic memory only after validation against source evidence, not because the agent noticed it in one transcript.

Extension: If the developer says "Same failure as last migration," how should the agent retrieve the right episodic memory? It should encode the query, search the episodic store with a recency boost, and verify the retrieved PR or test date before presenting it. Without recency weighting, an older but semantically similar retry from six months ago might outrank last week's replay of billing_event_78291.

In the practice scenario, which memory store should never be shared across repositories or users?

Answer

Episodic PR history and private developer preferences must stay per-repository or per-user. Shared retrieval can hold reviewed engineering standards and public product documentation. Aggregate insights need a separate privacy and quality review before they become shared knowledge.

The design rules to carry forward

If you retain only a few rules, keep these:

1. Agent memory organizes into three tiers: short-term (context window), medium-term (core memory of must-keep facts), and long-term (episodic, semantic, and procedural stores across external databases).
2. Working memory and KV cache are different layers. The context window is what the model reasons over, while the KV cache is an inference optimization that can be evicted and doesn't provide semantic persistence.
3. MemGPT-style paging expands usable context, not trust. A model may request memory movement, while the host validates scope, source, and retention. The FIFO's first slot is a recursive summary of evicted turns, not extra trusted instructions.
4. Ranking isn't conflict resolution or authorization. Similarity, recency, and importance rank eligible candidates only after tenant scope, provenance, and supersession checks.
5. Compression is lossy. Summaries and extracted candidate facts reduce prompt load but require a path back to source records when details matter.
6. Production systems need isolation, lifecycle controls, and checkpoints. Scoped stores prevent leakage, deletion and correction prevent stale persistence, and durable workflow state supports safe resume.