User Memory System

A user memory system is indispensable for building an AI Agent that offers truly personalized, continuous service. Memory is not a transcript of everything a user says. We don’t remember the raw content of every conversation with a friend either; through repeated interaction we gradually form a vivid mental model of them—their hobbies, habits, and values—and that model lets us understand and even predict what they need.

At its core, a user memory system is an active, continuous learning process aimed at building a concise, effective predictive model of the user. It uses additional compute—dedicated LLM calls that analyze, summarize, and structure—to explicitly extract and compress the key information scattered through long conversation histories. The contrast with in-context learning is sharp: user memory is persistent and reviewable; in-context learning is temporary and vanishes when the session ends.

Let’s understand this process with a concrete example. Suppose a user and an Agent have the following conversation:

User: Help me book a flight to Tokyo next Friday. I prefer window seats
      and I'm vegetarian, so I'll need a special meal.
Agent: I'll search for flights to Tokyo for next Friday...
       [calls flight_search tool, returns 3 options]
Agent: Here are your options. Based on your preference, I've filtered for
       window seat availability. Shall I book the ANA direct flight?
User: Yes, and use my United MileagePlus number 12345678.

After this conversation ends, the Agent framework calls a dedicated LLM to analyze the dialogue and extract information worth remembering long-term:

Extracted memories:
- User prefers window seats (preference)
- User is vegetarian, needs special meals on flights (dietary restriction)
- User's United MileagePlus number: 12345678 (loyalty program)
- User has travel plans to Tokyo (recent activity)

Note several key characteristics of this extraction process: Selectivity—the Agent won’t remember transient information like “the search returned 3 options,” only facts useful for the future; Abstraction—“I prefer window seats” is refined into a general preference, not tied to this specific flight; Structure—each memory is tagged with a type (preference, restriction, account number) for easier retrieval later. The next time the user books a flight, the Agent won’t need to ask about seat preference or meal requirements—this information is already in memory.

Evaluating Memory Capabilities: A Three-Level Framework

Before designing a memory system, first answer one question: what makes a memory system “good”? Setting the evaluation criteria up front gives us a common yardstick for every design discussed later. Several public benchmarks exist; a representative one is LoCoMo (Long-term Conversational Memory; Maharana et al., 2024, arXiv:2402.17753). It constructs ultra-long dialogues averaging about 300 turns across up to 35 sessions, and probes a model’s memory and understanding of long-range conversation through three task families: question answering (subdivided into single-hop, multi-hop, temporal reasoning, open-domain, and adversarial questions), event summarization, and multimodal dialogue generation.

Drawing on LoCoMo and its peers, together with the practice of commercial memory products, user memory capabilities can be distilled into eight categories (the author’s synthesis, not any single benchmark’s original taxonomy):

• Personal Information Retention: Remembering long-term personal information like user identity
• Preference Tracking: Tracking and remembering the user’s long-term preferences
• Context Switching: Maintaining coherence when switching between multiple topics
• Memory Update: Correctly handling new information that contradicts old information
• Multi-Session Continuity: Maintaining knowledge across sessions
• Complex Reasoning: Reasoning across multiple memory fragments, e.g., proactively reminding a user with a peanut allergy to watch for peanut ingredients when recommending Thai cuisine
• Temporal Awareness: Remembering dates, understanding relative time, performing time calculations
• Conflict Resolution: Identifying and handling inconsistencies between memories

Building on this, we designed a three-level evaluation framework more tailored to Agent scenarios, decomposing memory capabilities into progressive levels. This framework recurs throughout this chapter—Experiments 3-10 and 3-12 later will use it to measure how retrieval techniques improve memory capabilities.

Level 1: Basic Recall — This is the most fundamental capability of a memory system, requiring the Agent to accurately store and retrieve information that the user provides directly and that is structured and unambiguous. For example, “My membership number is 12345” should be precisely returned when needed later. This level ensures the basic reliability of the memory system and serves as the foundation for more complex capabilities.

Level 2: Multi-Session Retrieval — The Agent must retrieve and reason over all relevant information when conversations span different entities, service channels, and time periods; real-world tasks are rarely completed in a single conversation. When a user with two cars asks “Schedule maintenance for my car,” the system needs to find both cars and ask which one needs service, not guess. When the user asks about loan status, it must pick out the active contract currently in force and ignore past quote inquiries that never took effect. When canceling a “Los Angeles trip,” it must understand that a trip is a composite event and proactively link every related booking—flights and hotels alike.

Level 3: Proactive Service — This is the acid test of whether an Agent has truly reached assistant-level capability: synthesizing information across many sessions, some of them very old, to offer predictive help—finding deep connections between memories that look unrelated. When the user books an international flight, the system surfaces the passport stored months ago, notices it is about to expire, and warns them. When a phone breaks, it pulls together every protection option—the phone’s own warranty, the credit card’s extended-warranty terms, the carrier’s insurance—into one complete list. During tax season, it combs the past year’s records for every tax document (stock sales, freelance income, property taxes) and presents a full to-do list. All of this means heading off problems and integrating complex information without being asked.

The Hierarchical Structure of Memory

With evaluation criteria established, we can move to concrete design. The design of a memory system can be broken down into three independent dimensions—where to store it, how to store it, and what to store. This section addresses “where to store it.”

To enable the Agent to efficiently handle current tasks while providing personalized service across sessions, memory needs to be divided into different levels—much like humans distinguish between short-term working memory and long-term memory:

Trajectory is the complete historical record of a single Agent run—corresponding to the “dynamic trajectory” defined in Chapter 1 (user messages + model replies + tool execution results, collectively called the trajectory). The trajectory records every event from the start of the conversation to the current moment, in chronological order and never rewritten—new events keep getting appended to the end, but records once written are never modified or deleted (the pattern computer science calls append-only). The trajectory provides immediate context for Agent decision-making—“what did I just say,” “how did the user respond,” “what did the tool return.”

The trajectory is the complete raw record of a single session, appended chronologically and never modified; user long-term memory, on the other hand, is stable information distilled across sessions, which is repeatedly rewritten, merged, and pruned. The former is a log, the latter is an archive.

User Long-Term Memory is persistent storage across sessions and instances, typically bound to a specific user ID via key-value pairs. It stores preference settings, historical interaction summaries, and extracted facts. The Agent explicitly reads and updates long-term memory through specific tool calls, enabling cross-session personalization and continuity.

Additionally, some Agents support Business State—high-level state abstractions defined by developers, representing the logical stage of a task (e.g., “needs clarification,” “processing request,” “awaiting payment,” “request completed”). This type of state abstraction is particularly important in event-driven Agent architectures (Chapter 4 will discuss event-driven architecture design).

This chapter focuses on the two core levels: trajectory and user long-term memory. The layered design ensures the Agent can efficiently handle current tasks (relying on trajectory) while possessing long-term personalization capabilities (relying on long-term memory).

Four Storage Formats for User Memory

Having addressed “where to store it” and “how to evaluate it,” the next question is “how to store it”—the same piece of user information can be represented with different granularities and structures. The following four storage formats represent a progression in memory granularity and structural complexity.

Simple Notes embodies a minimalist design. Each memory is a minimal, indivisible fact (e.g., “User email: john@example.com”). The advantage is minimal overhead: O(1) operations (constant time, independent of data volume). The cost is that associations between facts are lost entirely—“Works as a Senior Engineer at TechCorp, responsible for recommendation system development” is decomposed into three independent facts (“Works at TechCorp,” “Job title is Senior Engineer,” “Responsible for a recommendation system”), severing the internal connections within a single job. When handling queries that require synthesizing multiple pieces of information, the system must use heuristic rules (e.g., guessing which facts might be related based on keyword overlap) to piece the fragments back together.

Enhanced Notes adopts a holistic perspective, saving each memory as a paragraph containing complete context. For example, the same job information is stored as: “The user has been a Senior Software Engineer at TechCorp, specializing in machine learning for three years, currently leading a recommendation system project with a team of five.” Preserving the narrative structure keeps the semantics complete and rich—well suited to scenarios that call for nuanced understanding (e.g., “Recommend a new project based on my background,” which requires inferring skill level, leadership experience, and technical preferences).

The costs are threefold: storage redundancy (the same information repeated across paragraphs), update complexity (one attribute change means rewriting several paragraphs), and paragraphs long enough to hurt later retrieval. The reason for the last cost is simple: when text must be converted into a form computers can search, the longer the paragraph, the harder it is for a vector embedding to capture its core meaning—just as a book’s blurb gets harder to grasp the longer it runs (the technical details of embeddings and retrieval come in this chapter’s RAG section).

JSON Cards adopts a three-level nested structure (Category → Subcategory → Key-Value Pair, e.g., personal.contact.email, work.position.title), mimicking the way humans categorize. It supports partial updates (modifying work.position.title does not affect work.company.name) and is predictable and extensible. But the rigid structure assumes information can be cleanly categorized—“Developing personal projects in Python on weekends” is at once a time preference, a technical preference, and an activity type; forcing it into a single category flattens those dimensions away.

Advanced JSON Cards represents a paradigm shift in memory system design—from information storage to knowledge management. Each card records not only facts but also the narrative context (backstory) of the information source, the subject’s identity (person), the relationship with the user (relationship), and a timestamp. The core idea is that the same piece of information can have completely different meanings in different contexts—“Dr. Zhang” could be the user’s own dentist or the user’s father’s cardiologist; stripped of its context, the information cannot be understood correctly.

This design solves the disambiguation problem of traditional systems. In real-world scenarios, a user may have multiple doctors (for themselves, their parents, their children), and simple key-value storage cannot accurately distinguish them. Advanced JSON Cards provide the context in which the information was acquired (the “why” for storing this information) through backstory, and establish a clear entity model (the “for whom” the information is stored) through the person and relationship fields. When the user says “Help me arrange annual checkups for my family,” the system can identify all family members through relationship and understand health history through backstory. The cost is higher generation and maintenance overhead.

Comparing these four modes reveals a fundamental tension in memory system design: the trade-off between simplicity and expressiveness. Simple Notes chooses extreme simplicity at the cost of semantic completeness; Enhanced Notes chooses narrative completeness at the cost of structure and updatability; JSON Cards chooses structure at the cost of flexibility; Advanced JSON Cards chooses comprehensiveness at the cost of simplicity. This trade-off has no absolute winner—it depends entirely on the specific use case. A mature AI Agent system may need to use a mix of modes: Simple Notes for quickly recording transient information, and Advanced JSON Cards for handling critical information that requires precise disambiguation and long-term maintenance.

The practical selection criterion is: use Advanced JSON Cards for critical, low-volume data (e.g., user preferences, key personal relationships) to ensure retrievability; use Simple Notes for large volumes of non-critical conversational facts to reduce cost. Most production systems adopt a hybrid approach—different types of information within the same Agent follow different paths.

Advanced Representation: From Executable Code to Parametric Memory

The four formats discussed above, whether simple or complex, are fundamentally text—meaning that the “storage” and “use” of memory remain two separate steps: first retrieve the relevant text, then feed it to an error-prone LLM to read and compute. Text-based memory excels at recalling individual facts but struggles with aggregating statistics across many records, detecting contradictory facts, or enforcing logical rules, because all these operations rely on the LLM’s “mental arithmetic.” User as Code proposes a solution: shift the representation medium from text to executable code. It treats the Agent’s model of the user as a living software engineering project—using typed Python objects to store user state and ordinary Python functions to encode constraint rules, so that “representing the user” and “reasoning about the user” happen in the same medium that can be executed by an interpreter.

It splits memory updates into two phases: the memory phase (after each session, the LLM extracts facts from the conversation one by one as strings, appending them to an append-only fact log) and the structuring phase (periodically, the LLM regenerates the entire typed Python representation from the complete fact log—organizing facts into dataclasses, using date() for dates, typed lists for collections, and notes: list[str] for miscellaneous items that are hard to type). This is the classic “write-ahead log + periodic checkpoint” design from databases, applied to LLM memory for the first time: the append-only log ensures no facts are lost, and the periodic checkpoint compresses them into a clean, queryable structure. (This periodic reconstruction process is consistent with the “memory compression and organization mechanism” discussed later in this chapter, except the output is code rather than text.)

Below is a simplified example. The structuring phase stores the user’s passport and trips as typed state:

from datetime import date

passport = PassportInfo(
    number="AB1234567", country="US",
    expiry_date=date(2025, 2, 18),
)
trips = [
    Trip(destination="Tokyo", departure_date=date(2025, 1, 15),
         is_international=True),
    # ... remaining trips
]

With typed state, three tasks that previously required the LLM to “read the text and do mental arithmetic” now become deterministic code:

First, statistical aggregation. “How many times did I go abroad in 2025?”—with text memory, you’d need to recall all trips and count them one by one, and accuracy drops as the number of records grows (the paper reports that retrieval-based memory achieves only 6%–43% accuracy on such aggregation problems); with User as Code, it’s a single expression, achieving nearly 99% accuracy:

>>> sum(1 for t in trips if t.is_international and t.departure_date.year == 2025)
2

Second, conflict detection. By placing “current medications” and “allergy history” side by side, a single function can cross-reference them by drug class, uncovering contradictions scattered across different conversations that would be nearly impossible to automatically associate in text form:

def check_drug_allergy(profile):
    for med in profile.current_medications:
        for allergy in profile.allergies:
            if med.drug_class == allergy.drug_class:
                yield (f"Medication conflict: {med.name} belongs to {med.drug_class} class, "
                       f"but the patient is severely allergic to {allergy.allergen}")

Third, constraint enforcement. The Agent can codify such check functions and trigger them automatically every time the state is updated—without the user needing to speak or the Agent needing to retrieve anything. For example, a passport validity constraint: alert if the passport expires less than 180 days after the departure date of an international trip.

def check():
    for trip in trips:
        if trip.is_international:
            days = (passport.expiry_date - trip.departure_date).days
            if days < 180:
                yield (f"Passport expires on {passport.expiry_date}, only {days} days "
                       f"between the {trip.destination} departure and passport expiry. "
                       f"Please renew as soon as possible.")

The same passport expiry date is both stored and available for computing how many days remain between trip departure and passport expiry—the arithmetic is done by a deterministic interpreter, not the LLM, so the Agent can warn “your passport is about to expire” before you even ask. Aggregation, conflict detection, and hard constraints are exactly where text memory struggles most and code excels. The cost is the engineering scaffolding for code generation and execution, and code offers no advantage for loosely structured miscellany—hence the notes field still keeps a place for text.

User as Code advances memory from text to executable code, but like the text formats before it, it remains an external store outside the model—the model must first retrieve it and then reason over it in context. Pushing further inward along this representation spectrum, user memory can also be written directly into the model’s own parameters, leading to two more cutting-edge forms.

Writing into Local Parameters: User as Engram. A natural idea is to write user facts directly into the model weights—for example, training a dedicated LoRA for each user. But this path encounters a puzzling obstacle: such fact-LoRAs can almost perfectly reproduce facts when asked directly, but fail when the model must reason indirectly over those facts—because the frozen backbone model never learned how to “consult” such a temporarily attached adapter. In other words, storing facts is one thing; making the model know when to retrieve them is another. User as Engram addresses precisely this: it does not train a LoRA, but instead precisely writes a user fact into an empty hash N-gram slot in the Engram model. Such models learn during pre-training to retrieve memories via hash table lookups, controlled by a context-aware gating mechanism; thus, newly written facts are naturally recalled when they should be, bypassing the “stored but not used” dilemma. Facts from different users fall into disjoint slots and can be stacked on top of one another (just as multiple Stable Diffusion LoRAs can be plugged in and combined)—without crosstalk between users and without touching the backbone model itself.

Multimodal: Storing Ineffable Perceptions. So far, everything stored has been facts that can be written as discrete symbols. But user memory also has a perceptual half—a face’s appearance, a voice sounding more tired today than last week, an artist’s brushstrokes across different periods—none of these is fully preserved when transcribed into text: when you write “a brown-haired man,” you lose precisely the subtle signals that distinguish two brown-haired men. The idea behind Parametric Multimodal User Memory is to preserve perception in its perceptual form: attach a small memory bank to a frozen model, where each identity to be remembered corresponds to one row—the key is a perceptual vector computed by an off-the-shelf encoder (ArcFace for faces, CLIP for art styles), and the value is the embedding of a token from the model itself (e.g., <id_11>). During generation, the current perception serves as a query, performing attention computation over this memory bank, gently steering the output toward the matching token—all without any text. Registering a new identity requires only adding a row to the bank, no training needed. Most intriguingly, perceptions stored this way not only match the effectiveness of direct vector retrieval but exceed it—because matching happens in the language model’s own representation space, it can be more discriminating than the encoder’s native similarity, precisely compensating for the encoder’s weakest and most error-prone step.

From plain text to executable code to local parameters and even continuous perception, user memory representations form a spectrum running from “outside” the model to “inside” it: the outer layers are easy to update, audit, and migrate; the inner layers are more compact, quicker at in-the-moment reasoning, and able to represent perceptions that words cannot capture. The two inward paths touch on Chapter 7’s parameter fine-tuning and Chapter 9’s multimodality, respectively—here they are only a preview.

Cognitive Science Foundations of User Memory

Having seen four concrete memory strategies, we now borrow a framework from cognitive science to examine another dimension of memory: the types of content it stores.

From a cognitive science perspective, the complexity of the human memory system offers important insights for AI memory design. Cognitive science divides memory into Working Memory and Long-Term Memory. Working memory corresponds to the Agent’s context window—a temporary information space for handling the current task (the trajectory is the core content of working memory, but working memory may also include information activated and loaded from long-term memory). Long-term memory is further divided into three types, each with a direct counterpart in Agent memory:

• Episodic Memory: Memory of specific events and experiences. Human example: “I had a great dinner with colleagues at that Italian restaurant last Wednesday.” Agent counterpart: In the earlier flight booking example, “The user booked an ANA flight to Tokyo next Friday”—recording the time, object, and details of a specific event.
• Semantic Memory: General knowledge abstracted from specific events. Human example: “The capital of Italy is Rome.” Agent counterpart: “The user is vegetarian,” “The user prefers window seats”—these are not records of a single conversation but stable features distilled from multiple interactions.
• Procedural Memory: Memory of behavioral patterns and procedures. Human example: The ability to ride a bicycle. Agent counterpart: A general procedure learned from the user’s repeated flight booking patterns—“First search for direct flights → confirm seat preference → use frequent flyer number → order a meal.”

Looking back at the content of this section, we have introduced three classification systems. To avoid confusion, Table 3-1 clarifies their relationships at a glance:

Table 3-1 Three Classification Systems for Memory Design

Classification System	Question Answered	Specific Categories
Memory Hierarchy (beginning of this chapter)	Where is it stored?	Trajectory (current session), User Long-Term Memory (cross-session), Business State (task stage)
Storage Format (section “Four Storage Formats”)	How is it stored?	Simple Notes, Enhanced Notes, JSON Cards, Advanced JSON Cards
Cognitive Type (this section)	What is stored?	Episodic Memory (specific events), Semantic Memory (general knowledge), Procedural Memory (behavioral procedures)

The three systems are orthogonal dimensions—they can be freely combined. For example, a semantic memory like “the user prefers window seats” can be stored in Simple Notes format within user long-term memory; a procedural memory like “first search for direct flights → confirm seat → use frequent flyer number” can be stored in Advanced JSON Cards format. The choice of format depends on engineering needs (simplicity vs. expressiveness), and the choice of what type to store depends on the business scenario (whether you need to remember facts, events, or procedures).

Memory Framework Case Studies

The storage formats and memory types discussed above must eventually be implemented in working code. The open-source community has produced several dedicated memory management frameworks; Mem0 and Memobase illustrate how two different design philosophies make their trade-offs.

Mem0: An Extract–Compare–Decide Two-Stage Pipeline. At its core, Mem0 (Chhikara et al., 2025, arXiv:2504.19413) operates an “extract–compare–decide” memory pipeline that runs in two stages.

Extraction Stage: Whenever a new conversation segment ends, Mem0 calls an LLM with the recent dialogue and summaries of existing memories to extract a set of candidate memories—concise factual statements such as “The user moved to Shanghai.” Update Stage: For each candidate memory, the system first uses vector retrieval to find semantically similar existing memories. The LLM then compares the relationship between the candidate memory and the retrieved memory and makes one of four decisions—ADD (completely new information, directly stored), UPDATE (supplement or correct an existing memory), DELETE (new information contradicts an old memory, delete the latter), or NOOP (duplicate information, take no action). For example, when a user says “I moved to Shanghai,” Mem0 retrieves the existing memory “The user lives in Beijing,” determines this is an UPDATE, and updates the old memory to “The user lives in Shanghai,” rather than retaining two contradictory records. This pipeline unifies the “selective extraction” described at the beginning of this chapter and the “conflict resolution” to be discussed later into a single mechanism—every record in the memory store has undergone explicit reconciliation with existing memories.

Engineered for adaptability, Mem0 uses a highly modular architecture to suit different application needs: embedding (converting text to vectors) and storage (persistence and retrieval of vectors) are separated, allowing independent optimization and replacement of each. It supports multiple backends through abstract interfaces, and a plugin mechanism enables flexible integration of new language models, embedding models, or storage backends. Beyond the basic version, Mem0 also offers a graph memory variant, Mem0-g: it represents memories as an entity-relationship graph rather than independent factual entries, explicitly capturing the relational structure between memories. This improves performance on multi-hop and temporal problems (the knowledge representation of graph structures will be discussed in detail later in this chapter in the GraphRAG section).

Memobase: User Profiles Plus Event Memory. Memobase (open-source project memodb-io/memobase) has a different design philosophy from Mem0: rather than building a general-purpose memory pipeline, it focuses on the specific form of “user profiles.” It organizes user memory into two parts. User Profile is a set of configurable slots organized by topic and subtopic (e.g., basic_info→name, interest→gaming preferences, work→job title), storing stable user attributes extracted from conversations. Developers can precisely control the scope and granularity of the profile. Event Memory records user experiences along a timeline, used to answer time-related questions like “When did we last discuss the budget?” On the engineering side, Memobase uses buffered batch processing: conversations accumulate until a size or time threshold triggers one memory-extraction pass. This amortizes the cost of LLM calls, and since the query side reads only the already-organized profiles and events, latency stays low.

Each framework covers only part of the memory design space: Mem0’s factual entries are close to semantic memory, while Memobase’s profiles approximate semantic memory and its event memory approximates episodic memory. Widening the lens, we can sketch a reference architecture for multi-type memory collaboration built on the cognitive science categories introduced earlier—a generalization of the design space rather than any particular project’s implementation:

• Episodic / Semantic / Procedural Memory: The episodic, semantic, and procedural categories follow the three cognitive science categories defined earlier; the human and Agent examples need not be repeated here. What this reference architecture genuinely adds is the multi-dimensional metadata retrieval for episodic memory—it stores event sequences with rich metadata (timestamps, emotional markers, task identifiers), enabling combined retrieval across multiple dimensions like time and topic (e.g., “When did we last discuss the budget?”).
• Working Memory: In addition to the three types of long-term memory, the reference architecture explicitly retains a working memory layer (its concept was introduced earlier), managing the current task state and dynamically interacting with long-term memory—important information is selectively transferred to long-term memory, and relevant long-term memories are activated and loaded into working memory.

A special note is needed on the relationship between working memory and the “trajectory” mentioned in the earlier “Hierarchical Structure of Memory”: both provide immediate context for current decisions, but a trajectory is an immutable complete event sequence (appended over time), whereas working memory is a dynamic subset that has been filtered and activated (trimmed by relevance).

This reference architecture shows how cognitive science’s memory classifications can become engineering components. Practical frameworks usually implement only one or two of the types—picking what the business needs is closer to engineering reality than chasing a do-everything design.

Memory Compression and Organization Mechanisms

As interaction continues, a memory system faces the twin pressures of storage space and retrieval efficiency. Simply accumulating everything leads to unbounded memory growth—it consumes storage and drags down retrieval accuracy.

In practice, a multi-tier compression strategy works well. The first tier filters memories by importance score. A common approach to importance scoring considers four factors: access frequency (frequently retrieved memories are more important), time decay (older memories are more likely to be forgotten), emotional intensity (memories with strong emotional markers are more likely to be retained), and information uniqueness (the importance of duplicate information decreases). Memories below a threshold are marked as compressible or deletable. For example, a memory accessed 5 times, created 3 days ago, with a strong emotional marker, and no duplicates would receive a high importance score. In contrast, a memory accessed only once, created 90 days ago, with no emotional marker, and three near-duplicates might fall below the compression threshold.

The second tier performs clustering. Similar memories are grouped, and a representative summary is generated for each group (e.g., multiple weather-related conversations are compressed into “The user frequently asks about the weather, with particular concern about rain”). Original detailed memories can be archived to secondary storage.

The third tier abstracts and generalizes—extracting general rules from specific episodic memories and converting them into semantic or procedural memory. For example, from multiple shopping conversations, the system might learn “Prefers cost-effective products and values user reviews.”

Conflict detection uses a versioning approach—historical versions are retained while the latest version is marked. For certain information (e.g., current address), only the latest version is kept; for other information (e.g., work history), the complete history is retained.

Finally, a boundary must be drawn to avoid confusion with other chapters. This section discusses organization algorithms at the memory storage layer—which memories to select, cluster, and abstract, and into what forms. Context compression in Chapter 2 addresses the window problem within a single session; the two mechanisms operate at different levels. This chapter is also responsible for knowledge storage, indexing, and retrieval. Chapter 8 generalizes the two-stage pattern of “append evidence online, consolidate it offline” to the evolution of Agent behavior, examining what operational evidence is sufficient to trigger persistent updates.

Privacy Protection: Log Sanitization

In building a user memory system, the core challenge is letting the Agent use personal information for personalized service without exposing sensitive data in the LLM context or system logs.

So far we have focused on the representation and management of memory—what format to store it in, how to update and compress it. The next problem is retrieval: once memory grows to thousands or tens of thousands of entries, how do we quickly find the relevant few? This is precisely what RAG solves—first for shared knowledge bases and, as we will see at the end of this chapter, for user memory retrieval as well.

Engineering Practice

Zapvol’s memory lives behind a narrow MemoryStore port: memories are stored as .md + YAML frontmatter (human-readable, agent-writable, greppable), with Desktop using the filesystem and the server using R2 / DB — same service logic, only the store backend swapped. It is independent of the per-task sandbox and isolated per user (openUserMemoryStore({ userId })). Only the first of the three tiers is implemented today (Auto-Memory — cross-session, per-user); Session / Team memory are planned.

What to store is four types — user (role, preferences), feedback (corrections or confirmations of how to work), project (project context not derivable from code), reference (pointers to external resources) — and explicitly excludes the code-derivable, the temporary, and the already-recorded; the key to preventing bloat is what not to store.

Writing is two mutually-exclusive channels: explicit (save_memory, when the user says “remember this” or the agent decides to) and background auto-extraction (each turn’s end enqueues a memory.extraction job that pulls structured candidates from recent dialogue with a lightweight model + Zod schema). If save_memory fired this turn, auto-extraction is skipped, preventing duplicates. Reading is also two paths: the MEMORY.md index is injected into the system prompt at startup, and at runtime the agent calls recall_memory (currently keyword match, top 5).

Related reading

• RAG Basics — how retrieval scales when memories grow to thousands of entries
• Knowledge Organization and Retrieval — the file-system route Zapvol actually uses