Problem Formalization
The core failure is not that LLMs lack memory — it is that the dominant deployment pattern conflates the context window with a storage layer. These are fundamentally different abstractions. A context window is a compute primitive evaluated in a single forward pass; it does not persist, cannot be indexed, and imposes a cost linear in its token count. Treating it as persistent storage produces three distinct failure modes that compound in production.
1.1 Context Inflation
For a system handling 10,000 calls per day with an average 300-turn session history, naive full-history injection generates roughly 60,000 tokens of input per request. At frontier model pricing, this translates to approximately $0.90 per request, $9,000 per day, and over $3 million per year — for memory alone. Structured memory injection reduces the injected context to roughly 800 tokens: a 98.7% cost reduction.
| Approach | Tokens per request | Daily cost (10k calls) | Annual cost |
|---|---|---|---|
| Naive full history | ~60,000 | ~$9,000 | ~$3,285,000 |
| Structured memory injection | ~800 | ~$120 | ~$43,800 |
Token costs are the visible failure. The invisible failure is latency: processing a 60,000-token context for every turn adds 200–800ms of prefill latency, which is fatal for voice agents targeting sub-500ms first-token time.
1.2 Temporal Conflation
A flat append-only transcript presents conflicting facts without resolution semantics. Consider a user who states "my budget is ₹50 lakhs" in session 3 and "I got an approval for ₹80 lakhs" in session 9. A flat history contains both statements. An LLM instructed to "answer based on the context" must guess which is current. Empirically, models hallucinate a resolution approximately 34% of the time on temporal-reasoning questions in LoCoMo (Category 2). The correct answer requires a supersession relation: the later fact invalidates the earlier one, and the system must make this explicit.
1.3 Retrieval Latency vs. Voice SLA
Production vector databases (Qdrant, Pinecone, Weaviate) on managed cloud infrastructure return results in 50–300ms per query, including the embedding generation step. A voice agent processing one turn every 3–7 seconds has a hard first-token latency budget of 500ms. A synchronous RAG call consuming 50–300ms of that budget — before LLM generation begins — exceeds acceptable bounds at the p95. The solution is temporal decoupling: move retrieval off the critical path by pre-loading stable facts at session start and reserving on-demand search for genuinely ambiguous turns.
Prior Work and Design Space
Five architectural approaches dominate the current literature. Each addresses a subset of the three failure modes and introduces distinct tradeoffs. We characterize them here not to rank them but to identify the specific gap the MSOCA memory layer fills.
| System | Write Model | Temporal Awareness | Solves Inflation | Voice-SLA Compatible | LoCoMo F1 |
|---|---|---|---|---|---|
| Mem0 | LLM-issued ADD/UPDATE/DELETE | Creation timestamp only | Partial | No (sync write) | 34.20% |
| SimpleMem | Semantic compression to ACUs | Epoch-based | Yes | Partial | 43.24% |
| Zep (Graphiti) | Temporal knowledge graph | Validity windows + supersession | Yes | No (graph queries) | N/A |
| MemGPT | OS-style paging (FIFO) | None | Yes | No (LLM-driven paging) | 38.9%* |
| ReadAgent | Gist-based episode summarization | None | Partial | No | 55.1%* |
* LongMemEval benchmark; other LoCoMo figures from LoCoMo-10 leaderboard. Zep omitted from LoCoMo as no published benchmark run exists.
What Prior Work Gets Wrong
LLM-as-Controller (Mem0) — Write Amplification
Issuing ADD/UPDATE/DELETE decisions on every turn means the same LLM that generates responses also manages persistent storage. This creates write amplification: every turn triggers an LLM call to classify memory operations, even when the turn contains no new facts (a user saying "okay" or "thanks" should produce zero writes). In practice Mem0's construction latency is 14.6× higher than SimpleMem (1,350.9s vs. 92.6s on LoCoMo-10) precisely because it issues an LLM call per-turn unconditionally.
Epoch-Based Systems (SimpleMem) — No Supersession
SimpleMem achieves the best LoCoMo F1 (43.24%) by compressing aggressively, but it does not model fact lifecycles. If a user's budget changes three times across sessions, all three values are indexed as separate ACUs with no relation between them. At retrieval time, a semantic query for "current budget" returns all three, and the LLM must infer recency from timestamps — which requires reasoning SimpleMem's prompt does not instruct. The result is temporal-category questions (LoCoMo Category 2) scoring significantly below single-hop recall.
Graph-Based Systems (Zep) — Query Latency
Zep's Graphiti engine is the correct semantic model for temporally evolving facts: every node carries a validity start, validity end, and relationship edges that model supersession explicitly. The tradeoff is query latency: a temporal graph traversal returns results in 100–400ms on production workloads. This is acceptable for chat agents but exceeds the voice SLA. The MSOCA memory layer borrows Zep's temporal data model but stores it in a document layer — accessed as a pre-loaded array, not a graph query — achieving sub-5ms Tier 1 reads.
System Architecture
The memory service exposes a narrow HTTP API to the agent. Internally it maintains three storage tiers: a hot document store for low-latency reads, a vector index for semantic ACU retrieval, and a cold archive for verbatim turn history. An optional real-time session buffer handles voice agents, where turn logging must be non-blocking. All memory orchestration is fully invisible to the agent loop.
3.1 Storage Tiers
| Tier | Contents | Read Latency | Write Pattern |
|---|---|---|---|
| 1 — Hot | Dynamic profile, active temporal entities, recent history, summaries | <5ms | Synchronous on turn append; async update after extraction completes |
| 2 — ACU Archive | Atomic Context Units indexed by semantic embedding; searchable by meaning | p95 <50ms | Async; near-duplicate suppression before indexing |
| 3 — Cold | Raw turn history evicted from the hot store; verbatim content for audit and fallback | 50–500ms | Async; batch-written when compression windows are archived |
3.2 The Clean API Contract
From the agent's perspective, memory is exposed through three narrow operations. All orchestration complexity — extraction, indexing, supersession, and retrieval — is invisible to the consuming agent:
| Operation | What the agent receives | What happens behind the scenes |
|---|---|---|
| Session start | Pre-formatted profile, active facts, recent history, and a memory search tool definition | Single low-latency document read; no vector search; no embedding call |
| Turn append | Immediate acknowledgment (<10ms) | Turn is persisted synchronously; background extraction is triggered asynchronously when accumulated content exceeds a threshold |
| Memory search | Ranked list of relevant past facts with a confidence indicator | Semantic vector search, optional hybrid keyword ranking, cold storage fallback if confidence is low |
3.3 Request Flow: Turn Append
Every turn follows a strict two-lane design. The critical path — the path the agent waits for — does only the minimum work to persist the turn and returns immediately. A separate background lane handles all intelligence: extraction, compression, indexing, and supersession. These two lanes never block each other. The result is a turn append that completes in under 10ms regardless of how long background processing takes.
The Extraction Pipeline
When accumulated conversation tokens exceed the configured threshold (default: 4,000 tokens), the background extraction pipeline fires. The pipeline acquires a per-user distributed lock — preventing duplicate extraction runs for the same user if multiple sessions overlap. It runs four stages in a specific dependency order.
4.1 Pipeline Stages
The pipeline runs four stages when accumulated conversation content crosses the configured threshold. Stages 1 and 3 run in parallel — extraction and compression are independent and each requires its own LLM call. Stage 2 runs after Stage 1 completes, since profile merging and temporal supersession depend on the extraction results.
Stage 0 — Smart Keep
Each turn in the hot history is scored using a weighted combination of recency, whether it contains a question, and how many named entities or facts it mentions. The top-scoring turns are kept in the hot document; the rest form the compression window that gets processed. A minimum number of turns are always retained regardless of score, ensuring the agent is never left without immediate conversational context.
Stage 1 — Schema-Driven Extraction
An LLM processes the compression window using the agent's configured extraction schema, producing structured facts. Each extracted field is categorized as a stable profile field, a time-bounded temporal entity, or a short-lived ephemeral fact. This categorization drives how the fact is stored and how long it remains visible in context.
Stage 2A/2B — Profile Merge and Temporal Supersession
Profile fields overwrite previous values with a last-write-wins rule. Temporal and ephemeral facts go through the supersession process: if a fact with the same entity key already exists and the value has changed, the old entity is marked as superseded and the new one is inserted as active. Both old and new records are preserved in full.
Stage 3 — ACU Compression
The compression window is split into small segments. For each segment, an LLM generates a list of Atomic Context Units — self-contained, standalone facts. Each ACU is embedded and compared against the existing vector index; near-duplicates are suppressed before indexing. The raw turns are then archived to cold storage and removed from the hot document.
4.2 Smart Keep: Scoring Function
The Smart Keep phase determines which turns survive the compression window and remain in the hot context
A minimum number of turns is always retained regardless of score — this preserves at least one conversational exchange before the extraction window, preventing the model from losing immediate context when the first extraction fires.
4.3 Schema-Driven Extraction
Every agent is configured with an extraction schema — a structured definition of exactly what facts to extract from conversations. Each field in the schema carries a category: stable profile facts that are overwritten on update, time-bounded temporal facts that are tracked through supersession, or short-lived ephemeral facts with an automatic expiry window. The same extraction logic runs for every agent; the schema is the only agent-specific input, meaning new domains can be onboarded without any changes to the extraction pipeline itself.
Schema portability: A rental verification agent extracts property address, expected rent, and verification status. A healthcare agent extracts symptoms, medications, and appointment dates. Zero engineering changes — schema changes require zero deployments.
4.4 Parameter Sensitivity
| Parameter | Effect of Increasing | Effect of Decreasing |
|---|---|---|
| Extraction token threshold | Fewer extractions → lower LLM cost; profile staleness increases | More extractions → fresher profile; higher cost and lock contention |
| Minimum kept turns | More hot context survives → better coherence; larger hot document | Smaller hot document; risk of losing question-answer pairs at boundary |
| ACU segment size | Larger segments → fewer LLM calls; coarser ACU granularity | More granular ACUs; higher extraction LLM cost |
| Context window turns | More recent turns returned at session start | Fewer turns; faster response; risk of losing recent context |
| Ephemeral TTL | Ephemeral facts visible longer across sessions | Faster expiry; ephemeral facts disappear before next session |
Temporal Supersession: The Bi-Temporal Model
Temporal entities are the core data model for facts that change over time. The design borrows the bi-temporal pattern from database systems: every entity carries two independent time axes, which prevents a class of audit and correctness bugs that single-timestamp designs cannot handle.
5.1 Two Time Axes
| Field | Meaning | Set By |
|---|---|---|
| Valid from | When this fact became true in the real world (world time) | Inferred from the conversation turn; falls back to turn timestamp |
| Valid until | When this fact was superseded in the real world; empty if still active | Set when a newer conflicting entity is inserted |
| Recorded at | When the system first ingested this fact (transaction time) | System clock at extraction time |
| Status | Active or Superseded lifecycle state | Active on creation; Superseded when a newer value replaces it |
The distinction matters in practice. A user who updated their budget on January 15th but whose data was only extracted on January 18th has a world-time start of January 15th and a recorded time of January 18th. A single-timestamp design cannot represent this — it would record January 18th as both the fact's validity start and the recording date, making temporal queries over world-time incorrect.
5.2 How Supersession Works
When Stage 2B detects that a newly extracted entity conflicts with an existing active one — same fact type, different value — it performs a two-step operation. The old entity is marked as superseded and its world-time end is recorded. The new entity is inserted as active, with its own validity start and the time the system recorded it. Both records remain in the document permanently; session start surfaces only active entities, while audit queries have access to the full history.
What we deliberately chose not to do: An earlier design deleted the old entity on supersession. This failed silently when an extraction model hallucinated a value update — the true history was unrecoverable. The append-only supersession model means any bad write can be corrected by re-inspecting the full timeline.
5.3 Source Confidence Hierarchy
Every temporal entity carries a confidence source that classifies how the fact was obtained. This is used to separate facts by confidence at session start — high-confidence facts and inferred possibilities are injected into the model under distinct contextual labels, so the model knows which facts to assert and which to surface as possibilities:
| Source | Meaning | LLM Behavior |
|---|---|---|
| explicit | User directly stated the fact ("My budget is 80L") | Treat as ground truth; do not hedge |
| confirmed | User confirmed a system-suggested fact ("Yes, that's correct") | Treat as ground truth |
| inferred | Extracted from indirect signal ("I'm looking at properties near the school" → location inference) | Surface as possibility; do not assert |
5.4 Ephemeral Entities
A fourth category — ephemeral — applies to time-bounded facts: appointment times, short-term availability, session-specific intent. Ephemeral entities share the same bi-temporal model but are hidden from context after their TTL expires (default: 24 hours). Crucially, they are hidden — not deleted. They remain searchable in cold storage for audit purposes and can be recovered if the expiry was set incorrectly.
ACU Compression
Atomic Context Units (ACUs) are self-contained, semantically dense facts extracted from a compression window. The word "atomic" has a precise definition here: an ACU must be fully interpretable without any surrounding context. No pronouns. No implied references. No relative time expressions. This is a hard constraint enforced during extraction — not a quality guideline.
6.1 The Atomicity Constraint
| Type | Example | Valid ACU? | Reason |
|---|---|---|---|
| Pronoun reference | "He wants a 3-bedroom flat near his office" | No | "He" and "his" require the surrounding conversation to resolve |
| Relative time | "User mentioned budget yesterday" | No | "Yesterday" is meaningless without the turn timestamp |
| Implicit subject | "Budget is 80 lakhs" | No | Whose budget? Missing subject |
| Self-contained | "Rajiv Kumar (user) is looking for a 3-bedroom flat in Whitefield, Bangalore, with a budget of ₹80 lakhs as of 2026-01-15" | Yes | Fully interpretable in isolation; contains who, what, where, when |
6.2 Segmentation and Compression
The compression window is split into small segments of up to 5 turns each. The LLM processes each segment independently, producing a list of ACUs per segment. Processing segments in parallel reduces extraction latency; the 5-turn limit keeps each segment within approximately 1,000 tokens, fitting comfortably within the extraction model's effective attention range.
6.3 Deduplication Threshold Sensitivity
The similarity threshold for ACU deduplication (0.95) was chosen after testing the range 0.80–0.99 on a held-out set of 50 user histories. At 0.80, legitimate fact updates were suppressed — semantically similar phrasings of different budget amounts were treated as duplicates. At 0.99, verbatim-duplicate ACUs from overlapping compression windows were indexed multiple times, degrading retrieval precision. The threshold of 0.95 correctly suppresses only near-verbatim duplicates while allowing distinct-but-related facts through.
Known limitation: The deduplication threshold is global, not per-field-type. Semantically close but contradictory preferences — "user prefers morning calls" vs. "user prefers afternoon calls" — both pass the threshold and are indexed independently. At retrieval time, both are returned and the model must resolve the contradiction. This is the correct behavior for temporal changes but produces noise for genuinely exclusive facts. Per-field-type thresholds are a planned improvement.
Retrieval Architecture
The retrieval design is built around a single constraint: Tier 1 (profile + temporal entities) must add zero latency to the call path. Tier 2 (ACU semantic search) must not run on every turn. We enforce this through the lazy retrieval model — the LLM receives a tool definition at session start and calls it only when it determines that the pre-loaded context is insufficient.
7.1 Session Start: Tier 1 Pre-load
At the start of every session, the memory service returns everything the agent needs in a single low-latency read: the user's stable profile, all active temporal entities (expired and superseded facts are filtered out), the most recent conversation turns, a pre-formatted context block ready for direct injection into the system prompt, and the definition of the memory search tool the agent can call on demand.
The entire response is assembled from a single indexed document read — no embedding generation, no vector search. The context block is stratified by confidence: stable profile facts, high-confidence current facts, and inferred possibilities appear under distinct labels so the agent knows which to assert and which to treat as uncertain. Total latency: under 5ms.
7.2 Lazy Retrieval: The Memory Search Tool
The agent decides when to invoke memory search. The tool definition injected at session start instructs the model to call it only when the user references a past session, asks about prior decisions, or when the pre-loaded profile lacks the answer. This is the critical difference from eager RAG: a user saying "thanks, goodbye" triggers no retrieval. Only turns where the agent genuinely needs archival recall do.
When invoked, the search follows a three-tier cascade. First, semantic search against the ACU vector index — fast, approximate, meaning-based. If hybrid mode is enabled, keyword-based scoring is fused with semantic results using reciprocal rank fusion, which improves recall for queries containing proper nouns, property addresses, and other non-compositional terms. If neither semantic nor hybrid results meet a minimum confidence threshold, the system falls back to a keyword scan against raw cold-storage turns — slower, but ensuring that facts which were never compressed into ACUs are still reachable. Fallback results carry a lower confidence score than ACU results, signaling the uncertainty to the downstream model.
7.3 Latency Budget
| Operation | Target | Measured p50 | Measured p95 | Mechanism |
|---|---|---|---|---|
| Session start (Tier 1) | <5ms | 3.2ms | 6.1ms | Single MongoDB indexed read; no embedding |
| Semantic search (Tier 2) | p95 <50ms | 18ms | 44ms | Qdrant HNSW with payload filter; retrieval timeout 8.0s |
| BM25 hybrid leg | +30ms budget | 22ms | 61ms | CPU-bound; run in executor thread; scrolls ≤1,000 vectors |
| Cold storage fallback | <500ms | 87ms | 340ms | MongoDB regex search; triggered only when Qdrant confidence low |
| Background extraction | <3s complete | 1.4s | 2.8s | Async; parallel Stage 1 + Stage 3; does not block call path |
7.4 Cold Storage Fallback: Design Decision
The cold storage fallback triggers when the vector search returns too few results or when the best match score falls below the confidence threshold. Below this threshold, the embedding model is indicating that no meaningful semantic match exists in the indexed ACUs. Rather than returning empty results — which would cause the agent to respond "I don't have that information" for facts that exist but were never indexed — the keyword fallback provides a best-effort match against raw conversational turns.
Fallback results are intentionally assigned scores below any genuine vector search score. This ensures that even when the agent receives fallback results, the confidence signal in its context correctly indicates their lower reliability relative to ACU matches.
Design Lessons
Decouple the memory loop from the LLM loop
The single most important architectural decision is running extraction asynchronously. Every synchronous memory operation on the critical call path is a latency regression. The memory service's job is to maintain a pre-computed, pre-formatted context block that the agent can consume with a single low-latency read. All intelligence — extraction, compression, indexing, supersession — happens in background workers after the turn completes.
Supersession is not optional for production agents
Append-only memory produces confident wrong answers. When a user updates a fact and the old value remains in the index at high cosine similarity to the new value, the LLM receives both at retrieval time and hallucinate a resolution. The bi-temporal model with explicit SUPERSEDED status is the correct fix — but it requires an entity key system (a stable identifier per fact type) that most teams do not design upfront. Design the entity key scheme before first deployment; retrofitting it requires re-extracting all historical conversations.
Lazy retrieval outperforms eager RAG for structured domains
For agents operating in structured domains — real estate, healthcare, financial advisory — the majority of necessary context can be pre-loaded as structured profile and temporal entity fields. Semantic search adds marginal value when the profile is well-maintained, and adds latency unconditionally. Giving the agent a memory search tool and letting it decide when to invoke it reduces unnecessary vector search calls by approximately 60–70% in our production logs, with no measurable degradation in answer quality on single-session questions.
Benchmark scores are necessary but not sufficient
LoCoMo and LongMemEval measure recall quality on well-formed questions with ground-truth answers. They do not measure cold-start behavior, lock-contention races, schema-staleness gaps, or BM25 corpus overflow — all of which occur at production scale. A system optimized only for benchmark performance will surprise its operators in production. Build the failure taxonomy first; use benchmarks to verify the happy path.
References