An agent that forgets is rarely short on room. Every “our agent lost the thread” postmortem I have watched ends in one of three places: the context degraded long before it reached the model’s advertised limit, the memory system bought to fix that published its own headline number, or the store quietly accumulated something wrong — sometimes because an attacker put it there.
None of those is a capacity problem. What is missing is a retrieval policy — the rules deciding what gets written, what gets read back, and what gets thrown away. Before that argument is worth having, it helps to be exact about what “memory” is holding.
Agent memory decomposes into four stores — working, episodic, semantic, procedural — each with its own lifetime and its own way of breaking.
If you have only read agent-memory papers, that split reads as a fresh synthesis. It was formalized in the Soar cognitive architecture: Laird, Rosenbloom and Newell’s Chunking in Soar: The Anatomy of a General Learning Mechanism appeared in the first issue of Machine Learning in 1986 — thirty-seven years before MemGPT. Chunking compiled successful problem-solving traces into reusable rules, which is, mechanically, what a 2026 paper means when it says the agent writes its own trajectory to procedural memory. The idea was implemented, evaluated and argued over in symbolic systems decades before anyone had a transformer to attach it to. Keep that in mind the next time an architecture diagram presents it as novel.
The stores are the easy part. Each of the three failures below happens in the policy wrapped around them.
The tempting fix is to buy a bigger window. The independent evidence against that is unusually consistent.
The foundational result is Liu et al.’s “Lost in the Middle,” which found a U-shaped curve: models retrieve reliably from the beginning and end of their context and measurably worse from the middle, including models explicitly built for long context. It is one of the few results in this area that has been peer-reviewed rather than merely posted.
The stronger, more recent results are about what happens when you take away the lexical crutch. NoLiMa (Adobe Research, ICML 2025) removes literal keyword overlap between the question and the buried fact. GPT-4o falls from 99.3% at short context to 69.7% at 32K tokens, and 11 of the 13 tested models drop below half their short-context baseline by 32K — a length most practitioners treat as trivially safe. RULER (NVIDIA) makes the complementary point: passing needle-in-a-haystack does not mean the window works. Of 17 models claiming 32K or more, only about half handled 32K effectively once the task stopped being pure string-matching. Chroma’s “context rot” report tested 18 models across the Claude, GPT, Gemini and Qwen families and found degradation that begins well below the stated limit and does not increase uniformly with length — read it with the standing caveat that Chroma sells a vector database, though the report itself is a methodology-transparent evaluation rather than a product claim.
None of this says long context is useless. It says long context is a buffer, not a filing system, and that treating it as memory means paying full price for retrieval you are not reliably getting.
The pitch is usually cost: stop re-sending the transcript, retrieve the three relevant facts instead. That is real, but the comparison is often against the wrong baseline.
As of 31 August 2026, Anthropic lists Claude Sonnet 5 at $2 per million input tokens and $10 per million output, with a cache read priced at 0.1× base — $0.20 per million. Opus 5 is $5 / $25, cache read $0.50 per million. OpenAI’s published GPT-5 pricing follows the same shape: $1.25 input, $0.125 cached input, $10 output. Both vendors converge on a cache hit costing about a tenth of a fresh input token, and Anthropic’s 1M-token window on recent models carries no long-context surcharge. (Both revise pricing often; that is the access date, not a guarantee.)
The honest question is “retrieval versus a cached window,” not “retrieval versus stuffing the window.” Against uncached re-sending, a memory layer looks like a 10× saving. Against prompt caching — which is a configuration change, not an architecture — much of that margin is already available for free. Memory earns its keep on the axes caching cannot touch: crossing session boundaries, surviving cache expiry, and holding facts that were never in this conversation at all.
Whether the layer works is harder to establish than what it costs. The purpose-built memory systems publish impressive figures, and almost all of them are published by their own authors, frequently on benchmarks their competitors designed. That is the detail that changed how I read this literature.
MemGPT (Packer et al.) introduced the OS-inspired framing — paging between a resident “main context” and an external store — and reports that fixed-context GPT-4 baselines collapse to 0% on synthetic nested key-value retrieval by three levels of nesting while the paged system holds up. Self-reported. Zep reports 94.8% versus MemGPT’s 93.4% on DMR, a benchmark MemGPT’s own team built, and up to 18.5% accuracy improvement with 90% lower latency on LongMemEval. Self-reported. Mem0 reports a 26% relative improvement over OpenAI’s memory feature on LoCoMo under an LLM-as-judge metric, plus 91% lower p95 latency and over 90% token cost savings. Self-reported.
The independent numbers are scarcer and less flattering. LongMemEval, built by academics rather than a memory vendor, finds roughly a 30% accuracy drop for commercial chat assistants and long-context models on sustained-interaction recall. LoCoMo (ACL 2024) is the honest stress test: conversations averaging about 300 turns and ~9K tokens across up to 35 sessions — about 9× the length and 6× the turns of the prior standard — on which models still trail humans.
I am not accusing anyone of fraud. Self-reported benchmark numbers are how this field works, and the papers disclose their affiliations plainly. But the asymmetry is real, and it is the same pattern I wrote about when measuring progress toward AGI: the instrument and the thing being measured are often built by the same people.
The third failure mode gets the least engineering attention and deserves the most. A store the agent writes to across sessions is, by construction, a store an attacker may be able to write to.
MINJA (NeurIPS 2025) demonstrates memory injection into agents with persistent shared memory through ordinary query-only interaction — no privileged write access required. I am deliberately not quoting its success rates here; I could not confirm the specific figures against the paper text, and this is not a number to get wrong. Unit 42 published a working proof-of-concept in October 2025 poisoning an agent’s long-term memory via indirect prompt injection from a malicious webpage, built on Amazon Bedrock Agents. That is a proof-of-concept, not a documented production breach — I found no public post-mortem of a real incident with quantified damage, which is a statement about disclosure practice as much as about risk.
The uncomfortable property is persistence. A bad retrieval is one bad answer. A bad write is every answer afterward, until someone notices. None of the memory systems above ship a default answer for provenance, expiry, or contradiction resolution — which is why, as I found reading twelve agent codebases, the projects that survive tend to be the ones that wrote a forgetting rule before they wrote a remembering one.
That is roughly the order I would build in. Start with prompt caching and measure. Add episodic storage — plain append-only session logs — before any vector store, because most “memory” requests turn out to be “what did we decide last Tuesday.” Make every semantic fact carry provenance and a write timestamp, so contradiction resolution is a query rather than an archaeology project. Treat procedural memory as code: reviewed, versioned, diffable, never silently self-modifying. And test retrieval on paraphrases, not on keyword matches — NoLiMa is the cheapest available lesson about what your eval is failing to measure.
Capacity was never the constraint. Judgment about what to keep is.
References