Two of Ten Were Trained for the Test

Memorilla went up on arXiv on 15 October, and its third version landed on 3 November. It’s a memory system for language models from a group led out of Stanford, with Jure Leskovec as senior author, and it was on the Hacker News front page today. The idea is simple to state. Encode every document once with a frozen embedding model. When a question arrives, a small trained module reads all of those embeddings and writes 16 “memory tokens” straight into the input embedding space of a frozen Qwen3-8B. No document text enters the context.

The results table looks like a sweep. Memorilla posts the best average across eight benchmarks, 49.68, against 41.92 for xRAG, 40.75 for LLMLingua-2 and 38.02 for Mem0. On FactKG it gets 84.33% where the full-context model gets 80.43%, with 108 times fewer document tokens. On PersonaMem-v2 it gets 44.60%, and the strongest baseline gets 30.90%.

I think the architecture is a real idea and some of these numbers will hold up. But the comparison in the main table isn’t the one most readers will think they’re looking at, and the reason is in Appendix B and Appendix C.

Who studied for the exam

Memorilla is trained in three stages. Stage 1 is 1.98 million Wikipedia paraphrase-reconstruction examples. Stage 2 is 1.34 million examples from ten reading-comprehension, QA and summarization datasets. Stage 3, in the paper’s words, “fine-tunes the resulting module on a unified mixture of the training splits from all eight evaluation benchmarks.” Two of them, PersonaMem-v2 and the authors’ own PersonalizationV4, then get five more epochs each on their own training split, “because their validation losses required more epochs to plateau.”

Fine-tuning on a benchmark’s training split and scoring on its test split is normal practice, and the paper is open about it. What matters is what the baselines got. Appendix C says Mem0 ran as “mem0ai 0.1.118 with its stock prompts” and MemoRAG ran from its “official repository with its prompts unchanged.” The RAG baselines are cosine-similarity retrieval feeding the same frozen decoder. Full context is the whole collection. LLMLingua-2 is a task-agnostic compressor. None of them saw a training example from these benchmarks.

Exactly one baseline did. xRAG’s bridge, a 27.3-million-parameter MLP that maps the top retrieved chunk to a single soft token, was reimplemented by the authors and “trained, from scratch on each benchmark’s training split with the same answer-token loss as Memorilla.”

Which of the ten compared methods trained on the benchmarks Two-panel chart from Table 11 and Appendices B and C of arXiv 2610.21375 version 3. Eight methods ran with no training on the eight evaluation benchmarks: closed-book, RAG-1, RAG-5, RAG-50, full context, LLMLingua-2, Mem0 version 0.1.118 with stock prompts, and MemoRAG from its official repository. Two methods were trained on the benchmarks' training splits with an answer-token loss: xRAG, a 27.3 million parameter bridge trained for 5 epochs on each benchmark, and Memorilla, a 194.1 million parameter module trained on 1.98 million Wikipedia rows, 1.34 million rows from ten other datasets, the training splits of all eight benchmarks, and 5 more epochs on PersonaMem-v2 and PersonalizationV4. All ten answer with the same frozen Qwen3-8B decoder. THE TEN METHODS IN TABLE 11 · ARXIV 2610.21375V3 · APPENDICES B AND C Ran as shipped: 8 methods Trained on the benchmarks: 2 methods No training rows from any of the eight Same answer-token loss on the training splits Closed-book RAG-1 RAG-5 RAG-50 Full context LLMLingua-2 Mem0 MemoRAG v0.1.118, stock prompts official repo, unchanged xRAG bridge · 27.3M parameters Memorilla · 194.1M parameters Reimplemented by the authors 5 epochs on each benchmark's training split 1.98M Wikipedia rows, then 1.34M rows from 10 sets Then the training splits of all eight benchmarks Then 5 more epochs each on PersonaMem-v2 and PersonalizationV4 All ten answer with the same frozen Qwen3-8B decoder and read documents through the same Qwen3-Embedding-4B encoder.
Eight of the ten methods came to the test cold. The two that trained on the benchmarks are the two soft-token methods, and on two of the benchmarks they finish first and second.

Where the trained runner-up shows up

Once you sort the table this way, a pattern falls out. On PersonaMem-v2 (5,000 test questions), the paper says Memorilla’s 44.60% beats “30.90% for xRAG, the strongest baseline,” and that every other baseline “stays within about three points of closed-book (25.64%), including full context.” On LaMP-7 the wording is almost the same: “every baseline except xRAG (46.84) stays within one ROUGE-L point of the closed-book model,” and Memorilla lifts it from 43.13 to 48.37.

So on both of these personalization benchmarks the order is: the method with the most task training, then the other method with task training, then everything that ran as shipped, bunched near the model that saw no documents. That’s what you’d expect if a good share of the gain comes from learning what these benchmarks’ answers look like, as well as from remembering the documents.

Gain over closed-book on PersonaMem-v2 and FactKG Horizontal bar chart from Table 11 of arXiv 2610.21375 version 3, showing accuracy points gained over the closed-book Qwen3-8B. On PersonaMem-v2 short-answer, where closed-book scores 25.64%, the best untrained baseline, MemoRAG at 28.76%, gains 3.12 points; xRAG at 30.90% gains 5.26; Memorilla at 44.60% gains 18.96. On FactKG, where closed-book scores 65.97%, the best untrained baseline, full context at 80.43%, gains 14.46 points; xRAG at 78.98% gains 13.01; Memorilla at 84.33% gains 18.36. ACCURACY POINTS ABOVE CLOSED-BOOK · ARXIV 2610.21375V3 TABLE 11 0 +5 +10 +15 +20 points PersonaMem-v2 · closed-book 25.64% FactKG · closed-book 65.97% Best untrained (MemoRAG) xRAG, trained Memorilla, trained Best untrained (full context) xRAG, trained Memorilla, trained MemoRAG 28.76%, +3.12 — Table 11 xRAG 30.90%, +5.26 — Table 11 Memorilla 44.60%, +18.96 — Table 11 Full context 80.43%, +14.46 — Table 11 xRAG 78.98%, +13.01 — Table 11 Memorilla 84.33%, +18.36 — Table 11 +3.12 +5.26 +18.96 +14.46 +13.01 +18.36 PersonaMem-v2 also got 5 extra epochs of Memorilla training on its own split. FactKG used the shared multi-task checkpoint.
On PersonaMem-v2 almost nothing that ran as shipped beats closed-book by much. On FactKG the untrained full-context model gets most of the way, and trained xRAG is no better.

FactKG is the counterexample, and it’s the result I’d trust most. There, xRAG’s task training buys nothing: 78.98% against 80.43% for plain full context. Memorilla still beats full context by almost four points, using the shared checkpoint with no extra FactKG epochs. The per-question breakdown backs this up. Memorilla’s lead over RAG-50 grows with the number of entities in a claim, from 4.23 points at one entity to 7.59 at four or more. And in an architecture ablation, swapping every document embedding for Gaussian noise cut Memorilla’s FactKG score by 5.93 points. So the stored documents are doing real work there, beyond whatever the decoder learns about the answer format.

The control that’s there, and the one I’d want

The authors did see this objection coming, which is to their credit. Appendix E.4 trains a rank-130 LoRA adapter, 268.4 million parameters, “on the same training rows as Memorilla’s per-task fine-tuning,” with no documents in context, and reports that Memorilla “leads on every primary metric, although the PubMedQA and LaMP-7 gaps are within noise.”

The caption of that figure has a detail that weakens it, though. The comparison is “Memorilla with an 8B decoder” against “a no-document rank-130 LoRA adapter with a 4B decoder.” The control runs on a model half the size. It does show that documents matter. It doesn’t show how much of the PersonaMem-v2 gap the 8B decoder would close on its own with that training.

The control I’d want is the obvious one: give RAG-50 or full context the same training rows, either as a LoRA on the same Qwen3-8B or as an xRAG-style bridge trained on all eight splits. If Memorilla still wins by 14 points on PersonaMem-v2 against a trained retriever on the same decoder, that’s a strong result. Until then, the 44.60% against 30.90% compares a heavily trained memory with a lightly trained one, and everything else in the table is an untrained reference point.

I made a related complaint in August about agent memory benchmarks, where nearly every published number came from the people shipping the system. This paper is better than that. It runs Mem0 and MemoRAG itself with the same encoder and decoder, which is more than most memory papers do. But “same encoder and decoder” isn’t the same as “same training data,” and the paper doesn’t measure how much of its headline depends on that difference.

Small denominators

Two other results get prominent placement and rest on very few items.

PubMedQA is where Memorilla loses: 59.00% against 67.00% for four baselines and 69.00% for LLMLingua-2. The test set is “the 100-example test portion of a 900/100 split,” so the gap to the four baselines is eight questions. The paper reads it as a case where condensing four abstracts, 324 tokens in all, “offers little benefit,” which is reasonable. It also reports, in the encoder-size ablation, that a 3.00-point PubMedQA gain “corresponds to three of 100 questions.” I’d apply the same humility in the other direction.

The agent result is smaller still. In TextWorld, a GRPO-trained memory module helps a frozen Qwen3-4B-Instruct agent finish games. Pass@3 goes from 0.250 with full history to 0.375, and avg@3 from 0.186 to 0.242. The test split is 40 games, and the appendix notes they’re “30 distinct, since the five medium and five hard games appear twice.” A pass@3 of 0.375 against 0.250 on 40 games is 15 against 10.

Two results that rest on 100 questions and 40 games Horizontal bar chart from Table 11, Appendix B and Section 6.2 of arXiv 2610.21375 version 3. On PubMedQA, with 100 test questions, LLMLingua-2 scored 69%, four baselines including full context scored 67%, and Memorilla scored 59%, an eight-question gap to the four baselines. On TextWorld, with a 40-game test split of 30 distinct games, pass at 3 was 0.375 for GRPO-trained Memorilla, 0.250 for full-history in-context learning, and 0.225 for an untrained memory module, which is 15, 10 and 9 of 40 games. PUBMEDQA N=100 · TEXTWORLD 40 TEST GAMES · ARXIV 2610.21375V3 0% 25% 50% 75% 100% PubMedQA accuracy · 100 questions TextWorld pass@3 · 40 games LLMLingua-2 Full context and 3 others Memorilla Memorilla, GRPO-trained Full-history ICL Untrained memory 69.00% — Table 11 67.00% — Table 11 59.00% — Table 11 pass@3 0.375 — Section 6.2 pass@3 0.250 — Section 6.2 pass@3 0.225 — Section 6.2 69 of 100 67 of 100 59 of 100 15 of 40 10 of 40 9 of 40 The 40 TextWorld test games are 30 distinct games; five medium and five hard games appear twice.
The loss on PubMedQA is eight questions. The agent win is five games out of 40, on a set where ten of the games are repeats.

The TextWorld section has a better result tucked inside it. Full-history in-context learning ran out of its 4,096-token budget on 71% of rollouts, against 58% with trained memory, while both read the same trajectory text. An untrained memory module of the same shape did worse than no memory at all (pass@3 0.225). Fewer loops is a behavior you can check by reading trajectories, and the paper shows one. I’d like to see it on a few hundred games.

What I’d take from it

The design itself makes sense. Long contexts are not used evenly. Liu and colleagues showed in TACL in 2024 that accuracy “significantly degrades when models must access relevant information in the middle of long contexts,” and in Memorilla’s own table full context trails Memorilla on every benchmark except PubMedQA. Replacing PersonaMem-v2’s average 31,587-token full-context prompt with 16 learned vectors is an appealing trade, and the paper reports 0.74 seconds against 19.9 seconds end to end with 512 documents on one H100.

The costs are the familiar ones for soft-token methods. Jung and Kim, writing in IEEE Access in 2024 about prompt compression, listed the problems with embedding-based compressors as “interpretability, the fixed number of embedding tokens, reusability across different LMs, and inapplicability when interacting with black-box APIs.” Memorilla has every one of them. You can’t read the 16 tokens. They go into the input embedding layer, so a closed API is out. And the authors say that “our experiments pair a Qwen encoder with a Qwen decoder,” so transfer across model families is untested. Each new decoder means training a new memory module, through the same three-stage curriculum, on your own task’s training data.

In practice, if you have labeled examples from the task you care about, Memorilla is a good way to spend them, and FactKG suggests it can beat even a full-context read. If you don’t, the paper doesn’t tell you much about how it would do, because the untrained version of the module isn’t in the main table, and in TextWorld the untrained one lost to plain history. The fair headline is narrower than the abstract’s: on these eight benchmarks, after training on all eight, Memorilla beat eight methods that had not trained at all, and one that had.

References

  1. Fallahpour, A., Idehpour, P., Kothapalli, V., Mounier, N., Wu, S., Pardis, S. and Leskovec, J. (2026). Memorilla: Latent Semantic Memory for LLMs. arXiv:2610.21375, submitted 15 October 2026; version 3, 3 November 2026.
  2. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F. and Liang, P. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12, pp. 157-173, 2024.
  3. Jung, H. and Kim, K.-J. (2024). Discrete Prompt Compression With Reinforcement Learning. IEEE Access, 12, 2024.
  4. Chhikara, P., Khant, D., Aryan, S., Singh, T. and Yadav, D. (2025). Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. arXiv:2504.19413, submitted 28 April 2025.