Nine days ago I wrote about Memorilla’s benchmark table, and how only two of its ten methods had trained on the benchmarks. This post is about one sentence near the end of the same paper, which I skipped then and keep coming back to.
Memorilla (arXiv 2610.21375, version 3 of 3 November) gives a frozen language model a memory. Every document, such as each of a user’s past chats, is cut into chunks of about 128 tokens and run once through a frozen encoder, Qwen3-Embedding-4B. Each chunk becomes a vector of 2,560 numbers. When a question comes in, a trained module of 194.1 million parameters reads the whole store and writes 16 “memory tokens” into the input of a frozen Qwen3-8B. The paper lists “user-LLM conversations, chapters of a lengthy book, or health records” as examples of what the store might hold.
The sentence is in the Limitations paragraph: “because the persistent memory store holds only embeddings,” a multimodal encoder could extend the system to images and audio, and “since each user’s memory is a separate store, it can be kept on device or deleted on request, which eases privacy concerns.”
Half of that is right, and it’s the better half. The other half reads as if storing embeddings instead of text were itself a privacy property. The paper’s own training recipe shows it isn’t, and so does the security literature.
What Stage 1 teaches
Memorilla is trained in three stages. Stage 1 uses 1.98 million English Wikipedia “paraphrase reconstruction” examples. The paper describes the job plainly: “the module must preserve enough information in its fixed memory budget for the frozen LLM to generate a paraphrase.” The module gets embeddings and nothing else. The training signal rewards the decoder for writing out what the original chunks said.
That is an embedding inverter. It’s a paraphrasing one, not a verbatim one, and it squeezes a whole collection into 16 tokens, so it will lose detail. The authors admit this loss elsewhere: Memorilla “can lose fine-grained factual details,” and on NarrativeQA it trails RAG-50, plain retrieval of 50 chunks, on “who” and “which” questions where the answer is an exact name. But the goal of the system is that whatever sits in the store can be turned back into language on demand. If it couldn’t, Memorilla wouldn’t work.
The paper’s own case study shows it. In Figure 6, a user asks a generic question, and Memorilla “personalizes a generic query by leveraging the user’s remembered iron-deficiency anemia history, despite no lexical overlap.” The users in that benchmark are synthetic. The ability is real: a medical fact the user mentioned once in a past chat is recoverable from 2,560-number vectors by a model trained to recover it.
What the attack papers found
None of this is new to security researchers. In 2020 Congzheng Song and Ananth Raghunathan, at CCS, showed that “embedding vectors can be inverted to partially recover some of the input data,” recovering “between 50%-70% of the input words” from popular sentence embeddings. They also found that authorship “can be easily extracted by training an inference model on just a handful of labeled embedding vectors,” using 10 to 50 labeled sentences per author.
Three years later, John Morris and colleagues at Cornell (EMNLP 2023) went from word bags to whole sentences. Their method, Vec2Text, guesses a text, re-embeds it, compares the result with the target vector, and corrects the guess, again and again. On 32-token passages embedded with GTR-base it recovered 92% exactly, with a BLEU score of 97.3. Against OpenAI’s ada-002 embeddings it recovered 60.9% of 32-token inputs exactly. On clinical notes from MIMIC-III it got back 89% of full names.
Two details from Vec2Text carry over to Memorilla. The first is length. Exact recovery drops fast as inputs get longer: on ada-002 passages of up to 128 tokens, averaging about 81, only 8.0% came back exactly. Token F1 was still 84, so most of the words did. Memorilla’s chunks are about 128 tokens, so I’d expect paraphrase-grade leakage rather than transcripts. For a chat that mentions a diagnosis, a name or an address, that difference doesn’t help much.
The second is the threat model. Vec2Text assumes the attacker has “text-embedding pairs from the model used to produce the data.” For a closed embedding API that takes some work. Memorilla uses Qwen3-Embedding-4B, which is open-weight under Apache 2.0 and was downloaded about 2.3 million times in the past month alone, by Hugging Face’s count. Anyone can make as many pairs as they like. I haven’t found a published inversion result for Qwen3-Embedding-4B, and I won’t guess at one. Its vectors are also much bigger than GTR-base’s, and nothing I read says whether that makes inversion easier or harder. But the precondition the attack needs is already met.
Where the leakage claims are weaker
Some leakage claims in this field have turned out weaker than they first looked.
Bargav Jayaraman and David Evans asked, at CCS 2022, whether attribute inference attacks are “just imputation.” Their answer for most of the published attacks was yes. Earlier methods “do not reveal more about the training data from the model than can be inferred by an adversary without access to the trained model, but with the same knowledge of the underlying distribution.” If an attack “infers” that a 70-year-old has high blood pressure, it may only be using base rates. Only their new white-box attacks found records that “would not be predicted without having access to the model.”
That critique applies here, with limits. If someone claims they can infer a user’s politics from that user’s Memorilla store, the fair test is whether they do better than a guess from demographics. Verbatim and paraphrase reconstruction are a different matter. No base rate gives you the name “Rhona Arntson” from a MIMIC note, which Vec2Text returned as “Rhona Arpson.” The imputation critique is a reason to discount attribute-level claims. It’s no reason to call embeddings anonymous.
There’s also a defense, and it is cheap. Morris and colleagues added Gaussian noise to GTR-base embeddings. At a noise level of 0.01, retrieval quality fell by about 2% while reconstruction “plummets to 13% of the original BLEU.”
I would like to know whether that works for Memorilla, and I doubt the transfer is clean. Retrieval only needs nearest neighbours to stay in roughly the right order. Memorilla needs the module to pull specific content out of every vector, so noise that blocks an attacker’s decoder would likely also blunt the memory. The same authors warn that “low reconstruction BLEU score is not necessarily indicative that coarser inferences, such as clinical area or treatment regimen, cannot be made from embeddings.” The paper doesn’t test noise, and I’m not claiming it fails. It just hasn’t been measured.
What deletion does buy
The per-user store is the part I’d defend. If you personalize a model by fine-tuning it on someone’s history, that history ends up spread through the weights, and “forget me” means retraining or trusting an unlearning method. Memorilla keeps each user’s data in a separate store of vectors outside the model, as a text retrieval index would. The ethics statement says it well: “a user’s embeddings can be removed without retraining.” That is a real design advantage, and it’s easy to audit.
Two caveats. The module itself is trained, and in the paper its last stage uses the training splits of the personalization benchmarks, including five extra epochs on PersonaMem-v2. The authors keep training and evaluation users “strictly disjoint,” which is the right call. But a deployment that fine-tunes the module on real users’ chats would have the same problem in the weights that the store design avoids. Deleting a store doesn’t remove what the module learned while training on it. Song and Raghunathan also found that embedding models “leak moderate amount of membership information for infrequent training data inputs.”
And “on device” protects the store only while it stays there. The 16 memory tokens are built to make the decoder say what the store holds. Memorilla writes into the LLM’s input-embedding layer, so it already can’t sit in front of a closed API. If a deployment hosts that decoder on a server, those 16 vectors are what leaves the phone.
What I’d take from it
The ethics statement is more careful than the Limitations paragraph. It says stored embeddings “can still encode sensitive information and deployments should obtain user consent and protect them accordingly.” That’s the right sentence. “Holds only embeddings, which eases privacy concerns” is the wrong one, because a reader will hear it as “doesn’t hold the text.”
I’d treat a Memorilla store, or any embedding store built from user chats, with the same rules as the chats themselves: encrypt it at rest, give it the same retention policy, and honour deletion requests the same way. The paper earns credit for making deletion clean. It doesn’t earn the idea that vectors are safer to keep than words.
References
- Fallahpour, A., Idehpour, P., Kothapalli, V., Mounier, N., Wu, S., Pardis, S. and Leskovec, J. (2026). Memorilla: Latent Semantic Memory for LLMs. arXiv:2610.21375, submitted 15 October 2026; version 3, 3 November 2026.
- Morris, J. X., Kuleshov, V., Shmatikov, V. and Rush, A. M. (2023). Text Embeddings Reveal (Almost) As Much As Text. Proceedings of EMNLP 2023, pp. 12448-12460.
- Song, C. and Raghunathan, A. (2020). Information Leakage in Embedding Models. Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security (CCS), 30 October 2020.
- Jayaraman, B. and Evans, D. (2022). Are Attribute Inference Attacks Just Imputation?. Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (CCS), 7 November 2022.
- Qwen team (2025). Qwen3-Embedding-4B model card. Hugging Face, published 3 June 2025, Apache 2.0 licence; download count checked 15 November 2026.