The Store Holds Only Embeddings

Nine days ago I wrote about Memorilla’s benchmark table, and how only two of its ten methods had trained on the benchmarks. This post is about one sentence near the end of the same paper, which I skipped then and keep coming back to.

Memorilla (arXiv 2610.21375, version 3 of 3 November) gives a frozen language model a memory. Every document, such as each of a user’s past chats, is cut into chunks of about 128 tokens and run once through a frozen encoder, Qwen3-Embedding-4B. Each chunk becomes a vector of 2,560 numbers. When a question comes in, a trained module of 194.1 million parameters reads the whole store and writes 16 “memory tokens” into the input of a frozen Qwen3-8B. The paper lists “user-LLM conversations, chapters of a lengthy book, or health records” as examples of what the store might hold.

The sentence is in the Limitations paragraph: “because the persistent memory store holds only embeddings,” a multimodal encoder could extend the system to images and audio, and “since each user’s memory is a separate store, it can be kept on device or deleted on request, which eases privacy concerns.”

Half of that is right, and it’s the better half. The other half reads as if storing embeddings instead of text were itself a privacy property. The paper’s own training recipe shows it isn’t, and so does the security literature.

Where a user's chats live inside Memorilla Flow diagram from Sections 2 and 2.6 and Appendix B of arXiv 2610.21375 version 3. A user's chats are cut into chunks of about 128 tokens, embedded by the frozen Qwen3-Embedding-4B encoder into 2,560 numbers per chunk, and kept in a per-user embedding store. A trained memory module of 194.1 million parameters reads the store and writes 16 memory tokens for a frozen Qwen3-8B. The store, per the paper, can be kept on device or deleted on request. In Stage 1 of training, on 1.98 million Wikipedia examples, the module and frozen decoder are trained to produce a paraphrase of the stored chunks from their embeddings. MEMORILLA'S DATA PATH · ARXIV 2610.21375V3 · SECTIONS 2, 2.6, APPENDIX B Per user: computed once, stored, deletable Per question: trained module + frozen LLM User chats Frozen encoder Embedding store Memory module Frozen Qwen3-8B ~128-token chunks Qwen3-Embedding-4B 2,560 numbers/chunk 194.1M parameters reads 16 tokens Stage 1 of training, 1.98M Wikipedia examples Input: the chunk embeddings only. Target: the frozen LLM writes a paraphrase of the original text. The encoder is open-weight (Apache 2.0), so anyone can make their own text-embedding pairs with it.
The store is the only place the chats persist, and the first lesson the module learns is how to get the text back out of it.

What Stage 1 teaches

Memorilla is trained in three stages. Stage 1 uses 1.98 million English Wikipedia “paraphrase reconstruction” examples. The paper describes the job plainly: “the module must preserve enough information in its fixed memory budget for the frozen LLM to generate a paraphrase.” The module gets embeddings and nothing else. The training signal rewards the decoder for writing out what the original chunks said.

That is an embedding inverter. It’s a paraphrasing one, not a verbatim one, and it squeezes a whole collection into 16 tokens, so it will lose detail. The authors admit this loss elsewhere: Memorilla “can lose fine-grained factual details,” and on NarrativeQA it trails RAG-50, plain retrieval of 50 chunks, on “who” and “which” questions where the answer is an exact name. But the goal of the system is that whatever sits in the store can be turned back into language on demand. If it couldn’t, Memorilla wouldn’t work.

The paper’s own case study shows it. In Figure 6, a user asks a generic question, and Memorilla “personalizes a generic query by leveraging the user’s remembered iron-deficiency anemia history, despite no lexical overlap.” The users in that benchmark are synthetic. The ability is real: a medical fact the user mentioned once in a past chat is recoverable from 2,560-number vectors by a model trained to recover it.

What the attack papers found

None of this is new to security researchers. In 2020 Congzheng Song and Ananth Raghunathan, at CCS, showed that “embedding vectors can be inverted to partially recover some of the input data,” recovering “between 50%-70% of the input words” from popular sentence embeddings. They also found that authorship “can be easily extracted by training an inference model on just a handful of labeled embedding vectors,” using 10 to 50 labeled sentences per author.

Three years later, John Morris and colleagues at Cornell (EMNLP 2023) went from word bags to whole sentences. Their method, Vec2Text, guesses a text, re-embeds it, compares the result with the target vector, and corrects the guess, again and again. On 32-token passages embedded with GTR-base it recovered 92% exactly, with a BLEU score of 97.3. Against OpenAI’s ada-002 embeddings it recovered 60.9% of 32-token inputs exactly. On clinical notes from MIMIC-III it got back 89% of full names.

How much text inversion attacks recover from embeddings Horizontal bar chart of token F1, the share of the original words recovered, from two papers. Song and Raghunathan, CCS 2020, on popular sentence embeddings: 50 to 70 percent of input words, F1 0.5 to 0.7. Vec2Text, Morris and colleagues, EMNLP 2023, Table 1, with 50 correction steps and sequence beam search: GTR-base on 32-token Natural Questions passages, token F1 99 and 92.0 percent exact matches; OpenAI ada-002 on 32-token MSMARCO passages, token F1 96 and 60.9 percent exact; ada-002 on MSMARCO passages of up to 128 tokens, averaging 80.9 tokens, token F1 84 and 8.0 percent exact. WORDS RECOVERED FROM EMBEDDINGS (TOKEN F1) · SONG & RAGHUNATHAN 2020 · MORRIS ET AL. 2023, TABLE 1 Sentence embeddings, 2020 Vec2Text, GTR-base, 32 tokens Vec2Text, ada-002, 32 tokens Vec2Text, ada-002, up to 128 bag of words, several encoders 92.0% recovered exactly 60.9% recovered exactly avg 80.9 tokens · 8.0% exact F1 0.5-0.7 (50-70% of words) — Song and Raghunathan, CCS 2020 Token F1 99, exact 92.0% — Morris et al., EMNLP 2023, Table 1 Token F1 96, exact 60.9% — Morris et al., EMNLP 2023, Table 1 Token F1 84, exact 8.0% — Morris et al., EMNLP 2023, Table 1 50-70 99 96 84 Longer inputs are harder to recover exactly, but most words still come back. Memorilla's chunks are about 128 tokens.
Inversion went from partial word lists in 2020 to near-verbatim sentences in 2023. Length is the main thing that slows it down.

Two details from Vec2Text carry over to Memorilla. The first is length. Exact recovery drops fast as inputs get longer: on ada-002 passages of up to 128 tokens, averaging about 81, only 8.0% came back exactly. Token F1 was still 84, so most of the words did. Memorilla’s chunks are about 128 tokens, so I’d expect paraphrase-grade leakage rather than transcripts. For a chat that mentions a diagnosis, a name or an address, that difference doesn’t help much.

The second is the threat model. Vec2Text assumes the attacker has “text-embedding pairs from the model used to produce the data.” For a closed embedding API that takes some work. Memorilla uses Qwen3-Embedding-4B, which is open-weight under Apache 2.0 and was downloaded about 2.3 million times in the past month alone, by Hugging Face’s count. Anyone can make as many pairs as they like. I haven’t found a published inversion result for Qwen3-Embedding-4B, and I won’t guess at one. Its vectors are also much bigger than GTR-base’s, and nothing I read says whether that makes inversion easier or harder. But the precondition the attack needs is already met.

Where the leakage claims are weaker

Some leakage claims in this field have turned out weaker than they first looked.

Bargav Jayaraman and David Evans asked, at CCS 2022, whether attribute inference attacks are “just imputation.” Their answer for most of the published attacks was yes. Earlier methods “do not reveal more about the training data from the model than can be inferred by an adversary without access to the trained model, but with the same knowledge of the underlying distribution.” If an attack “infers” that a 70-year-old has high blood pressure, it may only be using base rates. Only their new white-box attacks found records that “would not be predicted without having access to the model.”

That critique applies here, with limits. If someone claims they can infer a user’s politics from that user’s Memorilla store, the fair test is whether they do better than a guess from demographics. Verbatim and paraphrase reconstruction are a different matter. No base rate gives you the name “Rhona Arntson” from a MIMIC note, which Vec2Text returned as “Rhona Arpson.” The imputation critique is a reason to discount attribute-level claims. It’s no reason to call embeddings anonymous.

There’s also a defense, and it is cheap. Morris and colleagues added Gaussian noise to GTR-base embeddings. At a noise level of 0.01, retrieval quality fell by about 2% while reconstruction “plummets to 13% of the original BLEU.”

What a small amount of noise costs retrieval and inversion Horizontal bar chart from Section 6 of Morris and colleagues, EMNLP 2023. Adding Gaussian noise at level 0.01 to GTR-base embeddings of 32-token texts kept about 98 percent of retrieval quality, measured as mean NDCG@10 over 15 BEIR tasks, because retrieval was degraded by about 2 percent. Reconstruction by Vec2Text fell to 13 percent of its original BLEU score. SHARE KEPT AFTER GAUSSIAN NOISE, λ = 0.01 · GTR-BASE, 32-TOKEN INPUTS · MORRIS ET AL. 2023, §6 Retrieval quality Reconstruction mean NDCG@10, 15 BEIR tasks Vec2Text BLEU ~98% kept (degraded about 2%) — Morris et al., EMNLP 2023, §6 13% of original BLEU — Morris et al., EMNLP 2023, §6 ~98% 13% The authors caution that an attacker trained on noisy embeddings might do better, and that low BLEU doesn't rule out coarser inferences such as clinical area or treatment.
Noise breaks a naive inverter almost for free. It's an untested idea for a system whose whole job is to read the store back.

I would like to know whether that works for Memorilla, and I doubt the transfer is clean. Retrieval only needs nearest neighbours to stay in roughly the right order. Memorilla needs the module to pull specific content out of every vector, so noise that blocks an attacker’s decoder would likely also blunt the memory. The same authors warn that “low reconstruction BLEU score is not necessarily indicative that coarser inferences, such as clinical area or treatment regimen, cannot be made from embeddings.” The paper doesn’t test noise, and I’m not claiming it fails. It just hasn’t been measured.

What deletion does buy

The per-user store is the part I’d defend. If you personalize a model by fine-tuning it on someone’s history, that history ends up spread through the weights, and “forget me” means retraining or trusting an unlearning method. Memorilla keeps each user’s data in a separate store of vectors outside the model, as a text retrieval index would. The ethics statement says it well: “a user’s embeddings can be removed without retraining.” That is a real design advantage, and it’s easy to audit.

Two caveats. The module itself is trained, and in the paper its last stage uses the training splits of the personalization benchmarks, including five extra epochs on PersonaMem-v2. The authors keep training and evaluation users “strictly disjoint,” which is the right call. But a deployment that fine-tunes the module on real users’ chats would have the same problem in the weights that the store design avoids. Deleting a store doesn’t remove what the module learned while training on it. Song and Raghunathan also found that embedding models “leak moderate amount of membership information for infrequent training data inputs.”

And “on device” protects the store only while it stays there. The 16 memory tokens are built to make the decoder say what the store holds. Memorilla writes into the LLM’s input-embedding layer, so it already can’t sit in front of a closed API. If a deployment hosts that decoder on a server, those 16 vectors are what leaves the phone.

What I’d take from it

The ethics statement is more careful than the Limitations paragraph. It says stored embeddings “can still encode sensitive information and deployments should obtain user consent and protect them accordingly.” That’s the right sentence. “Holds only embeddings, which eases privacy concerns” is the wrong one, because a reader will hear it as “doesn’t hold the text.”

I’d treat a Memorilla store, or any embedding store built from user chats, with the same rules as the chats themselves: encrypt it at rest, give it the same retention policy, and honour deletion requests the same way. The paper earns credit for making deletion clean. It doesn’t earn the idea that vectors are safer to keep than words.

References

  1. Fallahpour, A., Idehpour, P., Kothapalli, V., Mounier, N., Wu, S., Pardis, S. and Leskovec, J. (2026). Memorilla: Latent Semantic Memory for LLMs. arXiv:2610.21375, submitted 15 October 2026; version 3, 3 November 2026.
  2. Morris, J. X., Kuleshov, V., Shmatikov, V. and Rush, A. M. (2023). Text Embeddings Reveal (Almost) As Much As Text. Proceedings of EMNLP 2023, pp. 12448-12460.
  3. Song, C. and Raghunathan, A. (2020). Information Leakage in Embedding Models. Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security (CCS), 30 October 2020.
  4. Jayaraman, B. and Evans, D. (2022). Are Attribute Inference Attacks Just Imputation?. Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (CCS), 7 November 2022.
  5. Qwen team (2025). Qwen3-Embedding-4B model card. Hugging Face, published 3 June 2025, Apache 2.0 licence; download count checked 15 November 2026.