Twenty-Three of Thirty-Six From the Text Alone

On 5 November Shivesh Rane of Goa College of Engineering posted a short paper to arXiv with a plain question: if someone hands you a LoRA adapter (the small low-rank add-on that Hu and colleagues introduced in 2021) and nothing else, can you tell what kind of data it was trained on?

The setup is small on purpose. One base model, Qwen3.5-0.8B, loaded in 4-bit. Eighteen public datasets across five domains: medical, code, mathematics, non-English text and creative prose. Two seeds per dataset, so 36 adapters, all trained with the same recipe: rank 16, a fixed budget of 1.5 million training tokens, one epoch. The only thing that changes between runs is the data. The paper says the whole study ran under “a hard limit of US$30 of cloud GPU time.”

The order of operations is what makes it worth reading. The dataset-to-domain map, the probe prompts, the folds and the scoring script were committed before any adapter was trained, and the scoring script was run once. That matters later, because one of the author’s own hypotheses failed.

Three ways to look at an adapter

The paper compares three levels of access, and builds a fixed-length feature vector for each.

How the study turns 36 adapters into domain guesses Flow diagram from sections 4.1 to 4.4 of arXiv 2611.07610. A base model, Qwen3.5-0.8B in 4-bit, is fine-tuned on 18 datasets with 2 seeds each, giving 36 LoRA adapters at rank 16 with 1.5 million training tokens each. Three feature families are extracted: white-box weight statistics with 48 features, gray-box activations on 200 probe prompts with 78 features, and black-box features from generated text with 455 features. Each family feeds a logistic regression scored by leaving one dataset out, which predicts one of five domains. STUDY PIPELINE · ARXIV 2611.07610 SECTIONS 4.1–4.4 · 5 NOV 2026 Base model 36 adapters White-box: weights Gray-box: activations Black-box: outputs Logistic regression Qwen3.5-0.8B, 4-bit 18 datasets × 2 seeds LoRA, rank 16 1.5M tokens each 48 features: norms, spectra 78 features on 200 probes 455 features from replies leave one dataset out → 1 of 5 domains Probes are domain-neutral, e.g. “Describe your morning routine.” The protocol was committed before any training.
Every route ends at the same classifier. The only difference is how much of the adapter it gets to see.

White-box means the adapter’s own parameters. The features are summary statistics: the Frobenius norm and top singular values of each layer’s update, a profile of how the change is spread across layers, a few ratios. Gray-box means hidden states on a fixed set of 200 probe prompts, without the parameters. Black-box means what the adapted model writes back to those probes, plus the entropy of its next-token distributions, which some APIs expose as log-probabilities.

The probes are deliberately domain-neutral, with no medical, code, numeric, non-English or narrative vocabulary; the paper’s example is “Describe your morning routine.” I think that’s the cleverest choice in the design. If the classifier can still tell a medical adapter from a code one, the signal is coming from the adapter, not from the question.

The headline, and the hypothesis that lost

The test that counts is leave-one-dataset-out. The classifier is trained on every run except the two from one dataset, then asked to name the domain of that dataset’s runs. To get a medical adapter right, it has to have learned what medical adapters look like in general from the other three medical datasets. Five domains, so chance is 20%.

Domain accuracy on held-out datasets, by access level Bar chart from Tables 2 and 3 of arXiv 2611.07610. Leave-one-dataset-out domain accuracy on 36 runs: white-box weights 41.7%, with a dataset-clustered 95% interval of 19.4% to 63.9%; gray-box activations 61.1%, interval 41.7% to 80.6%; black-box outputs 63.9%, interval 41.7% to 86.1%. Chance is 20%. With a third, independently resampled run per dataset, 54 runs in all, the accuracies were 40.7%, 64.8% and 63.0%. LEAVE-ONE-DATASET-OUT DOMAIN ACCURACY · ARXIV 2611.07610 TABLES 2–3 chance, 20% 0% 20% 40% 60% 80% 100% White-box: weights Gray-box: activations Black-box: outputs 48 features 78 features 455 features 41.7% (15 of 36) — Table 2 61.1% (22 of 36) — Table 2 63.9% (23 of 36) — Table 2 54 runs: 40.7% — Table 3 54 runs: 64.8% — Table 3 54 runs: 63.0% — Table 3 41.7% 61.1% 63.9% 95% interval, resampling whole datasets 54 runs, third run on resampled data Bars are the pre-registered 36-run result. The white-box interval reaches below chance.
The adapter's replies to bland prompts identified the domain of an unseen dataset 23 times out of 36. Its weights managed 15, and that result can't be told apart from chance.

Black-box outputs got 63.9%, which is 23 of 36. Activations got 61.1%. Weight statistics got 41.7%. The author reports two kinds of interval: one that resamples individual predictions, and a more conservative one that resamples whole datasets, since the two seeds of a dataset share their training data and aren’t independent. Under the conservative interval, activations and outputs still exclude chance. The weights don’t: their interval runs from 19.4% to 63.9%, and the paper says the white-box result “should not be read as established.”

The pre-registered hypothesis was that signal would decay as access got worse: weights best, then activations, then outputs. “The data refute this ordering,” the paper says, and I like that it says so in the abstract rather than burying it. The author’s own explanation is the likelier one: the hand-built weight features keep norms and spectra but throw away singular directions, so they may leave usable structure on the table. That’s a statement about these 48 features, not about weights in general. Richer white-box features are listed as the obvious next step.

The black-box number also holds up when you take away the part that needs special API access. Drop the three entropy features, the only ones that need token probabilities, and accuracy stays at 63.9%; only one of the 36 predictions changes. The 384-dimensional sentence-embedding centroid of the replies, on its own, reaches 61.1%. Most of the signal is in what the adapter says.

Where it goes wrong, and what survives a push

The two strong feature families fail in different places. With activations, medical and creative adapters get confused, 4 of 8 medical runs called creative and 3 of 8 creative runs called medical; both domains are English prose and shift internal statistics in similar ways. With outputs that confusion mostly goes away (6 of 8 medical and 5 of 8 creative correct), and the biggest remaining mix-up is code against mathematics, 4 of the 13 errors. Both are domains whose training answers are symbolic and step-by-step. The paper reads this as the two families capturing different aspects of the fine-tune, which seems right to me.

Reviewers asked for robustness checks after acceptance, and the author ran them on separate sets of runs with the same frozen script, so the pre-registered table stays as it was. One check trained a third run per dataset on a freshly drawn 1.5M-token subsample. For 15 of the 18 datasets that subsample is largely disjoint from the original, because the budget uses at most 26% of the dataset. Accuracy barely moved: 40.7%, 64.8% and 63.0% on 54 runs. The other check changed the LoRA rank.

Correct domain calls on ten adapters trained at rank 8 or 64 Paired bar chart from section 6.2 and Table 3 of arXiv 2611.07610. Ten extra adapters, one dataset per domain at ranks 8 and 64, were classified by a model trained only on rank-16 runs of other datasets. White-box weights got 3 of 10 right, gray-box activations 6 of 10, black-box outputs 8 of 10. When rank-16 runs of the same dataset were also allowed in training, accuracy on the ten runs was 0.40, 0.80 and 1.00, that is 4, 8 and 10 of 10. RANK TRANSFER · 10 ADAPTERS AT RANK 8 OR 64 · ARXIV 2611.07610 SECTION 6.2 0 5 10 of 10 White-box: weights Gray-box: activations Black-box: outputs 3 of 10 — section 6.2 6 of 10 — section 6.2 8 of 10 — section 6.2 0.40 — section 6.2 0.80 — section 6.2 1.00 — section 6.2 3/10 6/10 8/10 4/10 8/10 10/10 Trained on rank-16 runs of other datasets Same dataset's rank-16 runs also in training Five datasets per rank, so the author calls these counts indicative only. All five rank-64 adapters read as medical from the weights.
Changing the rank broke the weight features and left the replies alone: every dataset got the same black-box prediction at ranks 8, 16 and 64.

The weight features aren’t rank-invariant: all five rank-64 adapters were predicted as medical, correctly only for PubMedQA, because spectra and norms move with the rank itself. The black-box family gave each dataset the same prediction at every rank, four of five correct, with PubMedQA wrong at every rank. Of all the checks, this is the one I’d weigh most. A feature that tracks a hyperparameter instead of the data is a feature an adversary can move by changing the hyperparameter. One that tracks behavior is harder to shift without changing what the adapter is for.

What this means for people shipping adapters

The privacy literature has spent years on finer questions. Shokri and colleagues showed in 2017 that black-box access to a trained model can reveal whether a specific record was in its training set. Membership inference, dataset inference and property inference all start from a candidate: this record, this dataset, this attribute. Rane’s question is coarser and needs no candidate at all. You don’t need to guess the data. You ask the adapter about its morning routine and read the answer.

This matters to two groups. For anyone distributing adapters trained on data they’d rather not disclose, the general kind of that data leaks through ordinary use, and the paper notes this needs “nothing beyond the generated text for a fixed set of neutral prompts, i.e., ordinary API access.” For auditors, it’s a cheap first check. Regulators are already working out what a modified model owes. In the April issue of Computer Law & Security Review, Hacker and Holweg look at fine-tuning under the EU AI Act and propose a compliance structure built on joint testing of base and modified models. A probe set plus a classifier is the sort of test that structure could use to check whether an adapter’s behavior matches its stated purpose.

I wouldn’t push it further than that yet, and neither does the paper. It’s one base model at 0.8 billion parameters, LoRA only. The five domains are “deliberately well separated,” and the author expects finer distinctions, such as two medical sub-specialties, to be harder. With 36 runs the intervals are wide; the black-box result’s conservative interval runs from 41.7% to 86.1%. Two of the twenty planned datasets were dropped before any features reached a classifier, which leaves mathematics and multilingual with three datasets each. And a pilot result of 100% turned out, under a label-shuffle control, to be the classifier recognizing the same dataset across seeds, not learning domains at all. Re-identifying the exact dataset across seeds is close to perfect at every access level, 1.000 from weights and activations and 0.944 from outputs, against chance of 0.056. The author caught it and built the headline on leave-one-dataset-out instead, which a held-out dataset’s second seed can’t inflate.

That last caveat is the reason I trust the rest. The easy version of this paper reports the 100% and stops. This one has a frozen protocol, a hypothesis that failed in public, conservative intervals that sink one of its own results, and $30 of compute. The result is narrow: five tidy domains, one small model. Inside that scope, an adapter’s answers to questions about nothing told a classifier what it was trained on 23 times out of 36.

References

  1. Rane, S. (2026). Embedding-Space Fingerprinting of LoRA Fine-Tunes: Recovering the Training-Data Domain from the Model Artifact Alone. arXiv:2611.07610, submitted 5 November 2026.
  2. Shokri, R., Stronati, M., Song, C. and Shmatikov, V. (2017). Membership Inference Attacks Against Machine Learning Models. 2017 IEEE Symposium on Security and Privacy (SP), May 2017.
  3. Hacker, P. and Holweg, M. (2026). The regulation of fine-tuning: Federated compliance for modified general-purpose AI models. Computer Law & Security Review, April 2026 issue, online 2 December 2025.
  4. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L. and Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685, submitted 17 June 2021.