Same Data, Same Safety Training, Different Jailbreaks

Every jailbreak benchmark I have read tests one checkpoint. A team takes a released model, throws a few thousand adversarial prompts at it, counts the ones that get through and reports which attacks work. The unspoken assumption is that the result belongs to the model’s training: its data and its safety recipe. Retrain the same thing and you would get the same weak spots.

A preprint posted to arXiv on 25 October tests that assumption directly, and it does not hold up well. Chi-Wei Chang, Mu-Tien Kuo, Kuan-Wei Lee and Meng-Hsun Tsai, from Cerulean Labs, National Taiwan University and National Yang Ming Chiao Tung University, ask a narrow question: if you pretrain a second model on the same data with a different random seed and give it identical safety training, does it break on the same jailbreaks? Their answer, in the title, is “Identically Trained Language Models Break on Different Jailbreaks.”

The setup

The study builds on DataDecide, a 2025 suite that pretrains 1B-parameter models on 25 data recipes (DCLM, Dolma, FineWeb, Falcon, C4 and variants) with three pretraining seeds each, all for the same 100B tokens. In DataDecide the seed sets both the initialization and the data order. The authors took each of those 75 base models and gave it the same safety post-training, Tülu 3’s supervised fine-tuning followed by DPO on the OLMo 2 1B preference mix, under two post-training seeds. That makes 150 models.

Each model then answered the same prompt set, drawn from HarmBench, SORRY-Bench, StrongREJECT, XSTest, OR-Bench and WildJailbreak, with 20 sampled responses per prompt at temperature 1.0. The main analysis uses 3,212 adversarial-harmful prompts, each one a harmful request wrapped in a jailbreak. A model’s “vulnerability” on a prompt is the share of its coherent samples that do not refuse, judged by WildGuard.

How the 150 models were built and compared Study design. 25 pretraining data recipes from DataDecide, each pretrained with 3 seeds, give 75 base models of 1B parameters. Each base model gets the same Tülu 3 safety post-training under 2 post-training seeds, giving 150 models. Each model answers 3,212 adversarial-harmful prompts 20 times at temperature 1.0. Models are then compared in three kinds of pair: MZ twins share one pretraining run and differ only in post-training seed; same-recipe pairs share a data recipe but come from different pretraining runs; cross-recipe pairs share neither. STUDY DESIGN · CHANG ET AL., ARXIV 2610.34074 · 25 OCT 2026 25 data recipes DataDecide corpora 75 base models 3 pretraining seeds, 1B 150 models same SFT + DPO, 2 seeds 3,212 jailbreak prompts 20 samples each, temp 1.0 THREE KINDS OF PAIR MZ twins one pretraining run, two post-training seeds Same-recipe pair same data recipe, different pretraining run Cross-recipe pair different data recipe, different pretraining run The main data recipe also gets 4 post-training seeds and 150M and 300M runs; those are extra checks.
The design isolates one thing. MZ twins differ only in the order of their safety-training examples. Same-recipe pairs also differ in which pretraining run they came from, with the data held fixed.

The authors borrow a term from twin studies: two models built on the same base model are “MZ twins,” after monozygotic twins. Two models that share only a data recipe are a same-recipe pair. How much more alike the twins are than the same-recipe pairs is the paper’s main measure, which the authors call the ancestry gap.

They measure likeness as the correlation between two models’ per-prompt vulnerability, corrected for sampling noise. Twenty samples per prompt is not many, so even two identical models would not correlate perfectly. The fix is Spearman’s 1904 correction for attenuation, borrowed from psychology: divide the raw correlation by the square root of the product of each model’s own reliability, estimated by splitting its samples into odd and even halves.

The numbers

Averaged over the 25 recipes, MZ twins correlate at 0.984. Change the pretraining seed and the correlation falls to 0.811. Change the data recipe as well and it drops only to 0.779.

How alike two models' jailbreak profiles are, by shared ancestry Mean corrected per-prompt correlation of non-refusal rates. MZ twins, which share one pretraining run, correlate at 0.984 averaged over 25 data recipes. Same-seed reruns, retrained from the same seed on different hardware, correlate at 0.844 on the main data recipe. Same-recipe pairs from different pretraining runs correlate at 0.811. Cross-recipe pairs correlate at 0.779. The scale runs from 0 to 1. CORRECTED PER-PROMPT CORRELATION · 1B MODELS · CHANG ET AL. 2026 MZ twins Same-seed rerun Same-recipe pair Cross-recipe pair one pretraining run same seed, new GPUs same data, new run new data, new run 0.984, mean over 25 data recipes. Chang et al., arXiv 2610.34074, 25 Oct 2026 0.844, mean over 48 pairs, main data recipe. Chang et al., 25 Oct 2026 0.811, mean over 25 data recipes. Chang et al., 25 Oct 2026 0.779, mean over 25 data recipes. Chang et al., 25 Oct 2026 0.984 0.844 0.811 0.779 1.0 Rerun value is from the main data recipe only. Spearman-corrected for 20-sample noise.
Most of the drop happens when the pretraining run changes. Swapping the training data on top of that removes only another 0.032.

The authors put it as a ratio. Not sharing a base model lowers the correlation 5.4 times as much as not sharing a data recipe. That ordering holds in every one of the 25 recipes, with the per-recipe ancestry gap between 0.145 and 0.209. It holds at 150M and 300M parameters, where the gap is 0.204 and 0.211, against 0.171 for the main recipe at 1B. It holds when Tülu 3’s safety data is swapped for a different public mix (a gap of 0.178 against 0.175). And it holds in labels two of the authors wrote by hand on four models, 1,200 responses, blind to the judge and the model. The hand labels give a difference of 0.156 where the judge gives 0.203.

The rerun is the control that convinced me. The authors pretrained one DataDecide run twice more from its own seed, same initialization and same data order, on their H200 GPUs. If the seed were what made twins alike, these reruns should land near 0.98. They land at 0.844, only 0.030 above an ordinary same-recipe pair. Sharing only the initialization or only the data order adds nothing they could detect (−0.001 in both cases). What twins share is one particular run, with all the GPU nondeterminism that went into it, and fixing the seed does not reproduce it.

What carries over and what doesn’t

Correlations are abstract. What a red team wants to know is whether a result found on one checkpoint transfers to the next run of the same model.

Red-team findings carried from one model to another Three measures of carry-over. First, the attack template with the highest success rate on one model is still the best on an MZ twin in 67% of pairs, but on a same-recipe pair in only 27%. Second, a prompt that breaks one model at least nine times in ten is just as reliable against its MZ twin 75.0% of the time, a same-recipe partner 60.9% of the time and a cross-recipe partner 58.7% of the time. Third, when one template beats another by 10 to 20 percentage points on one model, an MZ twin reverses that lead 2.2% of the time, a same-recipe partner 25.3% and a cross-recipe partner 26.6%. CARRY-OVER BETWEEN TWO MODELS · BY KIND OF PAIR · CHANG ET AL. 2026 Best attack template is still the best Reliably breaking prompt (≥9 in 10) still reliable 10–20 point lead between two templates reversed MZ twin Same-recipe MZ twin Same-recipe Cross-recipe MZ twin Same-recipe Cross-recipe 67% of pairs. Chang et al., arXiv 2610.34074, 25 Oct 2026 27% of pairs. Chang et al., 25 Oct 2026 75.0%. Chang et al., 25 Oct 2026 60.9%. Chang et al., 25 Oct 2026 58.7%. Chang et al., 25 Oct 2026 2.2%. Chang et al., 25 Oct 2026 25.3%. Chang et al., 25 Oct 2026 26.6%. Chang et al., 25 Oct 2026 67% 27% 75.0% 60.9% 58.7% 2.2% 25.3% 26.6%
Under a new pretraining run, the top attack template is replaced about three times in four, and a solid lead of 10 to 20 points flips about one time in four. Between twins it flips 2.2% of the time.

The best of HarmBench’s 91 human-written attack templates on one model stays the best on its twin in 67% of pairs and on another run of the same data in 27%. A template that beats another by 10 to 20 points on one model loses that lead on a same-recipe partner 25.3% of the time. Even a lead above 20 points is reversed 8.9% of the time across runs, against 0.05% across twins. The authors also note that the binning of leads was chosen after seeing the data, so I’d treat those exact cut-offs as exploratory.

A second result matters more to me than the headline. Which prompts break a model follows the run. How often a model breaks follows the data. When the authors split the variance in each model’s mean vulnerability, the data recipe accounts for 49%, the base model within its recipe for 32% and the post-training seed for 19%.

The data sets how often a model breaks; the run sets where Two panels. Top: share of the variance in models' mean vulnerability on adversarial-harmful prompts. The data recipe accounts for 49%, the base model within its recipe for 32% and the post-training seed for 19%. Bottom: what sharing each factor adds to the per-prompt correlation. Sharing a pretraining run adds 0.173, the ancestry gap. Sharing a data recipe adds 0.032, the difference between same-recipe and cross-recipe pairs. HOW OFTEN VS WHICH PROMPTS · 25 RECIPES, 1B · CHANG ET AL. 2026 How often it breaks: share of variance in mean vulnerability Data recipe Base model (the run) Post-training seed 49%, interval 12% to 70%. Chang et al., arXiv 2610.34074, 25 Oct 2026 32%, interval 16% to 59%. Chang et al., 25 Oct 2026 19%, interval 11% to 37%. Chang et al., 25 Oct 2026 49% 32% 19% Which prompts break it: correlation added by sharing A pretraining run A data recipe 0.173, interval 0.167 to 0.179. Chang et al., 25 Oct 2026 0.032, interval 0.024 to 0.043. Chang et al., 25 Oct 2026 +0.173 +0.032 Panels use different scales: percent of variance on top, correlation units (0 to 0.2) below.
Data is the biggest single factor in how vulnerable a model is overall. It adds very little to which prompts get through. That part comes from the run.

Even the “how often” number is shaky on one checkpoint. Compare two recipes using one model each, and the ranking by mean vulnerability comes out reversed in 23.4% of comparisons, relative to the ranking from all six of each recipe’s models. A single model estimates its recipe’s mean vulnerability with a standard error of 3.44 percentage points. The recipes’ means themselves have a standard deviation of 3.5 points. The noise from using one model is about as large as the differences being measured. Three base models bring the error down to 1.99 points.

Each run leaves a fingerprint

The flip side of runs disagreeing is that each run is recognizable. The authors took each run’s model at post-training seed 1 as a reference and its model at seed 2 as a query, then matched each query to whichever reference its vulnerability profile correlated with best. Across 3,211 prompts, the match found the right run for all 75 queries, where chance is one in 75. With a random 50 prompts it still found the right run 98% of the time. Asked to identify the data recipe of a base model it had never seen, the same method got 24 of 50 right.

That fingerprint is a security property as much as a statistical one. Fifty jailbreak prompts and their refusal rates are enough to tell which of 75 near-identical training runs produced a model, at least at this scale and after a post-training change no bigger than a reshuffle.

None of this is new outside safety

Machine learning has known about seed variance for a long time. A 1999 paper in IEEE Transactions on Neural Networks noted that training was “conventionally repeated with ten, or so, random starting values” and worked out how many repetitions are needed for the best one to be good with a stated confidence. In 2020 McCoy, Min and Linzen fine-tuned 100 copies of BERT on MNLI. Every copy scored between 83.6% and 84.8% on the development set, but on one syntactic test (subject-object swaps) accuracy ranged from 0.0% to 66.2%. Zhong, Ghosh, Klein and Steinhardt found in 2021 that instance-level predictions are “highly sensitive to noise in the randomness in training” and recommended that researchers publish model predictions next to the weights.

What the new paper adds is the safety case, plus a design that separates the run from its data. The authors say no earlier safety study had varied a whole pretraining run, and I haven’t seen one either. Seed variance has usually been treated as noise to average away. In a jailbreak evaluation it is the result: the specific prompts a model fails on are partly an accident of one run.

Limits

The authors list most of these themselves. Every model comes from DataDecide and Tülu 3’s SFT and DPO stages, so this is one post-training recipe tested on one family. The largest model has 1B parameters. The authors explain why: the design needs several full pretraining runs of the same data, and at 7B, they note, a single run took about 15,000 A100-hours in earlier work. No one knows whether the gap shrinks or grows at frontier scale, where a lab trains one big run and so cannot measure run-to-run variance either.

The judge is also weak. WildGuard misses refusals on jailbreak prompts: it puts mean vulnerability at 0.528, and the authors’ audit adjusts that to 0.278. The human-label check covers only four models. Sampling at temperature 1.0 with a 512-token cap is a reasonable stress setting, but not how most deployments run.

None of that undercuts the core design. The rerun control in particular rules out the obvious objections. A shared seed, data order or initialization does not explain the twins’ similarity; being the same run does.

What I’d take from it

A jailbreak found on a checkpoint is evidence about that checkpoint. Its weight as evidence about the training recipe, or the next model from the same lab, is lower than red-team reports imply.

For labs, patching specific discovered prompts is a fix to one run. A retrain on the same data with the same safety mix will have a different set of holes, about as different as a model trained on other data, so a jailbreak list from the previous checkpoint is a weak regression test. For people comparing models, a single checkpoint is a sample of one. The safety equivalent of a benchmark score, a refusal rate or a ranking of attacks, needs a stated spread, and at this scale three runs cut the error almost in half. I made a related point about Mistral Large 4’s refusal-driven cyber score in an earlier post, where a single public checkpoint was all anyone had.

The authors released code and the weights of their safety-trained models, with per-prompt judge labels. Whether the gap holds at 7B and beyond is still open, and the release is built so that someone with the compute can check.

References

  1. Chang, C.-W., Kuo, M.-T., Lee, K.-W. and Tsai, M.-H. (2026). Identically Trained Language Models Break on Different Jailbreaks. arXiv:2610.34074, submitted 25 October 2026.
  2. Magnusson, I. et al. (2025). DataDecide: How to Predict Best Pretraining Data with Small Experiments. arXiv:2504.11393, submitted 15 April 2025.
  3. Spearman, C. (1904). The Proof and Measurement of Association between Two Things. The American Journal of Psychology, 15(1), p. 72, January 1904.
  4. Iyer, M. S. and Rhinehart, R. R. (1999). A method to determine the required number of neural-network training repetitions. IEEE Transactions on Neural Networks, 10(2), pp. 427-432, March 1999.
  5. McCoy, R. T., Min, J. and Linzen, T. (2020). BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance. Proceedings of the Third BlackboxNLP Workshop, pp. 217-227, 2020.
  6. Zhong, R., Ghosh, D., Klein, D. and Steinhardt, J. (2021). Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3813-3827, 2021.