On 16 November eleven researchers from UIUC, CMU, Rutgers, Princeton, Stanford and Yale posted “Language Models Can Rewrite Their Own Weights” to arXiv, with a revision on 18 November. The setup follows the MIT experiment I wrote about earlier this month, where GPT-6 Astra edited the weights of a much smaller Qwen model. Here the editor and the edited model are the same. Qwen3.5-27B gets its own weights in a Python session with no GPU and no gradients, its graded answers on practice math problems, and 20 forward passes it can spend running itself. It then writes an edit. The abstract’s headline: “its own edit makes it 6.6 points more accurate on problems it never saw.”
I read the full paper, including the long appendix, because the abstract makes three claims that can be checked separately: the model improves itself, it “keeps getting better,” and the ability “emerges with scale.” The first one holds. What the improvement consists of is narrower than the framing suggests, and the second claim comes down to the order in which the sessions happened to run.
The test is good
The 6.6 points come from 490 held-out problems, each answered eight times at a 16,384-token budget, for 3,920 answers per model. The base model and the edited model use the same seeds, so the answers form pairs. The test is a one-sided sign-flip permutation test over problems, chosen because eight answers to one problem may be correlated, and the paper says it was written down, with the analysis code, before the evaluation ran. For the winning edit, 439 answers are right only with the edit and 182 only without it. That’s p below 10⁻⁵. The authors also report all eight sessions of the protocol that produced it, including the five where the model withdrew. With that much disclosure, I take the number as real. What it measures is the open question.
What the edit was
Table 13 of the appendix lists each submitted edit in the model’s own terms. The 6.6-point edit, from session 8, is: “time-step bias +0.5 in all 48 linear-attention layers.” That’s one scalar added uniformly across every layer of one type. In the model’s own account it makes those layers decay faster, so that long reasoning gets stuck less. The main text says plainly: “The larger gain comes from one uniform change: it raises the time-step bias of every linear-attention layer by 0.5, and with it fewer of its answers run out of tokens.”
An answer whose thinking is still open at the budget is graded wrong, and before the edit 33.6 percent of the base model’s held-out answers ran out of tokens. After session 8’s edit, 26.1 percent did. After session 2’s edit, which subtracted a measured direction from two output matrices in layer 63, 28.8 percent did.
I split each model’s answers into three groups using the paper’s numbers: right, finished but wrong, and cut off at the limit. The middle group is just what’s left over.
The right-hand column is my arithmetic, not the paper’s, and it assumes every cut-off answer is graded wrong, which is the paper’s grading rule. Before the edit, 86.1 percent of the answers that finished were right. After session 8’s edit, 86.3 percent were. After session 2’s, 86.1 percent. If the edit had only rescued looping answers and those answers were right at the same rate as the rest, accuracy would go from 57.2 to about 63.6. The paper reports 63.8.
The result is still worth something. A model that loops on a third of its answers has a real, costly defect, and a weight change that fixes part of it without a training run is useful. It’s also the same pattern the MIT edit showed on the 0.8B model, at a smaller scale. But “6.6 points more accurate on problems it never saw” invites you to read it as better math. The data describe a model that stops looping a bit more often and is no better at the problems it does finish.
A knob the rules didn’t cover
The paper’s constraints section lists what the Python session refuses to apply. It refuses edits to the input embedding, the last normalisation and the output layer, “and edits that only rescale a whole tensor, which would act like a change of sampling temperature.” The authors saw that a global knob could pass for self-improvement and blocked one kind of it.
Adding 0.5 to the time-step bias of all 48 linear-attention layers isn’t a rescale, so it got through. But it’s a global setting in the same family: one number, applied everywhere, that changes how fast the model’s recurrent state decays. Session 1 made an edit of the same kind, +3.0 on the same bias in seven layers, and it gained 0.5 points.
So the comparison I’d want is the boring one. Sweep that one bias over a handful of values, spend the same 20 forward passes checking them, and see where you land. And separately, give the base model more than 16,384 tokens, or a repetition penalty, and see how many of the cut-off answers turn into right ones. The paper has neither. It compares the edited model with the unedited one and with smaller models doing the same task.
Looping isn’t a new failure, and neither are the cheap fixes for it. In their survey of decoding methods, Sina Zarrieß, Henrik Voigt and Simeon Schüz trace a long line of work on repetitive output from likelihood-based decoding and on decoding-side remedies, from sampling to explicit repetition penalties, that reduce it without touching the weights at all.
When Nick McGreivy and Ammar Hakim reviewed machine-learning papers that claimed to beat standard numerical solvers for fluid equations, they found that “79% (60/76) make a comparison with a weak baseline,” and that “the more impressive the result, the more likely the article used a weak baseline.” Their field is different, but the mechanism is the same. If you never run the obvious simple method, you can’t say how much of the gain needed the clever one.
“It keeps getting better”
The abstract says the model “keeps getting better: it tests its edits on practice problems and submits only what works, each edit improving on the last, and over eight consecutive sessions its accuracy only rises.” Appendix E.9 explains how that works. Every session starts from the base weights. Each submitted edit replaces whatever model is currently “in use.” Three of the eight sessions submitted: session 1 (+0.5), session 2 (+4.0) and session 8 (+6.6).
Those three edits are independent. None of them builds on another. “The model in use only rises” is true because the three results happened to arrive in increasing order. Run the same eight sessions in reverse and the model in use ends at session 1’s 57.8 percent, below where session 2 had put it.
Any three distinct values arrive in increasing order one time in six. That isn’t rare enough to need explaining, but it’s not a trend either. A claim that the model compounds its improvements would need sessions that start from the last submitted edit, and these don’t.
The check couldn’t tell them apart
Before submitting, each session in this protocol gets one check from the framework: 256 reserved practice problems, two answers each, at 4,096 tokens, edited against unedited. The model reads the result and decides to submit or withdraw. This is the step the paper credits for the model “submits only what works.”
Table 13 shows what the check said and what the held-out test later found.
Sessions 1 and 8 checked identically, +2.5 with a standard error of 1.2. On the held-out set, session 1 gained 0.5 points, not significant, and session 8 gained 6.6. Session 2 had the best check, +6.6, and came second on held-out at +4.0. With 512 answers at a quarter of the final token budget, the check can tell a clear loss from a small gain. It can’t rank edits, and it said nothing about which session would produce the headline.
None of this undermines the held-out test, which is separate and much larger. It does weaken the story that the model knows which of its edits work. And it explains why the paper’s evaluation design deserves the credit it gets. Statisticians call the pattern in the check column the winner’s curse: an estimate picked because it looked best on a small sample tends to come in lower when retested. Isaiah Andrews, Toru Kitagawa and Adam McCloskey describe it as arising “where conventional estimates are biased and conventional confidence intervals are unreliable” whenever the target was chosen by optimization. In a 2023 note in Anaesthesia, Sidebotham and Barlow tell the clinical version. A trial reported 57 and 58 percent survival at higher haemofiltration doses against 41 percent at the lowest, and a larger trial later found 44.7 percent mortality in both arms. Their point is that this shrinkage “is an expected consequence of null hypothesis significance testing.” Session 2 is that pattern on a small scale, best on the check and lower when retested. Session 8 went the other way, which is also what a noisy check does.
What the first protocol said
One more number from the paper belongs next to the headline. Before this protocol there was a first one, without the reserved-problem check, and its edit was the one the evaluation was originally fixed in advance for. On the same 490 problems it raised accuracy from 57.2 to 58.3 percent, +1.1 points, p = 0.038. The authors report it plainly in Table 8, and they say the second protocol’s edits were evaluated “on the same 490 problems, against the same answers of the base model and with the same permutation test.” That’s fair. The 490 problems never reached a session, so reusing them doesn’t leak anything. Still, the abstract’s 6.6 is the best of three submitted edits from the second protocol, and the first protocol’s edit gained 1.1. Both belong in the same sentence.
The scale claim
“The ability emerges with scale” rests on one session per size for the 2B, 4B and 9B models, plus the 27B sessions. The 2B model wrote “Since I’m a text model without actual neural network weights, I cannot directly edit weights.” The 4B and 9B models added random noise and broke themselves. The 27B model measured a direction in its hidden states and edited along it. The appendix also says one 9B session was stopped after 55 turns because it was editing “like the 4B model,” and a full-length 9B session is reported instead. I believe the qualitative difference, since the transcripts show it clearly. But one session per size is a sample of one, and the same paper shows a single 27B setup producing +0.5 or +6.6 depending on the seed of the model’s turns.
What I’d keep
The repair experiment is the part I find most convincing. Eight rows of one matrix in layer 30 overwritten with noise, the model told only that “some values were corrupted,” and it finds the matrix by measuring every output matrix and spotting a “1397x outlier.” Clipping that one matrix alone takes the damaged copy from 0 to 55 percent on 200 held-out problems. That’s a diagnosis with a single known cause, and the model found it.
The self-improvement result is real too, and the authors were more careful in evaluating it than most papers that claim larger gains. It’s narrower than the abstract. One global bias, +0.5, makes a 27B model loop less inside a 16,384-token budget. The extra finished answers are right at the same 86 percent rate as the rest, and nobody has yet checked whether a one-line sweep of that bias would have found the same thing. “The wall between training and inference is a limit of our tools, not a property of intelligence,” the authors write. Maybe. Before I’d take that from this paper, I’d want to see the edited model get better at problems it already finishes.
References
- Fan, J., He, N., Yang, C., Mao, A., Zhu, X., Cheng, C., Liang, K., Shen, S., Li, Z., Guo, R. and Liu, G. (2026). Language Models Can Rewrite Their Own Weights. arXiv:2611.21885, v1 16 November 2026, v2 18 November 2026. Main text Figs. 1-6 and Sec. 3.6; Tables 2, 7, 8 and 13; Appendices D.3 and E.9.
- Takehi, R., Teoh, J., Zhang, A. and Khattab, O. (2026). Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights. Blog post, November 2026.
- McGreivy, N. and Hakim, A. (2024). Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations. Nature Machine Intelligence, 6(10), 1256-1269, 25 September 2024.
- Andrews, I., Kitagawa, T. and McCloskey, A. (2024). Inference on Winners. The Quarterly Journal of Economics, 139(1), 305-358, published online 18 September 2023.
- Sidebotham, D. and Barlow, C. J. (2024). The winner’s curse: why large effect sizes in discovery trials always get smaller and often disappear completely. Anaesthesia, 79(1), 86-90, published online 27 October 2023.
- Zarrieß, S., Voigt, H. and Schüz, S. (2021). Decoding Methods in Neural Language Generation: A Survey. Information, 12(9), 355, 30 August 2021.