Two Percent of the Weights, None of Them Trained

On 30 October a project called nesformer reached the Hacker News front page under the title “Show HN: A working Nintendo in ~1GB of Qwen weights.” The README’s first line is more exact: “An NES CPU compiled into the weights of a stock Qwen3.5 model.” A few lines further down it says: “Nobody trained these weights; a compiler computed every one.”

I read the README and the blog post it builds on, and I checked the numbers against each other. Most of the headline survives, though not in the form the title implies. The checkpoint is about a gigabyte, but 98 percent of it is zeros. The CPU really does run in the model, but the picture is drawn by an ordinary program outside it. What surprised me was where the game itself sits. It gets loaded into the model’s recurrent state at startup, outside the weights, and nobody auditing a model file would think to look there.

What actually runs where

The NES has a 6502-family CPU, 2 KB of work RAM, a picture processor (the PPU) and a sound unit. nesformer puts the CPU and the RAM in the model. According to the README the model covers “all 151 official opcodes, every addressing mode, flags, stack, BRK and NMI entry,” together with 2 KB of work RAM and 32 KB of cartridge program ROM, “stored twice: one copy for instruction fetch, one for data reads.”

Everything else is a host program. The PPU is “a scanline renderer and the palette.” The controllers, the vblank interrupt timing and the answers to I/O reads come from the host too, and they enter the model as injected IN and NMI tokens. Going the other way, the model emits the CPU’s execution log one token at a time, and the host draws frames from that log. The README is careful about the boundary: “The host never sees the CPU’s registers or RAM except through the trace.”

What nesformer puts in the model and what it leaves to the host Two panels. Inside the Qwen3.5 checkpoint and its Gated DeltaNet state: the 6502 core with all 151 official opcodes, 2 KB of work RAM, and 32 KB of cartridge program ROM stored twice. In the host program: the PPU scanline renderer and palette, the controllers, NMI timing at vblank, and answers to I/O reads. The model sends execution-trace tokens to the host; the host injects IN and NMI tokens into the model. Sound registers are written but otherwise ignored. THE SPLIT · NESFORMER README, 30 OCT 2026 IN THE CHECKPOINT AND ITS DELTANET STATE IN THE HOST PROGRAM trace tokens IN, NMI tokens 6502 core 151 official opcodes, flags, stack, BRK, NMI 2 KB work RAM rows of the DeltaNet state matrices 32 KB cartridge ROM, stored twice one copy for fetch, one for data reads PPU scanline renderer, palette Controllers, I/O answers injected as IN tokens Vblank timing injected as an NMI token Sound registers are written but otherwise ignored. There is no audio, and mid-scanline effects are not reproduced.
The model is the CPU and its memory. The parts you see and hear are drawn by an ordinary program reading the CPU's log.

I don’t count that against the project. The CPU is the part that has to be exact on every instruction, and the project says plainly which half is which. But “a working Nintendo” means the NES’s processor and RAM, not the whole console.

The trick: a recurrent state used as RAM

nesformer reuses the compiler and the central idea from Qwenputer, which Divyansh Agrawal described on 22 October. Qwen3.5 mixes ordinary attention layers with Gated DeltaNet layers, a kind of linear recurrent network. Each DeltaNet head carries a state matrix, which in nesformer’s case is 128 × 128, and updates it at every token. Agrawal found that with the forget gate fixed at 1, the write strength fixed at 1 and one-hot keys, “every write we perform will overwrite exactly one row, and every one-hot query can retrieve an exact value.” The nesformer README puts the consequence briefly: “every row is a memory cell, and the cartridge, RAM and registers all live in rows.”

Agrawal credits the underlying idea to Schlag, Irie and Schmidhuber, who showed in 2021 that linearised attention is formally equivalent to the “fast weight” controllers of the early 1990s, and who replaced its purely additive updates with “a delta rule-like programming instruction.” Their delta-rule memory was something a network learned to use. Agrawal’s own summary of what’s new: “making it exact and addressable, inside a production model.”

Compiling programs into networks is older still. In 1995 Frédéric Gruau, Jean-Yves Ratajszczak and Gilles Wiber described a compiler whose “input of the compiler is a PASCAL Program” and which “produces a neural network that computes what is specified by the PASCAL program.” DeepMind’s Tracr (2023) compiled human-readable programs into “standard decoder-only transformer models” so that their known structure “can serve as ground-truth for evaluating interpretability methods.” Also in 2023, Giannou and colleagues programmed a looped, 13-layer transformer to emulate “a small instruction-set computer.”

Programs compiled into neural networks, 1995 to 2026 A timeline of six milestones, not drawn to scale. April 1995: Gruau, Ratajszczak and Wiber's neural compiler turns PASCAL programs into networks. February 2021: Schlag, Irie and Schmidhuber show linear attention is a fast weight programmer and add a delta-rule update. January 2023: Tracr compiles programs into decoder-only transformers. January 2023: Giannou and colleagues emulate a small instruction-set computer with a looped 13-layer transformer. 22 October 2026: Qwenputer compiles a RISC-V CPU into Qwen3.5-4B, using DeltaNet state as RAM. 30 October 2026: nesformer compiles an NES 6502 CPU into a Qwen3.5 checkpoint. COMPILED, NOT TRAINED · SELECTED WORK, NOT TO SCALE April 1995: A neural compiler — Theoretical Computer Science 141 February 2021: Linear Transformers Are Secretly Fast Weight Programmers — arXiv 2102.11174 January 2023: Tracr — arXiv 2301.05062 January 2023: Looped Transformers as Programmable Computers — arXiv 2301.13196 22 October 2026: Qwenputer — divagr.com 30 October 2026: nesformer — GitHub README and Show HN 1995 Neural compiler: PASCAL to network 2021 Delta-rule fast weight memory Jan 2023 Tracr: programs to transformers Jan 2023 Looped 13-layer transformer as computer 22 Oct 2026 Qwenputer: RISC-V in Qwen3.5-4B 30 Oct 2026 nesformer: 6502 in a Qwen3.5 file The 2026 steps differ from the earlier ones in one way: they target a production model's own checkpoint format.
The idea is thirty years old. What's new is the target: an off-the-shelf model format that loads in stock code.

About a gigabyte, mostly zeros

The README’s table of the checkpoint answers the “~1GB” in the title. It’s a Qwen3_5ForCausalLM with seven Gated DeltaNet layers that hold the circuit, plus “1 all-zero attention layer that Hugging Face’s cache needs.” It uses Qwen3.5-4B’s DeltaNet head shapes and its MLP width of 9,216, with a hidden size of 768 and a vocabulary of 263,808 tokens. The total is 466.6 million parameters in bf16, or 933 MB, “of which 9.3M are nonzero, so the weights zip to 3.3 MB.”

Nine point three million out of 466.6 million is just under 2 percent. Most of the gigabyte is zeros that keep the layers in Qwen3.5-4B’s shapes, plus a whole attention layer that is nothing but zeros. Those zeros are what let the README say the folder “loads with Qwen3_5ForCausalLM.from_pretrained and needs no custom code.”

How much of the nesformer checkpoint is actually used Two pairs of bars on a linear scale. Parameters: 466.6 million in total, of which 9.3 million are nonzero, just under 2 percent. Storage: the checkpoint is 933 MB in bf16, and it compresses to 3.3 MB as a zip file. The nonzero and zipped bars are slivers next to the full ones. THE "~1GB" · NESFORMER CHECKPOINT TABLE, 30 OCT 2026 PARAMETERS STORAGE All parameters Nonzero Checkpoint, bf16 Zipped 466.6M parameters — nesformer README, 30 Oct 2026 9.3M nonzero — nesformer README, 30 Oct 2026 933 MB in bf16 — nesformer README, 30 Oct 2026 3.3 MB zipped — nesformer README, 30 Oct 2026 466.6M 9.3M, about 2% 933 MB 3.3 MB Bars are to linear scale; the zipped bar is drawn at 2px so it stays visible.
The file is sized like a real Qwen3.5 layer stack so stock code will load it. The machine itself fits in a few megabytes.

Checked against an emulator, and slow

The README’s verification is the most convincing part. The reference CPU matches nestest’s golden log for 3,348 instructions, and matches the py65 emulator on 5.7 million instructions of game code. The compiled circuit predicts every reference token across nestest plus 283,610 positions of two games’ boot sequences. Run free, with nothing correcting it, the weights produce traces identical to the reference: 700,000 tokens over 271 frames of the homebrew game Thwaite in check mode, 1,916,527 tokens over 600 Thwaite frames, and 2,645,885 tokens over 420 frames of Full Game, with every saved frame matching.

That is the right way to test a claim like this. A trained model gets judged on a benchmark. A compiled CPU can be diffed, token by token, against a machine whose correct output is already known.

Speed is where it stops being a console. The fast engine, compiled with numba, “ran at about 1,650 tokens/s on one CPU core,” the plain numpy engine about 210, and stock transformers about 10. “A real NES would need about 700,000 tokens per second.” By my arithmetic that’s a factor of roughly 424 at full token count. The host also skips idle wait loops, which “cuts the token count by 2.6× to 4.2×,” and the README puts the overall result at “150–290× slower than the real console.” A frame takes one to six seconds, so ten seconds of play takes around 25 minutes.

nesformer decoding speed against what a real NES needs Horizontal bars on a logarithmic scale from 1 to 1 million tokens per second. Stock transformers: about 10 tokens per second. Plain numpy engine: about 210. Numba engine on one CPU core: about 1,650. A real NES would need about 700,000 tokens per second. With idle-loop skipping, the README puts the overall result at 150 to 290 times slower than the console. TOKENS PER SECOND · LOG SCALE · NESFORMER README, 30 OCT 2026 1 10 100 1k 10k 100k 1M Stock transformers Plain numpy engine Numba, one CPU core Real NES would need about 10 tokens/s — nesformer README, 30 Oct 2026 about 210 tokens/s — nesformer README, 30 Oct 2026 about 1,650 tokens/s — nesformer README, 30 Oct 2026 about 700,000 tokens/s — nesformer README, 30 Oct 2026 ~10 ~210 ~1,650 ~700,000 Skipping idle loops cuts tokens 2.6× to 4.2×; the README's overall figure is 150 to 290 times slower than the console.
Even the fastest engine is two to three orders of magnitude short. Nobody will play games this way; what the run shows is that every token comes out exact.

The cartridge is in the cache

The detail I keep coming back to is in the section on playing your own ROMs: “The CLI loads the ROM into the model’s state at startup, so the same weights run any NROM game without a rebuild.” The run procedure in the README says the same thing: “write the machine’s state into the cache’s DeltaNet states, then feed tokens one at a time.”

So the weights are the hardware. They define a 6502 and nothing more. Which program runs, whether Super Mario Bros., Donkey Kong or Thwaite, is decided by what gets written into the recurrent state before decoding starts. With identical weights and a different state, you get a different computation.

That turns the interpretability story around. Tracr’s pitch was that a compiled model has a known structure, so it can serve as ground truth when you test whether an interpretability method works. Follow-up work has shown how readable that structure is: Hannes Thurnherr and Kaspar Riesen trained a model to recover the source programs from Tracr-compiled weights, and it reproduced “more than 30%” of test programs exactly, with more than 70 percent functionally equivalent. Reading the weights recovers the program in that setting because the weights are the program. In nesformer they’re only the machine. You could decompile the checkpoint perfectly and learn that it’s a 6502, without learning anything about what it’ll execute.

Agrawal has already pointed at the uncomfortable version. The Qwenputer post links two follow-ups: one on “how hybrid models can learn to hide their reasoning in their RNN state and evade CoT monitoring,” and one on how compiled weights “can work as inbuilt monitoring tools that the model itself can’t tamper with.” I haven’t evaluated either, and they make claims that need their own checking. But both rest on the same fact nesformer demonstrates cleanly: in a hybrid model, the recurrent state can hold exact, addressable data that never appears in the weights or in the visible text.

Qwenputer also shows the other half of the claim, that a compiled machine can sit next to a trained model without changing how it talks. Agrawal reports that Qwen3.5-4B’s normal text stays “within noise of base Qwen (KL 6e-4, against a noise floor of 5e-4),” and that about 230,000 replayed CPU output positions had zero mismatches against a reference machine. Combine that with nesformer’s stock checkpoint that needs no custom code. A model file can load like any other and talk like its base model while carrying an exact computer, and the program that computer runs arrives through the state.

What I take from it

As a demo, nesformer is honest. The README gives the host/model split, the zero count, the slowdown and the limits: NROM cartridges only, official opcodes only, no IRQs, no sound. Its verification is the kind I wish trained-model claims could offer, a free-running diff against a reference that ends with every saved frame matching.

As evidence about models in general, I’d take away two things. First, “stock checkpoint” now describes a file format, not what the file does. A folder that from_pretrained loads without complaint can hold weights nobody trained. Second, a hybrid model’s behavior depends on more than its weights. Anyone building tools that audit, sign or scan model files should assume the state injected at runtime is part of the program, because here it’s the entire program.

References

  1. nesformer (2026). nesformer: An NES CPU compiled into the weights of a stock Qwen3.5 model. GitHub repository README, accessed 31 October 2026.
  2. Hacker News (2026). Show HN: A working Nintendo in ~1GB of Qwen weights. Posted 30 October 2026.
  3. Agrawal, D. (2026). Qwenputer: a fully functioning, deterministic RISC-V CPU inside Qwen3.5. divagr.com, 22 October 2026.
  4. Schlag, I., Irie, K. and Schmidhuber, J. (2021). Linear Transformers Are Secretly Fast Weight Programmers. arXiv:2102.11174, February 2021 (ICML 2021).
  5. Gruau, F., Ratajszczak, J.-Y. and Wiber, G. (1995). A neural compiler. Theoretical Computer Science, 141, 1–52, April 1995.
  6. Lindner, D., Kramár, J., Farquhar, S., Rahtz, M., McGrath, T. and Mikulik, V. (2023). Tracr: Compiled Transformers as a Laboratory for Interpretability. arXiv:2301.05062, January 2023.
  7. Giannou, A., Rajput, S., Sohn, J., Lee, K., Lee, J. D. and Papailiopoulos, D. (2023). Looped Transformers as Programmable Computers. arXiv:2301.13196, January 2023.
  8. Thurnherr, H. and Riesen, K. (2024). Neural Decompiling of Tracr Transformers. Artificial Neural Networks in Pattern Recognition, Lecture Notes in Computer Science, 25–36, 2024.