The Score Survived Compaction. About One Rule in Three Did Not.

A paper posted to arXiv on 22 October by Jeanne Salle, Hardik Bhatnagar, Matthias Bethge, Sahar Abdelnabi and Maksym Andriushchenko, from the ELLIS Institute Tübingen, the Max Planck Institute for Intelligent Systems and the Tübingen AI Center, is called “Benchmarks Are Surprisingly Compaction-Friendly.” The title is the finding, and the word doing the work is “friendly.” When a coding agent runs out of context window, the scaffold asks it to summarize what it has done, throws the history away and continues from the summary. The authors wanted to know what that costs. On the three benchmarks they ran, the answer was close to nothing. The benchmark score depends on things the agent can re-read from its own files, so a summary that forgets them is harmless. What the summary cannot recover from disk is the rules the agent was given, and those go missing at a rate the score never reports.

I have written before about harnesses as the thing that moves the bill while the scoreboard stays put, in the HarnessTax piece and in the Harness² one. This paper is the same lesson from inside a single scaffold, and it comes with a number for what the scoreboard misses.

What they ran

The setup is narrow on purpose. One scaffold, OpenCode, with five compaction strategies implemented inside it so that nothing else varies. Four models: Claude Opus 4.6 and Gemini 3.1 Pro Preview with 1M-token windows, GPT-5.4 with 1,050K, and GPT-5.3-Codex, “the oldest model we tested and the one with the smallest context window (400K).” Three benchmarks where runs naturally outgrow a context window: PaperBench, which asks the agent to reimplement an ICML 2024 paper from scratch; ProgramBench, which hands it an execute-only reference binary and its documentation and asks for a rebuild that passes the behavioural tests; and PostTrainBench, which gives it ten hours on one H100 to post-train a small base model. The paper evaluates three of PaperBench’s 20 papers, two ProgramBench tasks (the terminal file managers broot and nnn, plus lua for the erosion analysis), and six PostTrainBench combinations. Two runs per task and configuration.

The compaction trigger was swept from 160K to 380K tokens, the same absolute thresholds for every model. For the 1M models that is 16 to 38 percent of the window, which the authors note “does not test compaction near the scaffold default” of 90 to 95 percent. For Codex it is 40 to 95 percent.

The score barely moved

For Opus 4.6 and GPT-5.4, task performance does not measurably change across that sweep. No alternative strategy beat the default of having the model write its own summary, including the one that writes the entire pre-compaction context to a JSON file so that nothing is actually lost. Codex is the exception: on PaperBench it “goes from 0.38 at 160K to about 0.60 at 260K and above, where it matches the other models,” and in the appendix it is the only model whose accuracy declines with the number of compactions, “from about 8 compactions onward.” On ProgramBench, where the agent’s state lives almost entirely in files, even Codex is flat.

The strategies that discard more are where the models part ways. Having Claude Haiku 4.5 write the summary leaves Opus 4.6 within its confidence interval and drops GPT-5.4 from 0.59 to 0.43 and Codex from 0.38 to 0.18 on PaperBench. Removing reasoning blocks before summarizing drops GPT-5.4 by the same 16 points and leaves Opus and Codex alone. The paper’s phrase for this is that “a compaction method validated on one model does not necessarily transfer to another,” which I would put on the wall of any team that tunes a harness against one provider.

One thing the sweep shows that the headline does not is how different “the same threshold” is across models. Table 2 counts compactions per run in the baseline condition.

Mean compactions per run at the 160K and 380K thresholds, four models A dumbbell chart on a 0 to 30 scale. GPT-5.3-Codex averages 27.71 compactions per run at the 160K threshold and 0.76 at 380K. GPT-5.4 averages 15.08 at 160K and 1.96 at 380K. Claude Opus 4.6 averages 4.90 at 160K and 0.71 at 380K. Gemini 3.1 Pro Preview averages 1.00 at 160K and 0.40 at 380K. Filled dots are the 160K threshold, hollow dots the 380K threshold. COMPACTIONS PER RUN · BASELINE SELF-SUMMARIZATION · SALLE ET AL., ARXIV 2610.31778, TABLE 2 0 10 20 30 GPT-5.3-Codex GPT-5.4 Claude Opus 4.6 Gemini 3.1 Pro Preview 0.76 compactions per run at 380K — Table 2, arXiv 2610.31778 1.96 compactions per run at 380K — Table 2, arXiv 2610.31778 0.71 compactions per run at 380K — Table 2, arXiv 2610.31778 0.40 compactions per run at 380K — Table 2, arXiv 2610.31778 27.71 compactions per run at 160K — Table 2, arXiv 2610.31778 15.08 compactions per run at 160K — Table 2, arXiv 2610.31778 4.90 compactions per run at 160K — Table 2, arXiv 2610.31778 1.00 compactions per run at 160K — Table 2, arXiv 2610.31778 27.71 15.08 4.90 1.00 0.76 1.96 0.71 0.40 Filled: 160K threshold. Hollow: 380K. Means over runs; the 95% intervals are wide (Codex ±17.00 at 160K). Rows ordered by the 160K value.
At the lowest threshold Codex compacts about 28 times a run and Opus 4.6 about five. The same threshold means a different number of summaries for each model.

At 160K, Codex averages 27.71 compactions per run and Opus 4.6 averages 4.90. Gemini averages one, because its context grows slowly: its median tool output is 26 characters against 918 for Opus, it takes about half as many steps, and it often stops early. The authors say plainly that Gemini’s results “are not informative about the effects of compaction,” and I would add that the Codex result describes a model that was summarized 28 times a run more than it describes a 400K window. The paper offers two explanations it cannot separate: newer models write better summaries and reorient better afterwards, or models trained for 1M windows saw more post-compaction continuations in training.

What the summaries lose

The ProgramBench prompt states three rules: do not copy or vendor the reference, do not use the internet, and write an original implementation rather than a wrapper. The rules appear once, in the first user message. After a compaction they exist only if the summary restates them. The OpenCode summarization prompt, printed in the appendix, asks for “Key user requests, constraints, or preferences that should persist.” A Claude Sonnet 5 judge then read each summary and recorded which of the three rules it no longer states.

Anti-cheating rules dropped per compaction summary on ProgramBench, by model A horizontal bar chart on a 0 to 3 scale with two extra columns. GPT-5.3-Codex drops 2.73 of the three rules per compaction summary, drops at least one in 99 percent of compactions and all three in 80 percent. GPT-5.4 drops 1.76, at least one in 90 percent and all three in 25 percent. Claude Opus 4.6 drops 0.51, at least one in 43 percent and all three in 0 percent. RULES DROPPED PER COMPACTION SUMMARY, OF 3 · PROGRAMBENCH · SALLE ET AL., ARXIV 2610.31778, TABLE 3 ≥1 dropped all 3 GPT-5.3-Codex GPT-5.4 Claude Opus 4.6 2.73 of 3 rules dropped per compaction — Table 3, arXiv 2610.31778 1.76 of 3 rules dropped per compaction — Table 3, arXiv 2610.31778 0.51 of 3 rules dropped per compaction — Table 3, arXiv 2610.31778 2.73 1.76 0.51 99% 80% 90% 25% 43% 0% 0 1 2 3 Judge: Claude Sonnet 5, reading each summary alone. The three rules: no copying the reference, no internet, an original implementation.
Per summary, Codex forgets almost everything it was forbidden to do, GPT-5.4 forgets more than half, and Opus 4.6 forgets about one rule in six and never all three.

Per compaction, “GPT-5.3-Codex drops 2.73 of 3 categories and all three in 80% of compactions. GPT-5.4 removes 1.76. Claude Opus 4.6 drops 0.51, and never all three.” Pooled across models and tasks, about 0.92 of the three rules are missing after the first compaction, 1.93 by the tenth, 2.41 by the fifteenth and 2.65 by the thirtieth. The mechanism is the one anyone who has maintained a running summary will recognize: each summary is written from the previous summary, “and nothing restates a rule that has been removed, so the loss compounds.”

Rules missing from the compaction summary, by how many compactions the run has had A line chart with four reported points on a 0 to 3 vertical scale against compaction index 0 to 30. After the first compaction 0.92 of the three rules are missing on average; after the tenth 1.93; after the fifteenth 2.41; after the thirtieth 2.65. The line between points is a trend, not reported data. RULES MISSING, OF 3, BY COMPACTION INDEX · PROGRAMBENCH, POOLED · SALLE ET AL., ARXIV 2610.31778, SECTION 4.2 3 2 1 0 1st 10th 15th 30th 0.92 of 3 rules missing after the first compaction — Section 4.2, arXiv 2610.31778 1.93 missing by the 10th compaction — Section 4.2, arXiv 2610.31778 2.41 missing by the 15th compaction — Section 4.2, arXiv 2610.31778 2.65 missing by the 30th compaction — Section 4.2, arXiv 2610.31778 0.92 1.93 2.41 2.65 Four values reported in the text; the line between them is a trend. The paper says the curve is less reliable past the 28th compaction, where few runs remain.
About one rule in three is gone after the first summary and about two in three by the tenth. Nothing in a benchmark score records this.

The judge was told to “default each category to ‘dropped’” and to count a rule as preserved only when the summary explicitly restates it; a summary that says “follow the original task rules” without naming them scores as having lost all three. That is the right choice for measuring what a fresh context would know, and it makes the numbers an upper bound on forgetting in a looser sense. And the authors checked whether erosion explains cheating. It doesn’t, at least not in the direction one would guess: “reward hacking begins before the first compaction happens,” and an agent already mid-violation may have little reason to restate the rule it is breaking, which would make the cheating the cause of the erosion rather than the other way round.

Real sessions

The benchmarks are the friendly case: fixed objective, no user, a filesystem that remembers. To see what happens when a person is steering, the authors took SWE-chat, a dataset of real coding-agent sessions collected from public repositories, sampled 44 sessions that contain at least one compaction, and had a model extract every explicit constraint the user had stated before the compaction. That produced 269 constraints. A second judge read each one against the summary that followed. Overall, 81 percent survived and 19 percent were dropped; a different judge from a different model family put the loss at 24.5 percent, and the two judges agreed on 239 of 269 verdicts.

Share of user constraints dropped by the next compaction summary, by category, on SWE-chat sessions A horizontal bar chart on a 0 to 40 percent scale. Feature requirements, 96 of the 269 constraints, are dropped 6.2 percent of the time. Process instructions, 39 of 269, are dropped 35.9 percent of the time. Scope exclusions are dropped 36.8 percent of the time. Across all 269 constraints the drop rate is 19 percent, marked with a dashed reference line; a second judge from another model family put the overall rate at 24.5 percent. CONSTRAINTS DROPPED BY THE NEXT SUMMARY · 269 CONSTRAINTS, 44 SWE-CHAT SESSIONS · ARXIV 2610.31778, SECTION 4.2 Feature requirements Process instructions Scope exclusions All 269 constraints 96 of 269 39 of 269 count not given in the text 6.2% of feature requirements dropped — Appendix 6.11, arXiv 2610.31778 35.9% of process instructions dropped — Appendix 6.11, arXiv 2610.31778 36.8% of scope exclusions dropped — Section 4.2, arXiv 2610.31778 19% of all constraints dropped — Section 4.2, arXiv 2610.31778 overall, 19% 6.2% 35.9% 36.8% 19% 0% 10% 20% 30% 40% Claude Sonnet 5 judge. A GPT-5.6 Sol judge put the overall rate at 24.5% with the two worst categories in reversed order. Three of the paper's categories shown. The sessions were made with Claude Code, not the OpenCode scaffold used elsewhere.
What to build survives. How to work, and what not to touch, is lost about six times as often.

Feature requirements, the largest category at 96 of 269, are dropped 6.2 percent of the time. Process instructions, how the agent should go about the work, are dropped 35.9 percent of the time. Scope exclusions, what it should not touch, 36.8 percent. The paper’s explanation is the same one that explains the benchmarks: a feature requirement describes the deliverable, and the deliverable is on disk, so a summary that omits it loses nothing the agent cannot reconstruct by looking at its own work. “Don’t touch the migration files” is not written anywhere except the conversation, and once the summary leaves it out it is gone.

That is also a description of my own week. The instruction most likely to vanish from a long session is exactly the kind that says “do not” or “always run X first,” and the moment it vanishes is the moment the context looks tidy.

Where I would hold back

The sample is small, and the authors say so: four models, three benchmarks, “a few tasks per each benchmark due to cost and compute constraints,” two runs per configuration, 44 sessions and 269 constraints on the real-world side. PostTrainBench’s error bars “are wide and overlap across all thresholds for every model,” so the paper’s cleanest claims rest on PaperBench and ProgramBench, which is three papers and two command-line tools.

The judging is model-on-model. Claude Sonnet 5 scores the summaries written by Claude Opus 4.6, GPT-5.4 and Codex, and the model that comes out best is the one from the judge’s own family. The authors replicated the SWE-chat half with a GPT-5.6 Sol judge and got a higher overall loss rate with the same two categories at the top, which is reassuring, but I did not see a second-family replication of the ProgramBench erosion table, where the gap between Opus’s 0.51 and Codex’s 2.73 is the headline. I would want one before treating the per-model ranking as settled. The task performance figures also use LLM judges, gpt-4o-mini for the PaperBench rubric.

The sweep also stops short of where production scaffolds run. The thresholds tested for the 1M models never approach the 90 to 95 percent of the window at which production scaffolds actually compact, so the paper’s most confident finding, that threshold does not matter, is about early compaction, not late. Late compaction is where “lost in the middle” effects from Liu and colleagues would bite, and that is the regime most people run in.

I also checked the code link. The abstract and introduction say “Code: aisa-group/compaction.” The repository exists on GitHub, created on 22 October at 17:57 UTC, the day the paper appeared. As of 26 October it contains no files; the API’s phrase is “This repository is empty,” and the footnote in the HTML version that should carry the URL reads “Code is available at:” followed by nothing. Four days is not long, and I expect it to fill. But the five compaction strategies and the judge prompts are the reproducible part of this paper, and for now the prompts in the appendix are what there is.

What I take from it

The earlier work the paper builds on, by Z. Wang and colleagues at arXiv:2608.11242, found that compactors retained only 17 percent of injected side constraints and that an email agent “deleted emails without approval in 30/100 sessions after compaction, compared with none before compaction.” (The Tübingen paper calls that suite COMPINT; the arXiv abstract calls it CONPACT.) OpenAI’s own misalignment reports from September, which I wrote about in the disclosure-latency piece, include two incidents that lived inside compaction summaries. What this paper adds is the demonstration that on the benchmarks we use to decide whether a harness is good, the losses are invisible by construction, because the benchmark only asks about the part of the context that survives on disk.

Anthropic reported a year ago that clearing old tool results alone improved an internal agentic search evaluation by 29 percent. That number and this paper are not in conflict. Clearing and summarizing can make the score go up while the rules go missing, and the benchmark will report only the first half. If you want to know whether your agent still remembers what it was told not to do after its fourth summary, the score will not tell you. You have to go and read the summary.

References

  1. Salle, J., Bhatnagar, H., Bethge, M., Abdelnabi, S. & Andriushchenko, M. (2026). Benchmarks Are Surprisingly Compaction-Friendly. arXiv:2610.31778, 22 October 2026.
  2. aisa-group (2026). compaction. GitHub repository, created 22 October 2026; reported empty by the GitHub API on 26 October 2026.
  3. Wang, Z., Zhang, Y., Lee, D. & Yang, Y. (2026). Lost in Context Compaction: When LLM Systems Drop Side Constraints. arXiv:2608.11242, submitted 31 July 2026.
  4. Baumann, J., Padmakumar, V., Li, X., Yang, J., Yang, D. & Koyejo, S. (2026). SWE-chat: Coding Agent Interactions From Real Users in the Wild. arXiv:2604.20779, submitted 22 April 2026.
  5. Yang, J., Lieret, K., Ma, J. et al. (2026). ProgramBench: Can Language Models Rebuild Programs From Scratch?. arXiv:2605.03546, 5 May 2026.
  6. Rank, B., Bhatnagar, H., Prabhu, A. et al. (2026). PostTrainBench: Can LLM Agents Automate LLM Post-Training?. arXiv:2603.08640, submitted 9 March 2026.
  7. Anthropic (2025). Managing context on the Claude Developer Platform. 29 September 2025.
  8. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F. & Liang, P. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12, 157-173, 2024.
  9. Goldman, O., Jacovi, A., Slobodkin, A., Maimon, A., Dagan, I. & Tsarfaty, R. (2024). Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLP. Proceedings of EMNLP 2024.
  10. Gautam Parab (2026). The Harness Tax Shows Up in the Bill, Not on the Scoreboard. 24 September 2026.
  11. Gautam Parab (2026). Same Weights, Different Harness, Up to 11.5 Points. 19 October 2026.
  12. Gautam Parab (2026). OpenAI Found the Message Board on May 25. It Told Us on September 16.. 17 September 2026.