Same Weights, Different Harness, Up to 11.5 Points

Google Research has put Harness² on GitHub under the Apache 2.0 license, alongside a paper dated 9 October, arXiv:2610.14224. The README describes a framework for “recursive agent harnessing”: the agent keeps its weights, and a second agent edits everything wrapped around it. The headline in the paper is that one improvement step lifted a frozen GLM-5 by 11.5 points on WorkBuddy-Bench. I read the paper and the README instead of the coverage, because the number needs its context.

What gets edited

A harness, in the paper’s usage, is the system around a model: prompt, tools, memory, skills, orchestration. Harness² treats it as eight editable parts: the system prompt, guardrails, memory, tool guidance, subagents, skills, scripts and plugins. The paper’s own table sorts them by runtime cost. Four are always on, so every token in them is paid on every turn. Skills and similar parts load only when invoked, which means an edit that adds a skill costs nothing until the agent calls it.

The editing loop is called Propose, Probe, Compose. An editor reads the agent’s trajectory and drafts two kinds of candidate edits, one general and one specific to the domain. Each candidate is run on the same task. A composer then builds a refined harness from what differed between the runs. The paper’s pitch is that this needs no ground-truth verifier at test time, which is the realistic condition: nobody hands a deployed agent the answer key. A second layer, which the paper calls harnessing-level recursion, keeps a library of adopted harnesses and lessons and feeds them into later tasks. The --k flag sets the budget, and \(k=1\) is a single step.

What the numbers say

The paper tests two frozen executors, Gemini 3.5 Flash and GLM-5, each under two off-the-shelf harnesses, OpenCode and Codex. The default editor is Claude Opus 4.8. Three benchmarks: WorkBuddy-Bench (office, web and code deliverables), JobBench (65 professional tasks, scored on a 34-task test split) and Harvey AI’s LAB for legal work (scored on a 100-task test set). Scores are means over three runs with shuffled task order.

The README’s headline gains, up to 11.5 points on WorkBuddy-Bench, 4.5 on JobBench and 11.8 percentage points on LAB, are the best cell in each table, not typical cells. I matched each to its row. The 11.5 is GLM-5 under OpenCode, 64.2 to 75.7. The 4.5 is Gemini 3.5 Flash under OpenCode on JobBench, 31.5 to 36.0. The 11.8 is GLM-5 under OpenCode on LAB, 59.6 to 71.4. The differences are my arithmetic on the paper’s tables.

WorkBuddy-Bench average score before and after one Harness² step, four executor and harness pairs A dumbbell chart on a 60 to 80 scale. Gemini 3.5 Flash under OpenCode goes from 72.4 to 77.5, a gain of 5.1. Gemini 3.5 Flash under Codex goes from 74.5 to 77.5, a gain of 3.0. GLM-5 under OpenCode goes from 64.2 to 75.7, a gain of 11.5. GLM-5 under Codex goes from 64.4 to 75.3, a gain of 10.9. All four end between 75.3 and 77.5. WORKBUDDY-BENCH AVERAGE · ONE HARNESS² STEP · XU ET AL., ARXIV 2610.14224, TABLE 1 60 65 70 75 80 Gain Gemini 3.5 Flash, OpenCode Gemini 3.5 Flash, Codex GLM-5, OpenCode GLM-5, Codex 72.4 base — Table 1, arXiv 2610.14224 74.5 base — Table 1, arXiv 2610.14224 64.2 base — Table 1, arXiv 2610.14224 64.4 base — Table 1, arXiv 2610.14224 77.5 with Harness² — Table 1, arXiv 2610.14224 77.5 with Harness² — Table 1, arXiv 2610.14224 75.7 with Harness² — Table 1, arXiv 2610.14224 75.3 with Harness² — Table 1, arXiv 2610.14224 72.477.5 74.577.5 64.275.7 64.475.3 +5.1 +3.0 +11.5 +10.9 Open circle: base harness. Filled: after one Harness² step (k = 1 in the paper).
The weak starting points moved most, and the four endpoints bunch between 75.3 and 77.5. Gains are my subtraction from the paper's table.

The pattern across the four is the point. The two GLM-5 configurations started near 64 and gained about 11. The two Gemini configurations started near 72 to 75 and gained 3 to 5. Everyone lands within about two points of each other. That reads as a floor being raised more than a ceiling being broken, and the paper’s own comparison with fixed production harnesses points the same way: no single off-the-shelf harness led every domain, and Harness² starting from OpenCode at \(k=5\), with Gemini 3.5 Flash and more test-time compute, beat the strongest fixed alternative by 4.6, 5.8 and 4.9 points on code, web and office.

The parts I trust more than the headline

The ablation is the most useful table in the paper. On Gemini 3.5 Flash under OpenCode at \(k=1\), the full system scores 77.5 on WorkBuddy-Bench and 36.0 on JobBench. Removing the improvement library drops those to 74.4 and 34.2. Removing the scoped editor roles drops them to 74.6 and 32.3. Skipping composition drops them to 75.9 and 34.1. Every component earns something, and the scoping result is the biggest on JobBench, 3.7 points.

The control I like best is the blind one. They had the improvement harness edit the base harness directly, with no reference executions, no candidate probes and no library. Scores fell, on WorkBuddy code from 73.4 to 63.2 and on web from 69.9 to 65.0. An editor that cannot see the agent run will make a confident, wrong change. Evidence from real runs is necessary, whatever else contributes.

The paper also tests whether a strong editor is doing the work. It uses the same model as executor and editor at \(k=2\), and all five models, from GLM-5 to Claude Opus 5, improve on every benchmark. Opus 5 gains 3.3, 5.5 and 3.0 points on WorkBuddy-Bench, JobBench and LAB, and GLM-5 gains 13.7 on LAB. So self-editing works without a bigger model looking over its shoulder, at least on these three benchmarks.

One more result worth recording is the case study: a JobBench task where the baseline run scored 43.1. The trajectory showed procedural failures, never reconciling figures across source files and never reopening the files it wrote. The improved harness added a guardrail forcing an input inventory and a verification script, plus a domain skill for reconciliation, and the rerun scored 77.8. That is one task the authors chose to show. It illustrates the mechanism well and says nothing about the average.

What would make me cautious

Start with who judges. In the paper’s appendix, JobBench deliverables are scored against a rubric by Claude Opus 4.8, which is also the default editor. LAB is judged by Claude Sonnet 4.6. WorkBuddy-Bench uses its own composite verifier, with deterministic checks plus judge items, and where a judge is needed it is Gemini 3.5 Flash, which is also one of the two executors. On JobBench the paper says the rubric is never shown to the executor, the editor or the selector, and I believe it. But the authors’ own limitations section concedes that without ground truth, refinement “relies on behavioral contrasts and model judgment” and stays sensitive to “shared model biases.” An editor from one model family improving outputs that a judge from the same family then scores is a setup that can drift toward what that family likes. The paper does not measure that drift, and I have no number for it either.

Second, the baselines are not all beaten everywhere. The offline Meta-Harness baseline, which searches harness code against a separate training split, edges Harness² on some cells. Under Codex with Gemini 3.5 Flash on LAB it scores 76.5 against 75.6, and the standard deviations are 2.1 and 4.5, the paper notes that Meta-Harness leads that cell, but the deviations overlap, so I read it as within noise. The fair reading is that Harness² matches or beats an offline optimizer without needing a training split, and not that it dominates it.

Third, cost. Every improvement step is extra executions plus an editor session. Raising \(k\) from 1 to 5 took the Gemini 3.5 Flash OpenCode gain from 4.5 to 7.5 points on JobBench and from 5.1 to 8.4 on WorkBuddy-Bench. The paper plots total tokens, execution and improvement together, and puts both Harness² settings on the empirical Pareto frontier, with Meta-Harness spending more than Harness² at \(k=5\) for a lower score. I take the authors’ word that the accounting is complete. A reader budgeting a deployment needs the actual token counts, and I would read those off the figure’s axis, not estimate them here. The limitations section says the budget range tested is bounded, and that no universally optimal allocation was established.

Fourth, the safety notes in the README. The executor’s network access is unrestricted, and Codex runs with its sandbox disabled. Public benchmark rubrics may be reachable online, so the README tells users to restrict the network before evaluating, or the executor could look up the answers. The warning is useful, and it also describes an agent that edits its own guardrails file. I wrote earlier this month about a harness whose agent could switch off its own sandbox. A guardrails file the agent can edit is advice to the model, not a boundary, so the sandbox has to sit outside the agent’s reach.

How to try it

The README lists Linux, Python 3.12 and uv, with a local demo that needs no credentials. The model-backed benchmarks need Node.js 20 or later, the Google Cloud CLI and Vertex AI configuration, and Docker for WorkBuddy-Bench, and the README warns that runs cost API money and suggests starting with small task selections. A --dry-run flag previews task selection with no model calls. The project describes itself as not an officially supported Google product, and it is outside Google’s open-source vulnerability rewards program.

My read: the finding I expect to hold is the boring one. A good share of what separates a weak agent setup from a decent one is procedural, such as checking inputs, reopening outputs and managing the context budget, and a loop that watches the agent fail and edits those procedures can find it. Whether the 11.5 survives a judge from a different family, or a task stream that is not a benchmark, is the experiment I would run first.

References

  1. Ruiyao Xu, Yanfei Chen, Zhongying CuiZhu, Bhavana Dalvi Mishra, Yifei Ming, Han Yu, Rujun Han, Chen-Yu Lee, Tomas Pfister, “Harness^2: Recursive Agent Harnessing for an Open World”, arXiv:2610.14224, submitted 9 October 2026.
  2. Google Research, google-research/harness2 README, accessed 19 October 2026.
  3. Gautam Parab, “The Harness Couldn’t Tell Its Agent From You”, 2 October 2026.