The Skills Helped in All Nine Setups, Instruction Included. The 36.78 Percent Counts Something Else.

A preprint posted on 6 October has a sentence in its abstract that reads like a warning label: “The same Skills help some configurations and hurt others on 36.78% of tasks.” It is easy to compress that to “agent skills hurt performance on more than a third of tasks.” That isn’t what the paper found. On its own headline comparison, the benchmark’s skills, supplied together with an instruction to use them, raised the average pass rate in every configuration tested.

The paper is “An Empirical Study of Agent Skills’ Downstream Utility” by Yu Cheng and five colleagues, most of them at Zhejiang University. It is a preprint, revised on 8 October, and not yet peer reviewed. Its subject is the skill: a folder holding a SKILL.md file of instructions, sometimes with scripts and reference files, that an agent harness can load when a task seems to call for it. Anthropic introduced the format for Claude in October 2025 and published it as an open standard that December, and public marketplaces now hold tens of thousands of them. Cheng’s team took snapshots of two marketplaces on 20 August 2026 and counted 124,425 records, which they filtered to 37,596 usable skills. When I read twelve agent codebases in August, a folder of SKILL.md files already looked like the cheapest way to make an agent better at one thing. The question is whether it actually does.

What the table says

The first experiment is a clean design. The authors take the 87 tasks of SkillsBench, a benchmark that pairs each task with curated skills and a deterministic checker of the final result. They run three models (GPT-5.5, DeepSeek-V4-Flash and Qwen3.5-27B) in three harnesses (Codex, OpenCode and OpenClaw), which makes nine configurations. Every task runs three times without skills and three times with them, each run in a fresh environment.

The result is the paper’s Table 3, and it is not ambiguous: “Benchmark Skills improve the overall pass rate in all nine configurations, with gains of 4.60–19.54 percentage points over No-Skill.” These are averages over three runs per task, and the paper reports no confidence intervals.

Pass rate without and with the benchmark's skills, nine model-harness configurations A dumbbell chart of 87 SkillsBench tasks run three times per condition, from Cheng et al. (arXiv 2610.08875, Table 3). GPT-5.5 with Codex: 42.15% without skills, 54.02% with, a gain of 11.88 points; GPT-5.5 with OpenClaw: 30.65% without skills, 47.89% with, a gain of 17.24 points; GPT-5.5 with OpenCode: 24.14% without skills, 43.68% with, a gain of 19.54 points; DeepSeek-V4-Flash with Codex: 34.10% without skills, 44.44% with, a gain of 10.34 points; DeepSeek-V4-Flash with OpenClaw: 35.63% without skills, 50.19% with, a gain of 14.56 points; DeepSeek-V4-Flash with OpenCode: 36.40% without skills, 48.28% with, a gain of 11.88 points; Qwen3.5-27B with Codex: 2.30% without skills, 6.90% with, a gain of 4.60 points; Qwen3.5-27B with OpenClaw: 3.45% without skills, 17.24% with, a gain of 13.79 points; Qwen3.5-27B with OpenCode: 13.41% without skills, 24.52% with, a gain of 11.11 points. Every configuration improves; gains range from 4.60 to 19.54 percentage points. PASS RATE, NO SKILLS VS BENCHMARK SKILLS · 87 TASKS · CHENG ET AL., OCT 2026 0% 20% 40% 60% GPT-5.5 Codex 42.15% without skills, GPT-5.5 + Codex 54.02% with benchmark skills, GPT-5.5 + Codex +11.88 OpenClaw 30.65% without skills, GPT-5.5 + OpenClaw 47.89% with benchmark skills, GPT-5.5 + OpenClaw +17.24 OpenCode 24.14% without skills, GPT-5.5 + OpenCode 43.68% with benchmark skills, GPT-5.5 + OpenCode +19.54 DeepSeek-V4-Flash Codex 34.10% without skills, DeepSeek-V4-Flash + Codex 44.44% with benchmark skills, DeepSeek-V4-Flash + Codex +10.34 OpenClaw 35.63% without skills, DeepSeek-V4-Flash + OpenClaw 50.19% with benchmark skills, DeepSeek-V4-Flash + OpenClaw +14.56 OpenCode 36.40% without skills, DeepSeek-V4-Flash + OpenCode 48.28% with benchmark skills, DeepSeek-V4-Flash + OpenCode +11.88 Qwen3.5-27B Codex 2.30% without skills, Qwen3.5-27B + Codex 6.90% with benchmark skills, Qwen3.5-27B + Codex +4.60 OpenClaw 3.45% without skills, Qwen3.5-27B + OpenClaw 17.24% with benchmark skills, Qwen3.5-27B + OpenClaw +13.79 OpenCode 13.41% without skills, Qwen3.5-27B + OpenCode 24.52% with benchmark skills, Qwen3.5-27B + OpenCode +11.11 No skills Skills plus an instruction to use them Right column: gain, points
Averaged over all 87 tasks, the pass rate rose for every model in every harness. The smallest gain, 4.60 points, belongs to the configuration that started at 2.30 percent.

Two details in the table matter for what follows. First, the gains are not spread evenly. Qwen3.5-27B in Codex went from 2.30 to 6.90 percent, which is a gain, but on a configuration that barely works with or without help. GPT-5.5 gained more than Qwen in all three harnesses. Second, the harness effect is comparable in size to the model effect here. GPT-5.5 picks up 11.88 points in Codex and 19.54 in OpenCode, from the same skills.

The SkillsBench authors, led by Xiangyi Li, ran a larger version of this comparison across 18 configurations, Claude Code among them, and report in their fourth revision that curated skills lift the average pass rate from 33.9 to 50.5 percent. Two sets of runs on the same benchmark, same direction.

What the 36.78 percent counts

The headline number comes from the paper’s Figure 2, which plots the change for each task in each of the nine configurations. The authors sort the 87 tasks into four groups:

The first group, 32 of 87, is the 36.78 percent: the share of tasks on which the nine configurations disagreed about direction. The tasks that skills made worse with no offsetting gain anywhere number 8 of 87, or 9.2 percent.

How the 87 tasks moved across nine configurations when skills were added A unit chart, one square per task, from Cheng et al. (arXiv 2610.08875, Figure 2). 32 tasks showed a gain in at least one configuration and a loss in at least one other: these are the 36.78 percent. 32 showed gains with no losses, 8 showed losses with no gains, and 15 showed no change in any configuration. A footnote gives the writer's arithmetic: a task that passes half the time with or without skills, so that skills change nothing, would land in the gains-and-losses group about 95 percent of the time under nine configurations of three runs each. 87 TASKS, ONE SQUARE EACH · DIRECTION OF CHANGE ACROSS 9 CONFIGURATIONS · CHENG ET AL. Gains and losses 32 tasks Gains only 32 tasks Losses only 8 tasks No change 15 tasks Filled: the 32 tasks (36.78%) behind the headline. Gains and losses are in pass rate, three runs per condition. Writer's arithmetic: a task passing 50% of the time with or without skills lands in the filled group about 95% of the time.
The 36.78 percent is the filled block: tasks where at least one configuration gained and at least one lost. Pure run-to-run noise on a coin-flip task would put it there too.

That disagreement tells you something, and the authors use it well, but it carries a measurement problem that the abstract doesn’t mention. Each cell in Figure 2 compares two pass rates built from three runs apiece. A three-run pass rate can only be 0, 33, 67 or 100 percent, so one run flipping moves it by 33 points. The paper’s threats-to-validity section says as much: “With only three repetitions per task and condition, task-level pass rates have limited resolution and remain sensitive to execution randomness.”

To size the problem I did the arithmetic myself; it isn’t in the paper. Take a task that an agent passes half the time, skills or no skills, so that the skills do nothing at all. In one configuration, the chance that the two three-run pass rates come out equal is \(\tfrac{1+9+9+1}{64} = 0.3125\), so a gain and a loss are each about 34 percent likely. Treating the nine configurations as independent, the chance of seeing at least one gain and at least one loss is about 95 percent. A coin-flip task with useless skills lands in the 36.78 percent group nineteen times out of twenty.

Most of the 87 tasks aren’t coin flips. A task that always passes or always fails doesn’t wobble, which is presumably why 15 of them never moved. That doesn’t make the 32 mixed tasks noise. It means the paper can’t separate noise from real reversals, because three runs per cell can’t. The number tells you that the configurations disagreed. It doesn’t tell you how often a skill turned a pass into a fail by more than chance would. Small trial counts keep producing headline numbers in agent evaluation. I wrote about the same problem in September, when a voice-agent lead of 0.7 points rested on a rival scored on one trial.

The reversal that is real

The paper does document reversals that noise can’t explain, and they are the best part of it. On a task called exam-block-sequencing, the instructions permit solvers, heuristics or a mix, but the supplied skills push integer programming. DeepSeek-V4-Flash in Codex passed all three runs without the skills and timed out on all three with them, stuck debugging an integer program until it timed out. GPT-5.5 in the same harness, with the same skills, passed all three. In one run its solver process was also killed, and it switched to a permitted heuristic, kept a feasible answer and checked its output files before finishing. Three-for-three against zero-for-three is not a coin flip.

The authors’ reading is that “recommended procedures can become an execution burden” when a configuration can’t carry them out, and that skills help when their procedures match what the configuration “can execute and adapt.” My gloss: the useful question is whether the model can adapt the procedure or can only follow it. The same task turns up at the bottom of the SkillsBench results: 13 of its 87 tasks show negative skill deltas when averaged over its 18 configurations, and exam-block-sequencing is tied with one other task for the largest drop, at 7.4 points. Two teams, the same task, the same failure.

Help and instruction, bundled

A second caveat sits in the method section, and the authors flag it themselves. The skill condition doesn’t only supply the skills. “The task instruction explicitly asks agents to consult and use them,” while the no-skill condition has neither. “Gains relative to No-Skill therefore reflect this combined setting.” So the 4.60 to 19.54 points measure skills plus an order to use them. In ordinary use, as Anthropic described the design in 2025, an agent sees only a name and short description of each installed skill and decides for itself whether to open the full file. That decision is where much of the difficulty sits.

The rest of the 2026 record is mostly about that decision, and it is less cheerful than Table 3:

The sharpest result comes from Wei-Jung Huang’s July study of generated skills for data-science work: 7,560 runs across 56 tasks and nine model configurations found no significant improvement from a full generated skill or any part of one (all p-values at least 0.396). A token-matched control then found that the full skills “perform similarly to task-irrelevant skill-formatted content.” Put differently, on that workload the skill was doing about as much as the same amount of off-topic text.

That result has a precedent. In January 2023, Freda Shi and colleagues showed that adding a single irrelevant sentence to grade-school math problems was enough to drop model accuracy sharply. One fix they tested was to tell the model to ignore what didn’t matter. A skill is relevant context by intention. Whether a given model treats it that way is an empirical question, and the answer changes with the model, the harness and the task.

What the number should have been

None of this makes the Cheng paper weak. It is careful about its limits, its mechanism analysis is the useful kind, and its authoring advice in the second and third experiments is grounded in run data rather than taste. The problem is the abstract’s one-line summary, which counts disagreement and reads like harm, and which the paper’s own design can’t resolve into either.

The number I would want is different: for each task and configuration, enough runs to separate a real reversal from a wobble, and then the share of task-configuration pairs where skills lowered the pass rate by more than the run-to-run spread. Three runs can’t produce that. Ten would begin to, at more than three times the cost of the original experiment. Until somebody pays for it, the defensible summary of this paper is the one in its own Table 3: curated skills, plus an instruction to use them, raised pass rates in all nine setups tested, by 4.6 to 19.5 points, and on some tasks some models did worse with them. The 36.78 percent is how often the nine setups disagreed. It is not a measure of how often skills hurt.

References

  1. Cheng, Y., Zhao, D., Liu, Z., Huang, Q., Xing, Z., and Ren, X. (2026). An Empirical Study of Agent Skills’ Downstream Utility. arXiv:2610.08875, v1 6 October 2026, v2 8 October 2026.
  2. Li, X., et al. (2026). SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks. arXiv:2602.12670, first posted 13 February 2026; figures cited from v4.
  3. Han, T., et al. (2026). SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?. arXiv:2603.15401, 16 March 2026.
  4. Jiang, Z., et al. (2026). Demystifying Agent Skills: Why They Work-Until They Don’t. arXiv:2608.14036, 14 August 2026.
  5. Huang, W.-J. (2026). Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows. arXiv:2607.07504, 8 July 2026.
  6. Shi, F., et al. (2023). Large Language Models Can Be Easily Distracted by Irrelevant Context. arXiv:2302.00093, 31 January 2023.
  7. Anthropic Engineering (2025). Equipping agents for the real world with Agent Skills. 16 October 2025, updated 18 December 2025.