Surge AI published DAYJOB: Tax on 27 October. It is a benchmark of 50 professional tax assignments built from 1,789 workspace files: K-1s, IRS notices, general ledgers, a photo of a handwritten receipt, and a field dispatch log showing a client’s crews at work in five states when the client thinks it only operates in Alabama. The agent gets a short client request and a folder, and has to work out what the job actually is. The leaderboard is led by Claude Sonnet 5.5 at 10.8 percent, then Claude Fable 5.1 at 9.6 percent and Claude Opus 5.5 at 9.2 percent. Meta’s Muse Wonder 1.0 is fourth at 6.0 percent, and OpenAI’s GPT 6 Astra is sixth at 4.8.
The tasks look good to me, and the low scores are probably honest. What made me stop is a line in the evaluation harness README: “Grading uses Claude Opus 4.8 as an agentic LLM judge.” So the three models at the top of the board come from the same lab as the model grading them. On its own that proves nothing. Still, Surge raised exactly this problem in its own launch post for the DAYJOB family in September, when it criticised automated versions of GDPval because LLM judges “are themselves imperfect graders with biases around style, framing, and model family.”
I have written before about a headline reliability number that came out of a 20-run campaign and about percentages that noise alone can fill when there are only a few runs per task. This one has both problems, and the grader on top of them.
What gets measured
Each assignment comes with a task-specific rubric. The Hugging Face dataset card gives a median of 30.5 criteria per task, a median of 30 input files (range 12 to 123), and a published pass threshold of “At least 95% of rubric criteria satisfied.” The blog reports pass@1 at that threshold. An attempt passes only if it hits at least 95 percent of its criteria, and the leaderboard number is the share of attempts that pass.
The harness README fills in the rest. The agent runs through the OpenHands SDK inside a container with network access limited to the model’s API host. Leaderboard runs use five attempts per task. The judge, Claude Opus 4.8, runs as an agent inside the same task container and assigns each attempt a reward between 0 and 1. A task’s score is the mean over its attempts, “Trials that errored (exception_info set) are excluded from the mean,” and the domain score is the unweighted mean over tasks.
Why the grader matters more here than usual
The research here points one way. When Lianmin Zheng and colleagues introduced MT-Bench in 2023, they listed “self-enhancement” bias next to position and verbosity bias as known failure modes of LLM judges. A year later Arjun Panickssery, Samuel Bowman and Shi Feng went further. They showed that GPT-4 and Llama 2, out of the box, have non-trivial accuracy at telling their own outputs apart from other models’ and from human text, and that across fine-tuned variants the strength of self-preference rose linearly with self-recognition ability. A 2026 survey in The Innovation by Jiawei Gu and colleagues treats bias mitigation as one of the main open problems in making LLM judges reliable.
Those studies are about a model preferring its own text, and DAYJOB pairs Opus 4.8 against Sonnet 5.5, Fable 5.1 and Opus 5.5, which are relatives, not the judge itself. I don’t know how far self-preference extends to siblings, and as far as I can tell nobody has measured it for agentic rubric grading of tax work. But a family resemblance in phrasing, in how a memo is laid out, in how a caveat gets worded, is exactly the kind of thing a judge reads before it decides whether a criterion was “satisfied.”
The 95 percent cutoff makes this worse. When the score is an average of rubric hits, a judge that leans a little toward one style shifts that model’s mean by a little. When the score is a pass/fail cutoff set near the ceiling, a single criterion decides the outcome.
On a rubric near the median, an attempt passes with one miss and fails with two. Every borderline call the judge makes, whether a caveat counted as identifying the issue or a workpaper counted as supporting the number, lands right on that line. If Opus 4.8 is even slightly more willing to give a Claude-shaped answer the benefit of the doubt, the cutoff turns that small lean into whole passes and fails.
I can’t tell from the published material whether that happens, because nothing I found reports human-versus-judge agreement on this benchmark, a second judge from another lab, or a rerun of the top models under a non-Anthropic grader. The harness supports a judge override through REWARDKIT_MODEL, so the check is cheap for anyone with access to the full set. Surge knows how much rubric grading can go wrong, because it has audited someone else’s: the September post says its audit of GDPval found that “Only about 20% of the criteria we reviewed were good as-is.”
The ranking is inside the noise anyway
Leave the grader aside and assume it is perfect. The leaderboard still claims more precision than it has.
Five attempts on each of 50 tasks gives 250 graded attempts per model if nothing errors. Sonnet’s 10.8 percent is then 27 passes, Fable’s 9.6 is 24, and Opus’s 9.2 is 23. Every score on the board except one is a multiple of 0.4 percent, which is what 250 attempts would produce. The exception is Grok 4.7 at 5.0 percent. That fits the README’s rule that errored trials are dropped from the mean, which changes the denominator for that model. I am inferring this from the arithmetic; Surge does not say how many Grok trials errored.
I computed Wilson score intervals on those counts. Sonnet’s runs from 7.5 to 15.3 percent, Fable’s from 6.5 to 13.9, and Opus’s from 6.2 to 13.4. The top three are statistically indistinguishable. Sonnet’s interval even overlaps GPT 6 Astra’s (2.8 to 8.2), though overlapping intervals don’t prove two models are equal. These intervals are also too narrow. They treat 250 attempts as independent draws, but five attempts at the same task are correlated: a task one model can solve tends to get solved several times, and a task it can’t stays at zero. The effective sample sits somewhere between 50 and 250, closer to 50 when attempts on a task agree.
The ordering may well be right. It is also what you would expect from three models that are roughly equal on this task set, graded by a judge whose own lean hasn’t been measured. “Claude leads” may still be true. “Sonnet 5.5 beats Opus 5.5 at tax” is not something 27 against 23 can support.
The anecdotes cut both ways
The blog’s two case studies are the most useful thing in it, and they don’t all point toward Claude. In one, a client called Anna asks whether to close the sale of her condo in October 2026 or January 2027. Under the rule in IRS Publication 523, the $250,000 home-sale exclusion requires, among other things, that she owned and lived in the home for 24 months of the five years before the sale. Both dates qualify. The real risk is that the window closes around the end of August 2027 if the sale slips. Surge reports that GPT 6 Astra never mentioned the deadline in five attempts, and Opus mentioned it in all five.
The second case runs the other way. A separating couple asks how to split a Roth IRA and a 401(k). Opus 5.5 opened with “You aren’t legally married,” and built every one of its five answers on that assumption, even though the couple’s records show they had lived in Texas, which recognises informal marriage, and held a ceremony in Colorado. Astra stopped to check marital status in four of five attempts, with “We first need to confirm your legal marital status.”
That is the kind of evidence I want from a benchmark: one model consistently noticing a fact, another consistently skipping past it, with transcripts to back it up. It is also evidence a human can check without trusting the judge. The aggregate number, which compresses 250 attempts through an Opus 4.8 rubric pass, is the part I’d trust least.
What I’d want before quoting the board
DAYJOB: Tax is a real improvement on tax benchmarks that hand the model a clean question, and Surge’s comparison with TaxBench’s 114 well-defined questions makes that case fairly. The tasks reward the right thing, which is noticing what the client didn’t ask. The headline result is that the best agents pass about one attempt in ten at a strict threshold, and I believe that result, because a judge’s lean would have to be huge to turn a 10 percent board into a 50 percent one.
The ranking at the top is a different matter. Before I’d repeat “Sonnet 5.5 leads,” I’d want three things that the harness already makes possible. One is the top five models re-graded with a judge from a lab that has no model on the board, or from each of the top three labs in turn. Another is a human-agreement rate on the criteria that decide borderline attempts, meaning the ones scored at 93 or 94 percent. The last is per-task pass counts, so readers can see how much of the 27 against 23 comes from one or two tasks. Until then, the leaderboard says Claude models do best when Claude grades them, which is true but isn’t the question the benchmark asks.
References
- Surge AI, “DAYJOB: Tax — Benchmarking AI Agents on Professional Tax Work,” 27 October 2026. https://surgehq.ai/blog/dayjob-tax
- Surge AI, DAYJOB: Tax leaderboard, read 28 October 2026. https://surgehq.ai/benchmarks/dayjob-tax
- Surge AI, “DAYJOB: Can Agents Survive a 9 to 5?” (Introducing DAYJOB: Healthcare and DAYJOB: Finance), 23 September 2026. https://surgehq.ai/blog/dayjob
- surge-ai/dayjob,
README.md(evaluation harness, grading and scoring convention), read 28 October 2026. https://github.com/surge-ai/dayjob - surgeai/DAYJOB-tax, Hugging Face dataset card, last modified 28 October 2026. https://huggingface.co/datasets/surgeai/DAYJOB-tax
- Internal Revenue Service, Publication 523, “Selling Your Home,” read 28 October 2026. https://www.irs.gov/publications/p523
- L. Zheng, W.-L. Chiang, Y. Sheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” Advances in Neural Information Processing Systems 36 (2023), 46595–46623. https://doi.org/10.52202/075280-2020
- A. Panickssery, S. Bowman and S. Feng, “LLM Evaluators Recognize and Favor Their Own Generations,” Advances in Neural Information Processing Systems 37 (2024), 68772–68802. https://doi.org/10.52202/079017-2197
- J. Gu, X. Jiang, Z. Shi et al., “A survey on LLM-as-a-judge,” The Innovation 7(6), 101253 (June 2026). https://doi.org/10.1016/j.xinn.2025.101253