Level With the Market on Fifty-Four Questions

On 29 October Yuxuan Wang, Yishu Wang and Jiaqi Deng of Raven Labs and Zhejiang University posted “Benchmarking Forecasting Agents on Yesterday’s Tomorrows” to arXiv. It introduces FABench, a benchmark that asks forecasting agents questions as if it were 1 July 2026, about events that have since resolved, and lets them research on an archive and a screened version of the live web. The abstract carries the line people will quote: retrieval “lifts the leading agents to the market consensus on the 54 questions linked to Polymarket, with the best three scoring slightly above it.”

I read the whole paper, appendices included. Most of the engineering holds up, and the leakage controls are better than I expected. The claim about the market is the weakest part, and the paper’s own appendix says why.

What the benchmark does

The first release has 287 yes-or-no questions, all posed at 1 July 2026, 00:00 UTC, in nine domains from finance to weather. LLM agents wrote 254 of them from earnings calendars, central-bank schedules, official statistics and news, and 33 were taken directly from Polymarket. Horizons run from 1.8 to 184 days, with a median of 62, and every question had resolved by mid-September.

An agent researches each question with two sources. One is an offline archive: about 20.1 million English news captures from Common Crawl’s CC-News taken before the forecast date, plus 684 Wikipedia revisions frozen at the last edit before it. The archive can’t leak by construction, since nothing in it was captured after the cutoff. The other is live web search, where every page is fetched and shown to an LLM reviewer that sees neither the question nor the answer and decides only whether the text was public before 1 July. Prediction-market sites are blocked outright. In the web-search runs the reviewer withheld 16.0 percent of 302,561 page reviews.

The authors then checked their own filter. They re-reviewed 3,087 pages at high reasoning effort: the two passes agreed on 95.7 percent, and only 14 pages, 0.5 percent, had passed at low effort but were judged at high effort to give away later information. The comparison I find most convincing is between settings. With the archive alone, which can’t leak, a model gets about four fifths of the gain it gets from the full setup. If leaked pages were doing the work, the web settings would pull far ahead of the archive. They lead by at most 1.5 Brier Index points for every model.

The paper runs eight agents, built on GPT-6 Astra, GPT-6 Sol, Claude Opus 5.5, DeepSeek V4.1 Flash, Kimi K3, GLM 5.3, Qwen3.8-27B and Qwen3.6-35B-A3B. Most were released after 1 July. To check that they don’t simply remember the outcomes, the authors point out that without tools the best of them scores 55.0, close to a coin. That’s indirect evidence, but it’s the right check, and the March run in their appendix shows the same pattern.

How far above a coin

Scores are reported as a Brier Index, defined in the paper as 100 × (1 − √Brier). A constant forecast of 0.5 scores 50, and a perfect forecaster scores 100. Because of the square root, a few index points mean more than they appear to. Here is every agent without tools and with the full setup.

Brier Index without tools and with full retrieval, eight agents Dot plot on a Brier Index scale from 48 to 60, with a dashed line at 50 for a constant 0.5 forecast. Without tools, then with full retrieval: GPT-6 Astra 54.33 to 58.42, DeepSeek V4.1 Flash 53.78 to 58.02, GPT-6 Sol 53.51 to 57.60, Claude Opus 5.5 55.04 to 57.48, Kimi K3 51.39 to 57.23, GLM 5.3 53.15 to 56.18, Qwen3.8-27B 52.36 to 54.69, Qwen3.6-35B-A3B 50.31 to 51.78. Every agent improves with retrieval; the best full score is 58.42. BRIER INDEX · 287 QUESTIONS · NO TOOLS VS FULL · WANG ET AL., 29 OCT 2026 48 50 · coin 52 54 56 58 60 GPT-6 Astra DeepSeek V4.1 Flash GPT-6 Sol Claude Opus 5.5 Kimi K3 GLM 5.3 Qwen3.8-27B Qwen3.6-35B-A3B GPT-6 Astra, no tools: 54.33 BI — Wang et al., Table 2 DeepSeek V4.1 Flash, no tools: 53.78 BI — Wang et al., Table 2 GPT-6 Sol, no tools: 53.51 BI — Wang et al., Table 2 Claude Opus 5.5, no tools: 55.04 BI — Wang et al., Table 2 Kimi K3, no tools: 51.39 BI — Wang et al., Table 2 GLM 5.3, no tools: 53.15 BI — Wang et al., Table 2 Qwen3.8-27B, no tools: 52.36 BI — Wang et al., Table 2 Qwen3.6-35B-A3B, no tools: 50.31 BI — Wang et al., Table 2 GPT-6 Astra, full: 58.42 BI — Wang et al., Table 2 DeepSeek V4.1 Flash, full: 58.02 BI — Wang et al., Table 2 GPT-6 Sol, full: 57.60 BI — Wang et al., Table 2 Claude Opus 5.5, full: 57.48 BI — Wang et al., Table 2 Kimi K3, full: 57.23 BI — Wang et al., Table 2 GLM 5.3, full: 56.18 BI — Wang et al., Table 2 Qwen3.8-27B, full: 54.69 BI — Wang et al., Table 2 Qwen3.6-35B-A3B, full: 51.78 BI — Wang et al., Table 2 58.42 58.02 57.60 57.48 57.23 56.18 54.69 51.78 no tools archive plus screened web search (full); right-hand labels are the full scores Brier Index = 100 × (1 − √Brier). Each run is scored once per setting at a single forecast date, 1 July 2026.
Retrieval helps every agent. The whole field still sits in the bottom fifth of the scale between a coin and a perfect forecaster.

The paper doesn’t hide the size of the gap: “Even the best model removes only about 31% of the squared error of the constant forecast.” That checks out. A Brier Index of 58.42 is a Brier score of 0.173, against 0.25 for the coin.

I’d add a second baseline, which the paper uses in an appendix but not in its headline. In the first release, 178 of the 287 questions resolved Yes, 62.0 percent. A forecaster who said 0.62 to every question, without reading anything, would score a Brier of about 0.236, or roughly 51.5 on the index. That’s my arithmetic from the paper’s Yes rate, not a number the paper prints. Against that baseline the best agent removes about 27 percent of the error rather than 31. For their March 2026 runs the authors do exactly this comparison, noting that “a constant forecast at each set’s Yes rate scores 58.4 and 51.7 BI,” and they use it to argue the models didn’t memorise outcomes. Putting the same line on the headline chart would cost nothing.

The market comparison is narrow

The claim that agents now match a prediction market comes from a subset. Of the 287 questions, 54 are linked to Polymarket: the 33 taken from Polymarket itself and 21 agent-written questions matched to a contract on the same event. The market’s forecast is its last traded price, “taken 55 to 56 seconds before” the forecast date. Under full retrieval, “six models lie between 1.8 BI below and 2.7 BI above the market.” Without retrieval, every model trails it by 1.8 to 10.8.

Which questions the market comparison rests on Three stacked bars. First, all 287 questions: 54 are linked to a Polymarket contract and the rest are not. Second, the 54 linked questions: 33 taken directly from Polymarket and 21 agent-written questions matched to a contract. Third, the 21 matched contracts: 2 settle under exactly the question's rule and 19 differ in settling source or postponement handling. The 33 Polymarket markets had lifetime volume from 732 dollars to 40.5 million dollars, median 10.4 thousand, with no volume threshold. FABENCH FIRST RELEASE · MARKET-LINKED QUESTIONS · WANG ET AL., TABLE 3 AND APP. E All 287 questions The 54 questions linked to Polymarket The 21 matched contracts 233 questions not linked to a market (287 minus 54) — Wang et al. 54 questions linked to Polymarket — Wang et al., Section 4.2 33 questions taken from Polymarket — Wang et al., Table 3 21 agent-written questions matched to a contract — Wang et al., Table 3 2 contracts settle under exactly the question's rule — Wang et al., Appendix E 19 contracts differ in settling source or postponement rule — Wang et al., Appendix E 233 not linked to any market 54 linked 33 taken from Polymarket 21 written, then matched 2 19 with a different settling source or postponement rule 2 settle under exactly the question's own rule Lifetime volume of the 33 Polymarket markets: $732 to $40.5 million, median $10.4 thousand. No volume threshold. Market forecast = last price 55 to 56 seconds before 1 July 2026, 00:00 UTC. Under full retrieval, six agents land between 1.8 BI below and 2.7 BI above the market on these 54.
The market result covers 54 of 287 questions. In 19 of the 21 matched cases the contract and the question resolve under different rules.

Three details in Table 3 and Appendix E make me read “on par” loosely.

First, the matching. For 19 of the 21 matched questions, the Polymarket contract differs “in the settling source or the handling of a postponement.” A market price on a slightly different proposition is a fair reference, but it isn’t the same forecast, and the gap can go either way.

Second, the markets themselves. There was “No volume threshold: lifetime volume ranges from $732 to $40.5 million, median $10.4 thousand.” A market with ten thousand dollars of lifetime volume is a few traders. The literature on prediction-market accuracy is mostly about big, liquid markets. Wolfers and Zitzewitz’s 2004 review found market forecasts “typically fairly accurate” and better than “most moderately sophisticated benchmarks”. Strijbis and Arnesen, studying a real-money market on 65 Swiss referendums, found that “prediction market accuracy varies primarily according to the setup of the market.” Page and Clemen found prices are “reasonably well calibrated when time to expiration is relatively short, but prices are significantly biased for events farther in the future.” FABench’s horizons have a median of 62 days and run to 184. A thin market priced two months out is a weaker opponent than “the market consensus” sounds.

Third, the uncertainty. For the retrieval settings, the paper runs a paired bootstrap and says plainly that single-model differences mostly sit “within sampling uncertainty.” I couldn’t find a matching interval for the comparison with the market, which has about a fifth as many questions. A spread of −1.8 to +2.7 BI over 54 questions, with each model run once, is consistent with the agents and the market being level, and it’s also consistent with a moderate gap either way. “Slightly above” is the most a reader should take from it.

In the body the authors are careful. They call the market “a strong baseline,” and they note that earlier work found language-model forecasters trailing liquid markets. The abstract’s sentence is just more confident than the table under it.

A ranking of how hard each agent looked

The finding I liked most is about effort. Each agent could make up to 100 tool calls per question. How many it actually made, and especially how many pages it read, tracks its gain from retrieval. In the paper’s words: “The ranking depends less on what a model knows than on how much it researches.”

Pages read per question against gain from retrieval Scatter plot of eight agents under full retrieval. Horizontal axis: pages read per question, 0 to 10. Vertical axis: Brier Index gain over no tools, 0 to 7. Qwen3.6-35B reads 0.6 pages and gains 1.47; Claude Opus 5.5 reads 0.8 and gains 2.44; GLM 5.3 reads 2.3 and gains 3.03; GPT-6 Sol reads 3.2 and gains 4.09; Qwen3.8-27B reads 4.0 and gains 2.33; GPT-6 Astra reads 4.3 and gains 4.09; Kimi K3 reads 6.6 and gains 5.84; DeepSeek V4.1 Flash reads 8.9 and gains 4.24. FULL RETRIEVAL · PAGES READ VS BI GAIN · WANG ET AL., TABLES 2 AND 9 0 2 4 6 8 10 pages read per question 0 +2 +4 +6 BI gain Qwen3.6-35B-A3B: 0.6 pages, +1.47 BI — Wang et al., Tables 2 and 9 Claude Opus 5.5: 0.8 pages, +2.44 BI — Wang et al., Tables 2 and 9 GLM 5.3: 2.3 pages, +3.03 BI — Wang et al., Tables 2 and 9 GPT-6 Sol: 3.2 pages, +4.09 BI — Wang et al., Tables 2 and 9 Qwen3.8-27B: 4.0 pages, +2.33 BI — Wang et al., Tables 2 and 9 GPT-6 Astra: 4.3 pages, +4.09 BI — Wang et al., Tables 2 and 9 Kimi K3: 6.6 pages, +5.84 BI — Wang et al., Tables 2 and 9 DeepSeek V4.1 Flash: 8.9 pages, +4.24 BI — Wang et al., Tables 2 and 9 Qwen3.6-35B Claude Opus 5.5 GLM 5.3 GPT-6 Sol Qwen3.8-27B GPT-6 Astra Kimi K3 DeepSeek V4.1 Flash Gain = full minus no-tools BI. Each agent could make up to 100 tool calls per question.
The agents that read more gained more. Claude Opus 5.5 was the most accurate without tools and read under one page per question.

Claude Opus 5.5 is the clearest case. It had the best score without tools, 55.04, so it brought the most from training. Under full retrieval it averaged 3.8 tool calls and 0.8 page reads per question, against 33.0 calls and 8.9 reads for DeepSeek V4.1 Flash, and it gained only 2.44 points. Kimi K3 started near the bottom at 51.39, read 6.6 pages a question, and finished in the leading group.

So the benchmark measures a model plus a harness plus a habit. The GPT-6 models ran in Codex CLI and Claude ran in Claude Code, with built-in search swapped for the benchmark’s tools. The open models ran in the authors’ own agent loop. A model that answers after two searches might do much better with a prompt that tells it to keep going. That’s a finding about agents as deployed, which is what the paper sets out to measure, but I wouldn’t read it as a ranking of how well each model reasons about the future.

Calibration doesn’t track accuracy either. Claude Opus 5.5 and GPT-6 Astra are within about one index point of each other, but their expected calibration errors are 0.110 and 0.058. A Brier score is strictly proper, so in expectation a forecaster can only do best by reporting what it believes, as Gneiting and Raftery set out formally. A good average score can still sit on top of probabilities that are too confident to use directly, which I wrote about from a different angle in the five-token post.

Where it fails, and what the paper flags itself

The failure analysis is more useful than the leaderboard. Taking each question’s Brier score averaged over models, retrieval fixed 41 questions and made 11 worse. On 76, about a quarter of the benchmark, the average stayed worse than a coin both with and without retrieval, and on 28 of those no model beat the coin. The traced cases are familiar to anyone who has watched an agent research. A specific but unofficial report, like a launch window relayed on social media, outweighs a newer official statement. A wrong public consensus, such as box-office forecasts that all overestimated an opening, gets retrieved and repeated. Evidence about one case gets applied to a neighbouring one because the topics match.

The paper is candid about its own construction too. The question-writing agents worked in two rounds, and the first round, covering geopolitics, sports, culture, health and weather, “searched the open web.” The main text says that “some questions were written with outcomes visible.” The authors list that in Appendix B instead of burying it, which I appreciate. It matters less here than it would elsewhere, because the forecasting agents don’t see the question writers’ searches. The risk is in the selection: questions written by an agent that can see the outcome can drift toward events that turned out to be newsworthy. The 62 percent Yes rate is consistent with that, though it doesn’t prove it.

The authors also name the second limit themselves. All 287 questions sit at one forecast date, and each model forecasts each question once per setting. Their March 2026 run, with 300 Polymarket and 232 written questions on an earlier version of the environment, is the start of a fix.

What I’d take from it

FABench’s real contribution is the environment. Freezing the news at a date, freezing Wikipedia at a revision, and screening the live web with a reviewer that sees no answer is a sound way to get instant feedback on forecasting without letting the outcome in. The paper also tests the leak instead of just asserting it’s closed, and the archive-only results make it believable.

The leaderboard itself says something narrower than the abstract. On 287 questions, the best agents remove about a third of a coin’s squared error, a little over a quarter once you account for the 62 percent Yes rate. They get there mostly by reading more pages. On 54 questions linked to mostly thin markets, under mostly mismatched rules, they land within a couple of points of the price. I’d want three things in the next release: the Yes-rate baseline on the main chart, an interval on the market comparison, and a volume floor for the markets. With those, “level with the market” would be a claim I could check.

References

  1. Wang, Y., Wang, Y. and Deng, J. (2026). Benchmarking Forecasting Agents on Yesterday’s Tomorrows. arXiv:2610.40940, submitted 29 October 2026 (HTML full text, Tables 2, 3, 7, 8 and 9, Appendices B–H).
  2. Wolfers, J. and Zitzewitz, E. (2004). Prediction Markets. Journal of Economic Perspectives 18(2): 107–126, May 2004.
  3. Strijbis, O. and Arnesen, S. (2019). Explaining variance in the accuracy of prediction markets. International Journal of Forecasting 35(1): 408–419, January 2019.
  4. Page, L. and Clemen, R. T. (2013). Do Prediction Markets Produce Well-Calibrated Probability Forecasts? The Economic Journal 123(568): 491–513, published online December 2012.
  5. Gneiting, T. and Raftery, A. E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association 102(477): 359–378, March 2007.