Twenty-Seven of Thirty-Three Left Blank

On 2 November Anush Sankaran and Arijit Banerjee, both at Microsoft, posted a short position paper to arXiv titled “We Ran a Research Sprint With AI. The AI Worked. The Research Didn’t.” The setup is small. Twelve applied AI researchers were split into three teams of four and given 100 minutes to produce an empirical paper. Every desk had Claude, Claude Code, ChatGPT and Copilot. A silent observer sat with each team and logged every AI interaction along with what the team did with it: accepted, modified, rejected or unresolved.

The abstract leads with “one was accepted outright,” which reads like a story about people distrusting the tools. Go to Figure 2 of the paper and the actual numbers are different. Thirty-three interactions were logged. One was accepted and five were modified. None were rejected, and 27 have no disposition recorded at all. To me the blank column says more than anything else in the paper, and what it says isn’t distrust.

The room

The session had seventeen people in it. Besides the twelve researchers there were three observers, all AI researchers who didn’t take part in the task, plus an orchestrator and a critic. Participants were volunteers from an internal cohort, already familiar with the tools. The task was itself about research process: “how do we capture the evolution of a research idea, and find the gaps in it, in a human-agent setup?” Hard cutoffs split the 100 minutes into ideation, design, execution, analysis and writing.

Who was in the seventeen-person session A single bar of seventeen equal cells. Twelve cells are task participants, in three teams of four. Three cells are silent observers, one per team. One cell is the orchestrator and one is the critic. The session ran for 100 minutes with access to Claude, Claude Code, ChatGPT and Copilot. SEVENTEEN PEOPLE, ONE 100-MINUTE SESSION · SANKARAN AND BANERJEE 2026, §3 Team 1 researcher — §3 Team 1 researcher — §3 Team 1 researcher — §3 Team 1 researcher — §3 Team 2 researcher — §3 Team 2 researcher — §3 Team 2 researcher — §3 Team 2 researcher — §3 Team 3 researcher — §3 Team 3 researcher — §3 Team 3 researcher — §3 Team 3 researcher — §3 Silent observer — §3 Silent observer — §3 Silent observer — §3 Orchestrator — §3 Critic — §3 Team 1 Team 2 Team 3 Observers 12 task participants, 4 per team, no shared channel between teams 1 per team, logging dispositions orch., critic Tools available to all teams: Claude, Claude Code, ChatGPT, Copilot. No facilitation on prompting or division of labour. A separate autonomous pipeline got the same problem statement and ran with no human input.
Twelve people did the work and three watched. With a ratio like that you'd expect a complete log, and the log came back mostly empty.

A separate autonomous pipeline got the same problem statement once the teams had locked theirs, and it ran with no human input for the rest of the session. It produced a complete draft. The authors limit what that shows to production capacity, “not superior research quality,” because nobody scored either the pipeline’s draft or the team drafts independently.

What the blanks mean

Here is the log, as the paper reports it.

Disposition of 33 logged AI interactions Bar chart of the 33 AI interactions observers logged across three teams. One was accepted (3 percent), five were modified (15 percent), zero were rejected, and twenty-seven (82 percent) had no disposition recorded. 33 LOGGED AI INTERACTIONS, THREE TEAMS · SANKARAN AND BANERJEE 2026, FIG. 2 Accepted Modified Rejected No disposition recorded Accepted: 1 of 33 (3%) — Sankaran and Banerjee, Fig. 2, 2 Nov 2026 Modified: 5 of 33 (15%) — Fig. 2 No disposition: 27 of 33 (82%) — Fig. 2 1 · 3% 5 · 15% 0 27 · 82% A missing entry means no team-visible decision moment occurred. It does not show that nobody evaluated the output.
Six of 33 interactions had a decision anyone could see. Zero were rejected. The one accepted outright is the least interesting number on the chart.

The authors are careful about the blanks: “missing records alone do not establish absent evaluation.” A blank could be an observer who fell behind. It could be a researcher who read the output privately, decided, and moved on without saying anything. But the paper also says that observer logging “depended on a team-visible decision moment.” So at minimum, a blank means that any decision about that output happened somewhere the rest of the team couldn’t see. That is a finding about the team, whatever it says about any single person.

Then there is the zero in the rejected row. In 100 minutes of building an empirical paper with four assistants, the observers never once saw a team look at an AI output and throw it away. Maybe every output was good. The participants don’t think so. Two of the three teams independently wrote the same phrase in their self-assessment, “plowed ahead instead of vetting.” Team 3 put it this way in a mid-session survey: “No time to consume AI generated results and make knowledgeable decisions on it.”

The paper calls this the consumption bottleneck, meaning AI produces output faster than the team can evaluate it. The word doing the work is “evaluate,” which the paper defines as reading “closely enough to catch a substantively wrong claim,” and the authors note that under a deadline that kind of reading “is the first activity to be deferred.” Part of the lost time went somewhere duller. Two teams reported copy-and-paste friction between Claude Code and the shared pipeline, and that ate the time that should have gone to checking.

This matches a result from a very different setting. Klingbeil, Grützner and Schreck ran an incentivised behavioural experiment in 2024 and found that “the mere knowledge of advice being generated by an AI causes people to overrely on it,” following the advice even when it contradicted the context they had and their own judgement. In their study the cost of over-reliance fell partly on third parties. In a research team, the third parties are the teammates who build on a claim nobody checked.

The second failure is the shape of the room

The other pattern is what the authors call AI-induced research silos. Each researcher had a private loop with a model, and those loops didn’t expose their context to anyone else. Team 2’s most concrete artifact was a hypothesis summary Claude had drafted, “with acronyms without any context, making it difficult to understand for other team members.” Team 1 said integration was done “subjectively by a team member.” Team 3 named it mid-session: “AI is multiplier for personal work not team work.”

Three ways to arrange people and models Three panels from the paper's Table 1. Human-AI: each researcher has a private session with a model, and the contexts stay separate; this is the default, and private contexts may fragment team evaluation. AI-AI: one pipeline runs from problem statement to draft with no human in the loop; verification is deferred downstream. Human-AI-Human: researchers connect through a shared AI context to a team-visible checkpoint before a decision; this is a design agenda and collective evaluation remains to be tested. THREE INTERACTION TOPOLOGIES · SANKARAN AND BANERJEE 2026, TABLE 1 Human-AI AI-AI Human-AI-Human The default The trajectory The design agenda Four people, four private sessions, no shared context task agent draft No human in the loop. Checking moves to readers. shared AI context team checkpoint AI connects contexts; claims pass a visible gate first Risk: fragmented evaluation Risk: verification deferred Status: untested Circles are researchers, boxes are AI sessions or artifacts. The dashed checkpoint is a proposal, not something the pilot built.
The pilot ran the first two arrangements side by side. The third is what the participants asked for when the session ended.

When the debrief asked what they would change, the paper reports, “no team asked for stronger models or longer context. They asked to take the parallelism out.” One participant asked for “a way to share sessions to have a final consolidation.” Team 1 had already done a cruder version mid-session. About thirty minutes in, it switched the tools off for experimental design and wrote down why: “No AI tool used to design experiment.” That was the stage where a wrong output couldn’t be undone.

There is a known result behind the silo idea, and the authors use it carefully, as an analogy. Doshi and Hauser gave some short-story writers ideas from an LLM. Those stories were rated more creative, better written and more enjoyable, but they were “more similar to each other than stories by humans alone.” The individual gets better and the collective gets narrower. A research team that integrates four private sessions at the last minute faces a version of that dilemma, plus the extra problem that nobody can see where a claim came from.

How much this can carry

The paper is honest about its limits, and they’re big. It’s one session at one organisation, with twelve task participants, and the authors call it “mechanism-generating.” It has no no-AI control, so it can’t separate what AI did from what a 100-minute clock would do to any team. No artifact got an independent quality score. The limitations section says plainly that it generates hypotheses, “not causal, prevalence, or effect-size estimates.” The authors close with seven falsifiable design hypotheses rather than a result.

Three studies of people working with AI, by scale and design Three rows. The sprint pilot: twelve task participants in one 100-minute session, no control group, hypotheses only. Farach and colleagues: a field experiment with 388 employees at a Fortune 500 retailer, where a structured protocol of joint AI use within pairs was associated with lower document quality and substantially lower document production. Vaccaro and colleagues: a preregistered meta-analysis of 106 experimental studies and 370 effect sizes, finding human-AI combinations on average worse than the best of humans or AI alone, Hedges' g of minus 0.23 with a 95 percent confidence interval from minus 0.39 to minus 0.07. HOW MUCH EACH STUDY CAN CARRY · SANKARAN AND BANERJEE 2026; FARACH ET AL. 2026; VACCARO ET AL. 2024 STUDY UNIT AND SIZE WHAT IT FOUND Sprint pilot (2 Nov 2026) Farach et al. (Apr 2026) Vaccaro et al. (Oct 2024) 12 participants, 1 session 388 employees 106 studies, 370 effects Microsoft, position paper Fortune 500 retailer Nature Human Behaviour no control group field experiment preregistered meta-analysis 27 of 33 AI outputs had no recorded decision; hypotheses only Joint AI use within pairs: lower quality, much lower production Human+AI below the better of either: g = −0.23 (95% CI −0.39 to −0.07) Different units and outcomes, so the rows are not comparable effect sizes. Farach et al. flag their own confounds.
The pilot is the smallest piece of evidence here by a wide margin. It's useful because the larger studies point the same way, and because it gives the problem a shape someone can test.

Stronger work points the same way. The paper builds on a field experiment by Farach and colleagues with 388 employees at a Fortune 500 retailer. In that study a protocol requiring pairs to use AI jointly was associated with lower document quality and “substantially lower document production” than unstructured use, with confounds the authors flag themselves. Vaccaro, Almaatouq and Malone’s meta-analysis of 106 experiments found human-AI combinations on average worse than the better of humans or AI alone, with Hedges’ g of −0.23. The losses were in decision tasks and the gains were in content creation. A research sprint has both kinds of task. The participants found AI useful for producing material and “difficult to consume, verify, and consolidate under deadline.”

The cheap fix is a field

The paper’s agenda has seven items, R1 to R7, and the authors name shared context (R1) plus forced evaluation checkpoints (R3) as the smallest intervention worth testing. I’d start one step earlier, with R4: make the disposition log “team-visible by default.” The predicted effect is modest. Built-in logging should give a more complete record than an outside observer can. But the 82 percent blank rate came from three observers watching four people each, and if they couldn’t capture the decisions, nobody on the team could reconstruct them a week later either.

This is how I’d act on it. Any AI output that goes into a shared artifact gets a disposition next to it, written by a person: accepted, edited, rejected, or not checked. “Not checked” is allowed. A team that writes “not checked” 27 times in 100 minutes has learned something true about its process. A team that leaves the field blank has learned nothing. I wrote yesterday about a monitor that described the padding and flagged none of it, where seeing a thing and judging it turned out to be separate steps. This paper finds the same split in a room of people. Thirty-three outputs were logged, and only six were judged where the rest of the team could see it.

References

  1. Sankaran, A. and Banerjee, A. (2026). We Ran a Research Sprint With AI. The AI Worked. The Research Didn’t. arXiv:2611.03946, submitted 2 November 2026 (sections 3–6, Figure 2, Tables 1 and 3).
  2. Farach, A., Cambon, A., Tankelevitch, L., Hsueh, C. and Janssen, R. (2026). Scaffolding Human-AI Collaboration: A Field Experiment on Behavioral Protocols and Cognitive Reframing. arXiv:2604.08678, submitted 9 April 2026.
  3. Vaccaro, M., Almaatouq, A. and Malone, T. (2024). When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour 8: 2293–2303, published 28 October 2024.
  4. Doshi, A. R. and Hauser, O. P. (2024). Generative AI enhances individual creativity but reduces the collective diversity of novel content. Science Advances 10(28): eadn5290, published 12 July 2024.
  5. Klingbeil, A., Grützner, C. and Schreck, P. (2024). Trust and reliance on AI — An experimental study on the extent and costs of overreliance on AI. Computers in Human Behavior 160: 108352, November 2024.