AI Agents Finished 167 of 400 Psychology Experiments. Their Data Looked Human on the Biases, Not the Abilities.

A paper posted to arXiv on 27 October, by Kianté Fernandez and colleagues in UCLA’s psychology department, starts from a worry that has been growing in the social sciences: “AI agents can now complete online behavioral tasks and produce human-like data, which raises concerns that online datasets in the social and behavioral sciences are increasingly open to contamination.” Their benchmark, CogArena, tests the premise. Agents take ten classic experiments through a standard web browser, seeing the same screens and pressing the same keys as human participants, and are scored on three levels: did they finish, how accurate were they, and did their data show “the behavioral signatures each paradigm reliably produces in people.”

The answer to the first question is “often not,” and the answer to the third depends on which kind of signature you ask about.

Finishing is the first hurdle

The study ran four models, all from the cheap end of their providers’ lineups: kimi-k2.5, gemini-3-flash-preview, qwen3-vl-235b-instruct and gpt-5.4-nano. Each did each task ten times, for 400 sessions. “Only 167 of the 400 sessions passed all five Level 1 checks.” The time and step limits explain little of the rest. Of the 233 failures, 191 “ended because the agent declared the task finished when the experiment had not yet ended.”

How the 400 agent sessions ended One bar of 400 sessions. 167 passed all five completion checks. Of the 233 that failed, 191 ended because the agent declared the task finished early, 30 ran out of time, 11 were stopped by the scaffold after ten failed actions in a row, and 1 ran out of steps. 400 SESSIONS · 4 MODELS × 10 TASKS × 10 RUNS · COGARENA 167 passed all Level 1 checks — CogArena, arXiv, 27 Oct 2026 191 agent declared the task finished early 30 ran out of time 11 stopped by the scaffold, 1 out of steps 167 finished 191 declared done early 30 out of time 11 stopped by scaffold, 1 out of steps A random agent that presses a key on every trial completed every session it started. The best model, gemini-3-flash-preview, reached a Level 1 score of 0.936.
Most failures were not timeouts. The agent decided it was finished while the experiment was still running.

The authors are careful about what finishing means. A random agent “completed every session it started,” and none of the models matched it. “Finishing an experiment is therefore not by itself evidence of ability.” It is evidence that the agent and the platform worked together long enough to produce data.

Some signatures every time, some never

Where sessions produced data, the agents scored well above chance on average. The average hides a split. Some signatures were reproduced in nearly every session that could test them: telling phishing from legitimate email above chance (90%), preferring to save more people in moral dilemmas (100%), and cooperating after a partner cooperated in the prisoner’s dilemma (100%). Others were reproduced in no session at all: recognizing previously seen images above chance (0 of 25), staying longer in a patch when travel is costly (0 of 32), and responding faster when motion is clearer (0 of 21, which the authors partly blame on the browser scaffold, since a single screenshot carries no motion).

Signatures reproduced in nearly every testable session, and in none Horizontal bars on a 0 to 100 percent scale. Reproduced: phishing discrimination 90 percent of testable sessions, saving more people in moral dilemmas 100 percent, cooperating after cooperation 100 percent. Never reproduced: recognizing seen images 0 of 25, staying longer when travel is costly 0 of 32, faster responses at higher motion coherence 0 of 21. SHARE OF TESTABLE SESSIONS REPRODUCING EACH SIGNATURE · POOLED PANEL Reproduced almost always Never reproduced Phishing beats chance Save more people Cooperate after cooperation Recognize seen images Stay longer when travel costs Faster at clearer motion 90% of testable sessions — CogArena 100% — CogArena 100% — CogArena 90% 100% 100% 0 of 25 0 of 32 0 of 21 The motion task is partly a scaffold limit: a single screenshot carries no motion.
The same agents that reliably cooperated and spotted phishing never once showed above-chance memory for images or sensitivity to travel cost.

The authors sort the signatures into two kinds and the split follows it. “Over measurable sessions, agents reproduced capacity signatures in 31% of the cases in which they could be tested and bias signatures in 66%.” In the paper’s terms, a capacity signature is a pattern an ideal performer would also show, such as accuracy rising as motion gets clearer, so failing it is a performance deficit. A bias is a pattern that departs from any normative benchmark, or a preference no benchmark prescribes, like sparing more people in a moral dilemma. The agents imitate the second kind far more readily than the first. And finishing did not close the gap: “Even among the sessions that passed every Level 1 check, only 40% of testable signatures were reproduced.”

Capacity versus bias signatures reproduced by the agents Horizontal bars on a 0 to 100 percent scale. Bias signatures reproduced in 66 percent of testable cases. Capacity signatures in 31 percent. Among sessions that passed every completion check, 40 percent of testable signatures were reproduced. SIGNATURES REPRODUCED, SHARE OF TESTABLE CASES · COGARENA Bias signatures Capacity signatures All, completed sessions only 66% — CogArena 31% — CogArena 40% — CogArena, sessions passing every Level 1 check 66% 31% 40% Of 1,560 signature observations, 497 (32%) could not be tested, mostly because a session ended early.
Agents look most human on the tendencies people show in their choices, and least human on the abilities experiments measure.

What it means for contamination

This cuts two ways for researchers who worry about agents in their samples. The reassuring reading is that a cheap agent taking a study often quits early, or produces data without the capacity signatures a real participant shows. The less reassuring reading is the authors’ own: “The human patterns that current agents reproduce most reliably are preferences, so screening respondents by whether they show the expected preference will not separate agents from people.” Every agent session that could be tested preferred saving more lives. Response time is a better tell for now, since agents took up to ten times longer than people to respond, mostly because each response waits for the model to process a screenshot. Survey researchers have found the same problem from the other side: a 2025 paper in the International Journal of Social Research Methodology reported that established bot detection is not reliable for LLM-driven bots.

The study’s own limits point to the next run. All four models were economy-tier. A single state-of-the-art model, gpt-5.6-sol, run once per task, had a perfect completion score on nine of ten tasks and beat the panel’s mean alignment score on eight, and the authors say much of that gap came “from finishing experiments that the economy panel abandons rather than from closer alignment.” Scores also varied a lot between repeats: in 28 of 40 model-task cells, the same model scored zero on alignment in some sessions and above zero in others. CogArena publishes its tasks, scoring code and a public leaderboard, so those are now measurable rather than guesses.

References

  1. Fernandez, K., Chen, C., Zhou, J., Sadowski, B., Miceli, A. C., Krajbich, I. et al. (2026). CogArena: Benchmarking Multimodal Agents on Interactive Behavioral Experiments. arXiv:2610.36484, 27 October 2026.
  2. Höhne, J. K., Claassen, J. & Wolf, B. L. (2025). LLM-driven bot infiltration: protecting web surveys through prompt injections. International Journal of Social Research Methodology 29, 999–1006.