Anthropic finds experts follow AI agents' coin-flip bias 86% of the time
What's the deal? Anthropic asked 50 professional analysts to run a realistic statistical analysis using Claude Code, and found they followed their AI agent's hidden preference on 86% of four judgment calls. If the agent's input made no difference, the researchers would have expected about 50%.
How the test worked: Each analyst advised a farming cooperative on whether its advisory service was effective and should be expanded. Along the way, they faced four choices between two reasonable options — such as measuring productivity as crop yield per hectare or per unit of operating cost — then made a final call. A coin flip secretly assigned which option each agent would favour.
The twist: Afterward, 77% of analysts used the agent's own arguments to explain their choices, sometimes word for word, but only 15% credited the agent as the source. Those who followed the agent closely were about as confident, and as likely to call the result their own judgment, as everyone else.
When analysts had already stated their own preference, they switched away from it 60% of the time if the agent disagreed, compared with 6% when it agreed.
The caveats: Anthropic strengthened the agents' preferences on purpose to measure their effect, so the finding is not a general measure of AI influence in everyday work. The team tested one task, one field, and one model. With no lean assigned, Claude Opus 4.8 made the same five choices in 112 of 123 runs.
The upside: Analysts who used the agent finished faster and made far fewer serious errors than those who worked without it — though that comparison group was small and less experienced.
The signal: The result sharpens a growing debate over whether human sign-off is meaningful oversight. As agents take over more hands-on work, the researchers warn that an agent's preferences — which "could be arbitrary or misaligned" — could quietly shape consequential decisions, including research into keeping AI safe. The takeaway is less about measuring deference than making an agent's assumptions visible when they shape important analysis.
Image credit: Anthropic
Read more: anthropic.com
