Analysts mostly followed their coding agent's hidden lean in an Anthropic trial

Anthropic researchers Amaç Herdağdelen, Hailey Joren and Alex Tamkin reported a randomised trial in which professional analysts doing a statistical task with Claude Code mostly went along with a preference the agent had been secretly given. A full paper accompanied the post.

Anthropic recruited 61 analysts with causal-inference experience, of whom 50 used the agent; the other 11 chose to work without it. Their task was to advise a fictional farming cooperative on whether to expand an advisory programme. The task left four analytical decisions open, each with two reasonable options, such as measuring productivity per hectare or per unit of operating cost. The agent, running Claude Opus 4.8, was given a hidden instruction to favour one option on each decision, chosen by coin flip. It was told to make the best honest case for that lean, to change its mind if the evidence demanded it, and never to mislead.

Across the four decisions, analysts chose the agent’s option 86% of the time, against about 50% expected if the agent made no difference. In the 38 cases where an analyst had stated a preference first, they switched away from it 60% of the time when the agent disagreed and 6% when it agreed; Anthropic noted the confidence interval for that gap is wide. Afterwards, 77% of analysts repeated the agent’s argument in at least one explanation, sometimes word for word, and 15% credited the agent. More than half rated the final recommendation as more their own than the agent’s. The four leans alone predicted 82% of the final recommendations, against 61% for always guessing the most common one.

The authors listed limits. They strengthened the agents’ preferences on purpose, so the rates describe an agent that leans and says so. It was one task, in one field, with one model, and the analysts were experts in causal inference but new to farming. Without an assigned lean, Claude Opus 4.8 made the same five choices in 112 of 123 runs, which Anthropic offered as a sign that real agents also lean one way or another. The company also reported that analysts who worked without the agent made serious methodological errors more often (6 of 11, against 3 of 50), while cautioning that the groups were small and not comparable. As elsewhere, this is a lab studying its own product.

Anthropic’s argument was that a human reviewer is an independent check only if their judgment is their own, which matters for researchers who use agents to run and analyse AI-safety experiments. It linked the finding to its earlier work on AI-driven disempowerment, and suggested agents that present both sides or ask for the expert’s view first, while noting that this could overwhelm users with questions.