'The Illusion of the Illusion of Thinking' rebuts Apple's reasoning-collapse paper

Alex Lawsen, a researcher at Open Philanthropy, published a rebuttal to Apple’s “Illusion of Thinking” paper, listing Anthropic’s Claude Opus model as a co-author for its role in reproducing and analysing the experiments. Titled, in deliberate parody of the original, “The Illusion of the Illusion of Thinking,” the paper argued that Apple’s headline claim — that large reasoning models suffer complete accuracy collapse past a complexity threshold — was largely an artefact of how the experiments were scored rather than evidence that the models could not reason.

Lawsen identified two specific problems with Apple’s setup. First, several of the puzzle tasks required models to output an exhaustive list of every move in a solution; for the largest Tower of Hanoi instances this list alone exceeded the models’ output token limits, so models that plainly understood the correct solution were scored as failures simply because they ran out of room to write it out in full. Second, some of the River Crossing instances used at higher complexity levels were, Lawsen argued, mathematically unsolvable given the boat capacity specified — models were penalised for failing to solve puzzles that had no valid solution. When Lawsen reran a subset of tests changing only the output format — asking models for a compact generating function or algorithm rather than an exhaustive move list — he reported that Claude, Gemini and OpenAI’s o3 solved 15-disk Hanoi problems, well past the point at which Apple’s paper had reported collapse.

The rebuttal did not claim to have shown reasoning models reason without limits, only that Apple’s specific experimental design could not support the strong claim that had been drawn from it. It was widely covered as a corrective to a paper that had itself received heavy attention, and the exchange became a frequently cited case study in how conflating an evaluation harness’s limitations with a model’s underlying capability can produce a headline finding that does not survive closer scrutiny.