Researchers from OpenAI, Anthropic and academia warn of an AI "intelligence explosion"

Twenty-two researchers, among them OpenAI chief scientist Jakub Pachocki, Anthropic co-founder Jack Clark, Microsoft chief scientific officer Eric Horvitz and the Turing Award winners Geoffrey Hinton, Yoshua Bengio and Andrew Barto, published a working paper arguing that the automation of AI research could set off an “intelligence explosion”, compressing years of progress into months or less, and that governments should prepare now. It was led by Alan Chan of GovAI and Sören Mindermann of the University of Cambridge’s Programme on AI Science and Policy, which issued it; a footnote says the views are the authors’ own, not their employers’.

The paper’s evidence is the labs’ own disclosures. It cites Anthropic’s report that AI systems completed 26% of its internal R&D work with only high-level supervision in August, up from 1% five months earlier — figures Anthropic had published that month — and OpenAI’s statement that its systems routinely complete research tasks that would take staff days. Because AI systems can be copied and run in parallel, the authors argue, automating the work of the thousands of researchers doing frontier AI research could add the equivalent of millions more, and a loop of systems building their successors could overcome the diminishing returns that would otherwise slow progress.

Its recommendations fall under three headings. For visibility, it proposes embedded auditors at frontier companies and mandatory reporting on how far research has been automated, the pace of progress and how R&D spending is allocated. For steering, it lists concrete safety requirements for continued development, “a speed limit on capabilities growth”, the option to shut down high-stakes experiments, air-gapped research environments, international incident-sharing and agreements to pace progress. For preparation, it calls for emergency plans covering AI-driven cyber and biological incidents, labour disruption and loss of control. The abstract states the uncertainty directly:

Although there remains much uncertainty about these possibilities, the high stakes warrant serious further attention. Policymakers should urgently obtain more visibility into the automation of AI R&D, develop ways to steer and constrain an intelligence explosion, and prepare society to adapt to an intelligence explosion’s impacts.

— Chan, Mindermann et al., “What if automating AI R&D triggers an intelligence explosion?”

Several of its proposals had already surfaced in the month’s debate over pacing the frontier, including California’s order to place auditors inside frontier labs.