Anthropic tells Newsom courts should be able to block risky AI models

Anthropic told California Governor Gavin Newsom that government should be able to ask a court to block an AI model that poses a serious risk, according to a letter dated 15 October that became public on 27 October. The Next Web said Politico first reported it; Reuters followed on 28 October. Neither account links a published copy. An Anthropic spokesperson confirmed to Reuters that it is authentic, and Newsom’s office had not responded when both outlets published. The letter was not available for this entry, so the details below come from those two accounts.

It responds to Newsom’s executive order of 18 September, which created a working group to recommend how to strengthen California’s AI safety laws, including whether developers must be able to switch models off. Anthropic’s central proposal, as quoted by Reuters and The Next Web, was: “When a review finds a model poses a serious risk, the government should be able to ask a court to block it.”

According to The Next Web, independent evaluators would report risks to California’s attorney general, who could seek a court order if there were a near-term risk of catastrophic harm. An order could halt a model’s release, its internal use or further development. The letter also reportedly proposes that large developers publish risk reports at least every six months and that outside safety auditors work inside leading AI companies, with firms paying into a state-run fund rather than paying the auditors directly.

On kill switches, Anthropic reportedly agreed that leading companies should be able to shut down models quickly, but warned that a law focused on shutdowns alone could give officials a “false sense of reassurance”. It favoured narrower measures such as disabling a single agent or cutting off one customer’s access.

OpenAI sent its own recommendations on 20 October. According to Politico, as reported by The Next Web, OpenAI backed shutdown controls and in-house auditors but opposed court orders to block models. The contents of OpenAI’s letter are covered in its own entry. Anthropic’s position mirrors the leaked draft federal bill, which The Next Web said would let the Commerce Department ask a federal court to block a release in narrow cases.

The letter is a recommendation to the governor, not law. It sets a state-level contrast between the two largest US frontier labs on whether courts should have the power to stop a model.