UK AISI and Thorn publish safety protocol to prevent AI-generated CSAM

The UK AI Security Institute and the child-safety organisation Thorn published a joint protocol setting out “safe-by-design” principles for AI developers and the platforms that host their models, aimed at preventing systems from being used to generate child sexual abuse material (CSAM). The protocol sets out approaches for identifying misuse patterns and for building access controls and monitoring into models and hosting infrastructure before release, rather than treating CSAM generation as a problem to be caught after the fact.

The two organisations cited a specific measure of scale from the Internet Watch Foundation: 3,512 AI-generated CSAM images found on a single dark-web forum in the course of one month. AISI said it was also running its own threat modelling and funding further research into defences across the AI lifecycle.

The publication followed legislation enacted in the UK the previous month that permits authorised organisations — including AI developers and child-protection groups such as Thorn — to test whether models can be made to generate CSAM under strict, controlled conditions. Before the change, a blanket prohibition on creating or possessing CSAM meant that developers and safety researchers had no lawful way to test their own systems for this failure mode directly, leaving vendors largely reliant on downstream reports of misuse rather than pre-deployment evaluation. AISI said it would consult with industry, law enforcement and academic experts to establish operational safeguards for this newly permitted testing.

The move sits alongside a broader pattern through 2025 of governments carving out narrow legal exemptions to let safety researchers probe for specific categories of harm — cyber-offence capability and biological-weapons uplift among them — that would otherwise be illegal to test directly.