OpenAI and Google say they will honour a shared opt-out signal for AI training

OpenAI and Google each announced on 30 October that they would honour a common signal in robots.txt, the file websites use to tell crawlers what they may fetch, for a site’s preference on whether its content is used to train generative AI models. OpenAI called the arrangement a “voluntary effort”. The signal is defined in an Internet Engineering Task Force (IETF) draft written by Kevin Kelley, listed as affiliated with Anthropic, dated the same day.

OpenAI said that site operators have until now had to name individual crawlers and keep separate rules for each provider, and that a shared signal reduces that work. It said it aims to implement the signal “in the coming months” and will publish details when it does. The new signal does not replace its existing crawler controls, which “will continue to apply and be respected”. It said it was joining other participants in the IETF’s AI Preferences working group in the announcement, and encouraged other AI providers to honour the signal.

Google’s post described a “policy token” that a site names in a robots.txt group. If a site uses the token to opt out of training and does not separately address Google-Extended, Google’s existing training control, Google said it would treat the site as having disallowed Google-Extended. Google-Extended stays available, so a site can set a broad preference and still write rules for individual developers. Google gave 2027 as its target for support, and its post did not name other companies taking part.

The status of the proposal is modest. The Datatracker lists the draft as an individual Internet-Draft, not a working-group document: it is not endorsed by the IETF and has no formal standing, and it expires in May 2027. It specifies a token, __usage_ai-training, written as a User-Agent line in an ordinary robots.txt group, and says it adds no new syntax and is not a crawling directive. The draft says that where no group names the token, no preference is expressed. Neither company’s post addressed content already collected, and the sources consulted contain no reaction from publishers.

If adopted, the practical effect is a single place to record an opt-out rather than one per crawler, with compliance resting on each provider’s choice to honour it.