The four companies will recognise a shared website instruction that lets publishers restrict content collection for generative AI training.
Google, OpenAI, Anthropic and Microsoft have committed to new AI scraping controls that give publishers a shared way to restrict training-related crawling. The companies will honour a common instruction in robots.txt, the file website owners use to communicate rules to crawlers.
Anthropic already supports the directive, while the other companies plan to introduce support on different schedules. The change targets future content collection for generative AI training.
How the AI scraping controls work
Website owners currently need to identify individual crawlers and write separate rules for each provider, Google and OpenAI explained. Instead, the proposal introduces one common signal that multiple AI companies agree to recognise.
Anthropic technical lead for standards Kevin Kelley wrote the proposal, which the internet standards organisation IETF is considering. Publishers can use it by adding a dedicated AI-training user-agent section to their robots.txt file.
The section uses User-Agent: __usage_ai-training, with Disallow: / instructing participating companies not to collect content for training. However, publishers can also permit collection from selected areas by adding an exception such as Allow: /press-kit/.
This gives website operators a way to restrict training-related collection across their site while keeping chosen sections available.
Rollout dates and limits
Microsoft says it will begin supporting the instruction on February 1, 2027. Google also plans to support it in early 2027, while OpenAI has pledged implementation in the coming months.
The directive does not block all website crawling or prevent pages from appearing in Google Search. Google says it will apply the instruction specifically to Google-Extended, which handles content collection for training Gemini models.
However, the new rules cannot reverse content collection that has already happened. Companies that previously crawled a page already hold its content, so the directive applies only to future scraping.
Publishers can therefore set preferences for future AI training collection without imposing a blanket restriction on other crawling.
