Google, OpenAI, Anthropic Join New Scraping Controls
Major artificial intelligence companies have committed to honoring a new robots.txt directive that allows publishers to control how their content is used for generative AI training.
Radwan AbdulhadiEditor and technology writer
New Scraping Controls for Publishers
Google, Anthropic, OpenAI and Microsoft have committed to honoring a new robots.txt directive. This directive will allow publishers to specify their preferences when it comes to the scraping of their content for generative AI training. Details of this initiative were outlined in a recent report by Engadget, noting that the companies have decided to adopt a proposal under consideration at IETF, the open standards organization for the internet.
The framework is based on a proposal written by Anthropic technical lead for standards Kevin Kelley. According to OpenAI and Google announcements, website owners currently have to identify individual crawlers and add separate rules for each provider. Kelley's proposal aims to change that by getting multiple AI companies to recognize and honor a single common signal instead.
How the New User-Agent Directive Works
Under the new system, publishers must set up a new user agent in the robots.txt file that is specifically designated for AI training. This configuration blocks crawlers specifically meant to gather data for AI training while leaving standard search indexing intact. Furthermore, the 'Allow' section gives website operators the option to permit scraping for AI training on specific parts of the page, such as a press kit.
This new user-agent section does not prevent crawling altogether or stop content from showing up on regular web search engines. For instance, Google will specifically interpret it as a stop sign for the 'Google-Extended' user agent, which crawls websites for materials used to train Gemini models.
Limitations and Past Scraping Data
An important caveat regarding the new directive is that it cannot undo any past scraping the companies have already completed. If the firms crawled a page previously, they already possess its content in their training sets. This framework is strictly designed to prevent them from scraping websites or specific pages going forward.
Timeline for Implementation Across Companies
Adoption schedules vary among the participating technology giants. Anthropic has already started supporting the new directive, marking the earliest rollout among the major AI developers. Meanwhile, Microsoft announced that it will start supporting it on February 1, 2027.
Google stated it will support the directive in early 2027 as well, joining the timeline of other key players. OpenAI has similarly pledged to implement the new signal in the coming months, completing the collective industry commitment to standardizing AI training preferences.
Sources
- EngadgetGoogle joins OpenAI and Anthropic in promising new scraping controls
Continue chronologically
Related entity coverage
Related stories
Major AI firms back new multi-company web scraping opt-out standard
OpenAI, Google, Microsoft, and Anthropic plan to adopt a new shared policy token standard allowing publishers to manage AI training preferences collectively.
Google's SynthID AI Detector Goes Public for Everyone
Google's SynthID Detector is now available globally for regular users, offering a simple way to check if images, videos, or audio files are AI-generated.
Google and Meta Challenge OpenAI and Anthropic in AI Race
Google and Meta have unveiled powerful new frontier models, challenging Anthropic and OpenAI and completely reshaping the competitive AI landscape.
Anthropic, OpenAI, Google, and SpaceXAI Face Antitrust Lawsuit
A class-action lawsuit claims major AI firms violated antitrust laws by coordinating to slow development, alleging a self-serving strategy for market control.
