Artificial Intelligence

Major AI firms back new multi-company web scraping opt-out standard

Major artificial intelligence companies are backing a new standard that allows publishers to apply AI training preferences across multiple platforms at once.

Radwan AbdulhadiEditor and technology writer

Published: October 30, 2026 · Updated: October 30, 2026 · 3 min read

Major AI firms back new multi-company web scraping opt-out standard — Image: The Verge

A New Unified Approach to Web Scraping Control

The leading names in artificial intelligence have announced plans to adopt a unified standard that publishers can use to opt out of being scraped by AI systems, as [reported earlier by Search Engine Roundtable](https://www.seroundtable.com/google-ai-training-control-42430.html).

With this new control, publishers will be able to apply their AI training preferences across multiple AI companies simultaneously, instead of having to configure and set them individually for every single platform.

How the Shared Policy Token Signal Works

The initiative centers around a [shared “policy token” signal](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/) proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF).

This mechanism is designed to live within a website’s [robots.txt file](https://www.theverge.com/24067997/robots-txt-ai-text-file-web-crawlers-spiders), a standard tool that has historically signaled which web crawlers are allowed or blocked from scraping a site for tasks like search indexing, archiving, or AI training.

Under the current system, website owners must signal their AI training preferences separately to each distinct AI developer, creating a cumbersome administrative burden as the ecosystem expands.

Industry Context and Existing Solutions

The announcement arrives as digital publishers continue to grapple with a steady influx of automated AI crawlers scraping their content to feed into large language models.

Various organizations and technology providers have previously attempted to combat unauthorized scraping through different measures, such as the [Really Simple Licensing (RSL) standard](https://www.theverge.com/news/841222/rsl-licensing-ai-spec-launch) giving publishers the ability to charge crawlers that visit their sites, or through [Cloudflare blocking AI crawlers by default](https://www.theverge.com/news/695501/cloudflare-block-ai-crawlers-default).

Company Commitments and Granular Controls

Major industry players including [OpenAI](https://openai.com/index/helping-websites-express-ai-training-preferences/), [Google](https://developers.google.com/search/blog/2026/10/ai-training-control), [Microsoft](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/), and [Anthropic](https://mailarchive.ietf.org/arch/msg/ai-control/mddjatOxSR0h6GmLRBWlc839c7w/) have confirmed they will restrict their respective AI crawlers from accessing any site that implements the shared token.

In addition to basic opt-outs, Microsoft announced it [will offer more granular controls](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/) that provide publishers the ability to specifically opt out of AI training or AI grounding.

AI grounding is the process of pointing an AI model to a verified source of information to help reduce hallucinations. Microsoft explicitly noted that participating websites will still continue to appear in standard Bing search results regardless of their chosen preferences.

Rollout Timelines

Implementation schedules vary among the participating technology companies. While Anthropic has already adopted the signal, Google and Microsoft have laid out plans to support it next year.

Meanwhile, OpenAI has stated that it intends to implement the standard in the coming months as the broader web community shifts toward more standardized consent mechanisms for artificial intelligence.