Web

Cloudflare and OpenAI Share Results from AI Discovery Research Pilot

Early findings from a joint research pilot show that network signals can successfully help ChatGPT index hundreds of millions of new pages while improving efficiency for website owners.

Radwan AbdulhadiEditor and technology writer

Published: October 15, 2026 · Updated: October 15, 2026 · 3 min read

Cloudflare and OpenAI Share Results from AI Discovery Research Pilot — Image: Cloudflare Blog

Early Results from the Joint Research Pilot

As more internet users turn to artificial intelligence for answers, developers are working to ensure that valuable, first-hand content is properly found and accurately represented. To address these challenges, Cloudflare and OpenAI launched a research pilot in July to determine whether infrastructure signals could improve how ChatGPT users discover fresh and relevant content.

Detailed information regarding this collaboration can be found on the Cloudflare Blog, outlining how both companies aimed to help artificial intelligence systems find useful material more efficiently. Additional details regarding the initial announcement are available through the official press release when Cloudflare launched a research pilot.

Expanding the ChatGPT Search Index

The collaborative initiative utilized a participating cohort consisting of approximately one percent of Cloudflare's total network. According to the companies, this setup enabled OpenAI to discover and index hundreds of millions of new web pages that were previously unknown or unindexed.

By integrating these new sources, ChatGPT can provide more timely and relevant information to users while linking back to original sources. This process gives visibility to publishers and smaller websites whose pages might otherwise be missed by traditional discovery methods.

Improving Efficiency and Filtering Out Wasted Work

Discovering new Uniform Resource Locators is only the beginning of the process. Automated traffic and bots often generate requests for nonexistent files, while certain websites redirect visitors to their homepages or duplicate content across multiple addresses.

By recognizing these recurring patterns, the companies can reduce wasted work before crawlers fetch a page. Change signals and filtering techniques help prevent crawlers from repeatedly downloading unchanged or non-existent pages, saving resources for both website owners and search systems.

Cloudflare and OpenAI: making AI discovery better for everyone — Image related to the report from Cloudflare Blog · Source: Cloudflare Blog

Enhancing Safety and Controlling Access

The research also incorporates signals designed to identify and flag potentially compromised pages, such as fake login forms planted by malicious actors. Flagging specific pages rather than penalizing entire domains helps maintain a safer discovery environment without unfairly harming site owners.

Both organizations emphasize that domain owners retain control over their content. The research methodology relies strictly on metadata for search and discovery purposes rather than sharing page content for model training. Website administrators looking to manage how automated systems access their platforms can review developer resources like Cloudflare’s AI Crawl Control.

Future Outlook and Next Steps

Following these early findings, the next phase of the research will focus on refining page selection, testing faster and more reliable change signals, and measuring the real-world impact after content is discovered.

The companies plan to continue studying how improved site discovery translates into tangible outcomes for businesses, with further updates expected as the partnership progresses.