The Rise of the Second Audience

For the majority of internet history, the web catered primarily to a single audience of human users who read articles, viewed advertisements, and purchased subscriptions. While bots have always existed, they were traditionally large automated operations that did not consume media in any meaningful sense. According to data released on the Cloudflare Blog, that dynamic is shifting dramatically as software-driven visitors outpace people.

At the end of 2024, Cloudflare handled an average of 63 million HTTP requests per second. That figure has nearly doubled to 115 million requests per second, with peak volumes exceeding 150 million. Over the past year, daily requests originating from AI agents on the network expanded by more than 1,700%, pushing non-human traffic above the 50% threshold for the first time.

Economic Strain on Website Owners

The traditional web economy relied on search engines sending human visitors to publisher websites, where ads or payments converted traffic into revenue. The rise of answer engines has disrupted this balance by summarizing page content directly for users, which avoids sending human visitors to the underlying sites. Heavily crawled industries—including retail, computer software, IT and services, and financial services—have experienced human traffic drops of up to 40% in less than a year.

This shift has increased operational costs through higher bandwidth and compute usage while reducing revenue per request, as automated visits often carry no referral, ad impression, or subscription. To address this evolving landscape, Cloudflare outlines a vision for the Agentic Internet we're building that aims to make automated interactions sustainable for publishers.

Granular Controls for Site Owners

In response to the influx of automated traffic, Cloudflare introduced granular management options to help site administrators distinguish between different types of non-human visitors. In July, the platform replaced a single blocking switch with separate Search, Agent, and Training controls, allowing site owners to fine-grained access permissions across all plans.

Additionally, to address mixed-use crawlers that combine data collection for search indexes and AI model training, Cloudflare launched a dedicated feature. The Disallow AI Training option enables site operators to keep their pages indexed for search engines while instructing specific crawler operators not to utilize their data for training machine learning models.

Transparency and Verification

Identifying and categorizing automated traffic remains a central challenge for website publishers. Cloudflare notes that generic terms like 'AI bot' are no longer sufficient, and site operators require deeper visibility into which automated visitors are consuming resources and taking specific URLs.

To help verify the identity of automated systems, major operators such as OpenAI, Google, and AWS cryptographically sign their agent requests. This verification process allows site owners to distinguish legitimate software agents from impersonators without relying solely on IP addresses or user-agent strings. Furthermore, public data and metrics regarding crawler behaviors can be monitored directly through Cloudflare Radar.

Sources

• Cloudflare BlogThe Internet has a second audience