Cloudflare adds continuous checks for Verified bots
Cloudflare is introducing a system that continuously evaluates whether Verified bots respect website instructions after their identities have been established. The company says the approach is intended to make trust depend on sustained behavior rather than verification alone.
Radwan AbdulhadiEditor and technology writer
Identity is only the starting point
Cloudflare says verifying a bot’s operator establishes identity and accountability, but does not prove that the bot will behave responsibly in the future. A known bot may still ignore website preferences, place excessive load on an origin or attempt to evade controls. The company says such behavior is not always malicious and can also result from a configuration mistake or software bug.
The new system is designed to address the second question in Cloudflare’s framework for automated traffic: not only “Are you who you say you are?” but also “Can we trust you?” Cloudflare previously described those questions in a discussion of good and bad agentic behaviors, and says Verified bots must meet both identity and conduct expectations.
From verified identity to observed conduct
Cloudflare’s existing verification model uses Web Bot Auth to sign requests and establish a bot’s identity. BotBase then catalogues that information in an open format. With identification addressed, the company is shifting attention to whether verified automation follows basic rules of responsible access.
Cloudflare distinguishes between risk and trust in this process. Risk describes the possibility that a particular request or action could be harmful at a given moment. Trust, by contrast, is built through repeated and consistent behavior. The proposed assessment therefore focuses on patterns over time instead of treating one isolated event as a permanent judgment about a bot or its operator.
Robots.txt becomes a measured preference
The first signal in the system is whether a crawler’s observed traffic is consistent with the preferences published in a website’s robots.txt file. That file can instruct all bots, or specific bots, to crawl only particular parts of a site. Cloudflare says the system is intended to identify repeated patterns that disregard those instructions rather than overreact to isolated violations.
Cloudflare describes robots.txt as the minimum set of rules for what bots should not do, not as a complete statement that everything else is permitted. A site may communicate that continued access is unwelcome through other signals even when a robots.txt file does not explicitly prohibit it. The company says responsible bots should consult and honor the preferences that websites publish.
HTTP 429 responses provide another test
The system will also examine how a bot reacts to HTTP 429, or “Too Many Requests,” responses. A 429 response tells a visitor to reduce its request rate and may include guidance about how long to wait before trying again. Cloudflare will assess whether the bot slows down, pauses or continues sending requests at the same pace after receiving that signal.
A bot that consistently slows down or pauses after a 429 response can strengthen its trust assessment, while repeatedly ignoring the signal can weaken it. Cloudflare presents this as a common-sense courtesy to website operators, whose systems provide the information that automated services seek.
Trust bands and future customer controls
The observations will be translated into an explainable trust band: a range showing how consistently a bot respects site preferences and which observed behaviors informed the assessment. Cloudflare says the band is not intended to be a permanent verdict. Instead, it gives responsible bot operators a way to demonstrate that trust is being maintained over time and gives site owners more context about the automated traffic reaching them.
Cloudflare says it is already using the system internally to evaluate Verified bots and is currently building a history of sustained behavior while refining how the observations should guide decisions. The company plans to make the system available to all Cloudflare customers, who will be able to choose the trust threshold used to decide how Verified bot traffic is handled. Cloudflare also plans to share scores publicly on Radar and expand the signals it measures.
An accountability loop for automated traffic
The continuous evaluation system is intended to work with Cloudflare’s Bot Preference Sync and Bot Management products. Bot Preference Sync helps website owners publish their preferences, while observing bot interactions with those preferences can show whether Verified bots honor them consistently. Bot Management allows site owners to decide how the resulting automated traffic should be handled.
Cloudflare points to its 2025 decision to remove Perplexity’s bot from the Verified bot list as an example of why ongoing accountability matters. The company said the bot sought access to resources on websites that had opted out of scraping and later attempted to circumvent controls after those requests were blocked. Under the new approach, verification is paired with continuing evidence of behavior rather than treated as a one-time guarantee.
Sources
- Cloudflare BlogContinuous accountability for Verified bots
Continue chronologically
Related entity coverage
Related stories
Websites Are Blocking AI Agents From Accessing the Web
Websites and financial institutions are increasingly blocking AI agents from shopping and managing tasks due to security and commercial concerns.
Websites Move to Block Autonomous AI Agents
Major websites, airlines, and financial institutions are blocking AI agents from OpenAI, Meta, and Google, undermining their utility.
Cloudflare Redesigns Radar to Broaden Data Accessibility
Cloudflare has redesigned its Radar platform to make real-time global internet traffic and outage data more accessible to a wider audience.
Cloudflare Launches Private Origins for All Customers
Cloudflare services for private origins are now generally available, bringing WAF, bot management, caching, and Workers to private applications.
