Starting September 15, network infrastructure provider Cloudflare will begin blocking all ‘mixed’ AI-based web crawlers by default on its pages.
Cloudflare’s new Attribution Business Insights dashboard helps website owners understand crawler behavior, appetite, and potential value. https://t.co/1F9HO3Fiv7
— Cloudflare (@Cloudflare) July 1, 2026
These crawlers use the same type of bots for search, model training, and AI agents. Cloudflare requires separating solutions into search, training, and agent categories.
Company representatives explained the stricter measures by noting that publishers and website owners want to maintain visibility in search and AI services but are unwilling to provide content for free.
“The internet risks evolving into a ‘zero-click’ ecosystem where chatbots gather original content for instant answers, bypassing the original sources entirely. We’ve already seen a significant shift from an SEO-focused space to an AEO world, and now to GEO,” the statement said.
The measure is also expected to reduce the load on web page infrastructure. According to Cloudflare, over 50% of AI crawler traffic is due to repeated page scanning, which rarely adds value.
The change will affect new provider clients, existing user sites, and all free-tier customers. Hosting owners can manually disable the blocking option.
Simultaneously, Cloudflare is expanding its commercial scheme for publishers. The new Pay Per Use system involves payment not only for access to material but also when content is valuable to an AI service.
Initially, the company is working with Ceramic.ai and You.com. Publishers receive payments if their materials appear in chatbot search results.
Previously, on November 18, 2025, Cloudflare experienced significant technical issues, causing disruptions to numerous websites and applications, including Web3 services.
