Cloudflare's new AI traffic options for customers
blog.cloudflare.com
July 25, 2026
14 min read
🔥🔥🔥🔥🔥
57/100
Summary
Website owners now have new AI traffic options that allow them to regain control over their content. The shift in the relationship between crawlers and website owners is a response to AI's impact on content distribution and referrals.
Key Takeaways
- Cloudflare introduced a one-click "Block AI Bots" option and a Pay-Per-Crawl marketplace to help website owners control their content and receive compensation for its use in AI training.
- The classification of bots will now focus on their behavior, categorizing them into three main use cases: Search, Agent, and Training, to provide website owners with better management options.
- The shift in AI technology has transformed search engines into full answer engines, prompting a need for website owners to balance content visibility with protection against unauthorized AI training.
- Website owners are encouraged to expect equitable compensation for their content when it is used for indexing or answering queries by search engines.
Coverage & discussions
Hacker News · blog.cloudflare.com
Community discussions
Community Sentiment
Positives
- Cloudflare's new policies could foster a more ethical scraping environment, balancing the needs of producers and consumers in the digital space.
- The move to block multi-purpose crawlers like Googlebot shows Cloudflare's commitment to transparency and protecting website owners' rights.
Concerns
- Cloudflare's approach feels like a double-edged sword; they’re caught between enabling AI development and protecting website traffic, which many see as hypocritical.
- The lack of stakeholder involvement in Cloudflare's decisions raises concerns about the future of internet traffic management and could lead to fragmentation.
