
blog.cloudflare.com
September 16, 2026
12 min read
47/100
Summary
Have it both ways: stay discoverable in search while disallowing AI training Without proper controls, website owners have long faced a difficult tradeoff: allow your content to be used for AI training, or risk losing discoverability in search. That tradeoff exists because some of the largest organizations on the Internet use mixed-use crawlers: a single crawler serving both search and AI training....