Cloudflare has officially picked a side in the growing fight between publishers and AI companies, and thankfully, it is probably the side most website owners were hoping for.
The company announced a series of new tools and policies designed to give publishers more control over how AI companies access and use their content. The idea is simple enough: if you want AI systems using your content, great. If you do not, that should be your choice too.
That sounds obvious, but the reality has been anything but.
For years, websites have largely been forced to accept an all-or-nothing arrangement. If you wanted to appear in search results or remain discoverable online, there was a good chance your content would also end up feeding AI systems in one way or another.
Cloudflare wants to change that. You see, starting September 15, new Cloudflare customers and new websites added by existing customers will automatically allow traditional search crawling while blocking AI training and AI agent access on pages that contain advertising. Existing free customers that do not manually change their settings before that date will receive the same protections automatically.
The company is also taking aim at what it calls mixed-use crawlers. These are bots that perform multiple functions at once, such as indexing content for search while also collecting data for AI training or agent use. According to Cloudflare, that forces publishers into an unfair choice between remaining discoverable and giving away content for free.
Under the new rules, mixed-use crawlers that do not clearly separate those functions could find themselves blocked from ad-supported pages entirely.
Cloudflare is introducing new analytics too. Publishers will be able to see which AI companies are accessing their content, how often those companies are using it, and whether any actual human traffic is coming back in return.
That last part matters more than some AI companies would probably like to admit.
If an AI system is summarizing your reporting without sending readers back to your site, that is not exactly a great trade for publishers that depend on advertising, subscriptions, affiliate revenue, or all three.
Cloudflare also believes we are entering the era of Answer Engine Optimization, or AEO.
The idea is that publishers will eventually spend less time worrying about where they rank in search results and more time worrying about whether AI systems cite them when answering questions. It feels a little strange to say out loud, but considering how quickly people are replacing searches with chatbot prompts, Cloudflare may be onto something.
Perhaps the most interesting announcement, however, involves money.
Last year, Cloudflare introduced Pay Per Crawl, which allowed publishers to charge AI companies for crawling their content. Now it wants to evolve that into Pay Per Use, where publishers get paid when their content actually contributes to an AI-generated answer instead of simply being downloaded by a bot.
Several companies are already testing the idea, including AI search startups that want publishers compensated whenever their content appears in search results or is accessed by AI agents.
As someone who runs an independent publication, I find that approach a lot more appealing than the current system where only giant media companies have the leverage to negotiate licensing deals with AI firms.
AI companies want as much content as possible. Publishers want compensation. Users want instant answers. Cloudflare apparently believes all three can coexist. For the sake of the open web, I hope it is right.
Support independent tech journalism
NERDS.xyz is independently owned and operated. If you enjoy my coverage of Linux, AI, hardware, cybersecurity, and tech culture, consider supporting the site on Ko-fi.
Support NERDS.xyz