Cloudflare has unveiled the Content Signals Policy, a free addition to its managed robots.txt service that aims to give website owners and publishers more control over how their content is accessed and reused by AI companies.
The idea is pretty simple: robots.txt already lets site operators specify which crawlers can enter and where. Cloudflare’s new policy adds a layer that signals how the data may be used once accessed, with plain-language terms for search, AI input, and AI training. “Yes” means allowed, “no” means not allowed, and no signal means no preference.
Matthew Prince, Cloudflare’s co-founder and CEO, said: “The Internet cannot wait for a solution, while in the meantime, creators’ original content is used for profit by other companies. To ensure the web remains open and thriving, we’re giving website owners a better way to express how companies are allowed to use their content.”
Cloudflare says more than 3.8 million domains already use its robots.txt tools to signal they don’t want their content used for AI training. Now, the Content Signals Policy makes those preferences clearer and potentially enforceable.
Danielle Coffey of the News/Media Alliance called it “an important step towards empowering publishers of all sizes to reclaim control over their own content.” Meanwhile, Stack Overflow CEO Prashanth Chandrasekar pointed to his company’s large corpus of developer Q&A and said he applauds Cloudflare “for playing a central role to empower and protect content creators in this new AI era.”
Starting today, the new policy language will be automatically added to Cloudflare-managed robots.txt files for customers who opt in. For others, Cloudflare is publishing tools to help them declare content preferences manually. Whether AI companies will respect these signals remains uncertain, but Cloudflare is betting that legal clarity and public pressure will push them to comply.
