Skip to content
NERDS.xyz
  • Home
  • AI
  • Linux
  • Open Source
  • Hardware
  • Security
  • All Articles
  • RSS

web crawling

Cloudflare is helping OpenAI crawl the web it once helped publishers protect

July 10, 2026 by Brian Fagioli
Cloudflare Wide

Cloudflare says it wants to protect publishers from aggressive AI crawlers. Now it is helping OpenAI discover and index web content more efficiently.

Categories Cloudflare, OpenAI Tags AI crawlers, AI search, artificial intelligence, ChatGPT, Cloudflare, online publishers, OpenAI, web crawling, website indexing Leave a comment
Add us to
Google
Add

NERDS.xyz
Newsletter

Join
⚡
Support
NERDS
Enjoying NERDS.xyz?
Send a Lightning tip.

Featured Articles

  • Lenovo Tab Plus Gen 2 packs nine JBL speakers and AI features into a $400 Android tablet
  • Sovereign AI sounds independent until you notice everything still runs on NVIDIA
  • Q&A: Ditto says social media became sterile and the internet should be fun again
  • Tim Cook is stepping down and Apple desperately needs this reset
  • Joanna Stern’s exit from The Wall Street Journal is a shock, and a sign of the times
© 2026
NERDS.xyz
All rights reserved
About Contact Privacy Policy Editorial Policy
© 2026 NERDS.xyz • Built with GeneratePress