From Blocking AI Crawlers to Guiding Them: Cloudflare Opens a Lane for OpenAI
TL;DR
Cloudflare feeds real-time network signals to OpenAI to cut ChatGPT's wasted crawls on 20% of the web.
Cloudflare and OpenAI announced on July 8 the first network-layer research pilot for AI search, with Cloudflare feeding real-time signals from its global network — content freshness, traffic quality, and actual page changes — directly to OpenAI's ChatGPT search indexer in exchange for fewer wasted crawls. Over 20% of web traffic sits behind Cloudflare, and this is the first time a CDN has turned "this page hasn't changed" into a signal delivered straight to an answer engine.
The contrast is in the timeline. A year ago Cloudflare rolled out Content Independence Day, blocking AI crawlers by default for every new customer and launching pay-per-crawl so sites could charge by the hit. Matthew Prince publicly called out Perplexity for spoofing user agents to bypass the block. In the new announcement he said, "By sharing our sophisticated network signals, we can find a better way to make AI search more efficient." OpenAI VP of Research Nick Ryder said, "Up-to-date information is important for delivering accurate answers to people using ChatGPT."
The technical numbers are sharper. Per the Cloudflare blog, more than 50% of good-bot crawl traffic today re-fetches pages that have not changed, and the share only grows as crawl volumes climb. The pilot hands crawlers a "nothing to see here" signal so OpenAI's bot can skip the request — site owners save bandwidth, OpenAI saves compute. Cloudflare stressed that no page content is shared, and nothing from the pilot is used to train foundation models.
Opt-in details were not fully disclosed. Cloudflare pointed to Ceramic.ai and You.com as existing pay-per-crawl participants; Ceramic founder Anna Patterson said, "To scale the future of AI search, we need a partner with massive reach and shared commitment to transparency and fair compensation." ChatGPT search currently mixes Bing with OpenAI's own crawler; this pilot is the first time OpenAI plugs into a third-party network-layer signal.
A year after slamming the gate on AI crawlers, Cloudflare upgraded itself into the data broker for AI search. The toll on crawlers still exists — the road just forks two ways now: full block, or the express lane Cloudflare opened for OpenAI.
via Cloudflare Press / Cloudflare Blog / Businesswire
The contrast is in the timeline. A year ago Cloudflare rolled out Content Independence Day, blocking AI crawlers by default for every new customer and launching pay-per-crawl so sites could charge by the hit. Matthew Prince publicly called out Perplexity for spoofing user agents to bypass the block. In the new announcement he said, "By sharing our sophisticated network signals, we can find a better way to make AI search more efficient." OpenAI VP of Research Nick Ryder said, "Up-to-date information is important for delivering accurate answers to people using ChatGPT."
The technical numbers are sharper. Per the Cloudflare blog, more than 50% of good-bot crawl traffic today re-fetches pages that have not changed, and the share only grows as crawl volumes climb. The pilot hands crawlers a "nothing to see here" signal so OpenAI's bot can skip the request — site owners save bandwidth, OpenAI saves compute. Cloudflare stressed that no page content is shared, and nothing from the pilot is used to train foundation models.
Opt-in details were not fully disclosed. Cloudflare pointed to Ceramic.ai and You.com as existing pay-per-crawl participants; Ceramic founder Anna Patterson said, "To scale the future of AI search, we need a partner with massive reach and shared commitment to transparency and fair compensation." ChatGPT search currently mixes Bing with OpenAI's own crawler; this pilot is the first time OpenAI plugs into a third-party network-layer signal.
A year after slamming the gate on AI crawlers, Cloudflare upgraded itself into the data broker for AI search. The toll on crawlers still exists — the road just forks two ways now: full block, or the express lane Cloudflare opened for OpenAI.
via Cloudflare Press / Cloudflare Blog / Businesswire
