Anthropic's crawl-to-refer hits 2,978:1 — 23× better than a year ago, still the greediest bot on the web
TL;DR
Cloudflare June data: Anthropic crawls 2,978 pages per referral vs Google's 5 and OpenAI's 530. ClaudeBot's share jumped +66% to become the #2 AI crawler.
Cloudflare's June crawler traffic report puts Anthropic at 2,978 pages crawled per single referred visitor — dead last on the crawl-to-refer leaderboard. Google's ratio is 5:1, OpenAI's GPTBot is 530:1. Anthropic is ~596× less generous than Google and 5.6× worse than OpenAI.
2,978:1 is already a massive improvement. Last month the number was 9,478:1; a year ago Cloudflare's first read pinned Anthropic at 71,000:1. That is a 23× improvement in twelve months — but the driver is Claude's built-in web search sending traffic back out, not ClaudeBot crawling less. In fact the opposite happened: ClaudeBot's share of global AI crawl traffic jumped from 12.1% in May to 20.0% in June, +66%, overtaking Meta-ExternalAgent, GPTBot, and Bytespider to sit second only to Googlebot (24.9%).
Add Claude-SearchBot in and Anthropic's two crawlers together represent about 23% of all identified AI crawler traffic. ByteDance's Bytespider went the other way — up to 10.1% in May, down to 7.3% in June and out of the top five. Meta-ExternalAgent slid three straight months to 10.2%; GPTBot held at 9.6%.
The "crawl more, ratio improves" paradox comes from the product side. Claude web and app started referring more third-party links, dragging the denominator down faster than ClaudeBot expanded the numerator. Cloudflare flags one caveat: Claude's native app does not send a Referer header, so the real ratio may still get revised.
Anthropic also identified its three crawlers publicly in June: ClaudeBot for training, Claude-SearchBot for user-query retrieval, and Claude-User for on-demand page summaries. Its spokesperson told PPC.land the split is "to give webmasters more granular control." On paper. In practice the only opt-out that shifts the ROI math is a site-wide block — 89.4% of AI crawler traffic is for training, and referrals for that side of the ledger are the 1 in the 2,978.
Win the bet, and Anthropic pushes the ratio into the 500:1 range within a quarter, catches OpenAI, and takes the pressure off the next copyright-suit wave. Lose it, and 2,978:1 becomes exhibit A in the next New York Times-tier complaint.
via Cloudflare / WebSearchAPI Monthly / PPC.land / SEOmator
2,978:1 is already a massive improvement. Last month the number was 9,478:1; a year ago Cloudflare's first read pinned Anthropic at 71,000:1. That is a 23× improvement in twelve months — but the driver is Claude's built-in web search sending traffic back out, not ClaudeBot crawling less. In fact the opposite happened: ClaudeBot's share of global AI crawl traffic jumped from 12.1% in May to 20.0% in June, +66%, overtaking Meta-ExternalAgent, GPTBot, and Bytespider to sit second only to Googlebot (24.9%).
Add Claude-SearchBot in and Anthropic's two crawlers together represent about 23% of all identified AI crawler traffic. ByteDance's Bytespider went the other way — up to 10.1% in May, down to 7.3% in June and out of the top five. Meta-ExternalAgent slid three straight months to 10.2%; GPTBot held at 9.6%.
The "crawl more, ratio improves" paradox comes from the product side. Claude web and app started referring more third-party links, dragging the denominator down faster than ClaudeBot expanded the numerator. Cloudflare flags one caveat: Claude's native app does not send a Referer header, so the real ratio may still get revised.
Anthropic also identified its three crawlers publicly in June: ClaudeBot for training, Claude-SearchBot for user-query retrieval, and Claude-User for on-demand page summaries. Its spokesperson told PPC.land the split is "to give webmasters more granular control." On paper. In practice the only opt-out that shifts the ROI math is a site-wide block — 89.4% of AI crawler traffic is for training, and referrals for that side of the ledger are the 1 in the 2,978.
Win the bet, and Anthropic pushes the ratio into the 500:1 range within a quarter, catches OpenAI, and takes the pressure off the next copyright-suit wave. Lose it, and 2,978:1 becomes exhibit A in the next New York Times-tier complaint.
via Cloudflare / WebSearchAPI Monthly / PPC.land / SEOmator
