Anthropic Crawler Policy Cohort: ClaudeBot, Claude-SearchBot, and Claude-User Across 10 Public Technology Sites
Anthropic's current crawler guidance separates model-development collection, search-oriented indexing, and user-requested retrieval into ClaudeBot, Claude-SearchBot, and Claude-User. We measured whether the same fixed ten-site technology cohort used in our OpenAI study expresses those three purposes explicitly or lets them inherit broader wildcard policy at the homepage path.
Aggregate result
- 10/10 cohort robots.txt files were reproducibly retrieved.
- 10/10 permitted ClaudeBot, Claude-SearchBot, and Claude-User at
/under the observed named or wildcard rules. - 1/10 explicitly named ClaudeBot.
- 1/10 explicitly named Claude-SearchBot.
- 1/10 explicitly named Claude-User.
- 1/10 explicitly named all three current Anthropic crawler identities together.
- 1/10 named other Anthropic-related crawler tokens that are not part of Anthropic's April 7, 2026 three-bot table.
The bounded finding is that homepage permission was broadly permissive while exact purpose-specific expression of Anthropic's current three identities was uncommon in this cohort. An absent named crawler stanza is not a block when wildcard rules permit the path.
Why Anthropic's three-way distinction matters
Anthropic's April 7, 2026 guidance describes ClaudeBot as collecting public web content that could contribute to model training, Claude-SearchBot as navigating the web to improve search result quality, and Claude-User as retrieving web content when a person asks Claude to access it. Anthropic says all three honor robots.txt and explicitly lets site owners make different access choices for those different purposes.
That makes a single provider-level "allow Anthropic" or "block Anthropic" label too coarse for operators who want search visibility or user-directed retrieval while making a different decision about model-development crawling. A wildcard policy can still be valid; it simply expresses less purpose-specific intent.
Reproducible observations
Observed at . Scope: homepage path /. "Named group" means the exact current Anthropic crawler token appeared in a matching user-agent group. "Wildcard Allow" means the crawler inherited an explicit User-agent: * allowance. "Wildcard/default" means no matching root-path disallow was observed, so the homepage remained permitted under ordinary robots matching semantics.
| Domain | ClaudeBot | Claude-SearchBot | Claude-User | Other Anthropic-related tokens observed |
|---|---|---|---|---|
| cloudflare.com robots.txt | Allowed · wildcard Allow | Allowed · wildcard Allow | Allowed · wildcard Allow | Anthropic-AI, Claude-Web |
| vercel.com robots.txt | Allowed · wildcard/default | Allowed · wildcard/default | Allowed · wildcard/default | — |
| wordpress.com robots.txt | Allowed · wildcard/default | Allowed · wildcard/default | Allowed · wildcard/default | — |
| github.com robots.txt | Allowed · wildcard/default | Allowed · wildcard/default | Allowed · wildcard/default | — |
| stripe.com robots.txt | Allowed · wildcard/default | Allowed · wildcard/default | Allowed · wildcard/default | — |
| shopify.com robots.txt | Allowed · wildcard/default | Allowed · wildcard/default | Allowed · wildcard/default | — |
| hubspot.com robots.txt | Allowed · wildcard/default | Allowed · wildcard/default | Allowed · wildcard/default | — |
| zapier.com robots.txt | Allowed · named group | Allowed · named group | Allowed · named group | — |
| notion.so robots.txt | Allowed · wildcard Allow | Allowed · wildcard Allow | Allowed · wildcard Allow | — |
| wix.com robots.txt | Allowed · wildcard/default | Allowed · wildcard/default | Allowed · wildcard/default | — |
One configuration detail worth noticing
Cloudflare's public robots.txt explicitly names Anthropic-AI and Claude-Web, while Anthropic's current April 7, 2026 guidance presents ClaudeBot, Claude-SearchBot, and Claude-User as its three robots. In this study we therefore treat Cloudflare's current-three policy as inherited from its wildcard Allow: /, and record the other tokens separately rather than assuming they are equivalent identities.
Zapier is the only site in this cohort that explicitly names all three current Anthropic identities together and allows them at the homepage. This is a neutral configuration observation, not a quality ranking.
Cohort methodology
Technical observability cohort of ten public technology/marketing platforms selected before Anthropic-policy inspection. It reuses the fixed cohort from the preceding OpenAI policy study so provider comparisons are not reselected after seeing the rules. Inclusion required ordinary public retrieval of /robots.txt; the cohort is descriptive and not intended to estimate prevalence across the web.
The cohort consists of Cloudflare, Vercel, WordPress.com, GitHub, Stripe, Shopify, HubSpot, Zapier, Notion, and Wix. Reusing the same cohort as the preceding OpenAI study reduces selection drift between provider analyses.
This is a technical observability cohort, not a statistically representative sample of the web. The counts describe only these ten public robots files at the timestamp above.
Evidence boundary
A named_group rule is purpose-explicit for the exact current Anthropic token. wildcard_allow and wildcard_default are inherited policy. Alternate Anthropic-related tokens are recorded separately and are not treated as equivalent to Anthropic's April 7, 2026 three-bot table. robots.txt permission does not prove crawler traffic, successful edge access, indexing, retrieval, training use, citation, referral traffic, or commercial outcome.
This study did not impersonate ClaudeBot, Claude-SearchBot, or Claude-User, did not bypass blocks, and did not access authenticated surfaces. It observed public robots.txt configuration only. Edge/WAF treatment remains a separate evidence layer, and actual Anthropic request traffic requires first-party logs or provider-specific evidence.
Primary references
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- RFC 9309: Robots Exclusion Protocol
Last reviewed September 8, 2026.
Related AIWebSignals evidence
Related technical evidence
Anthropic crawler controls explained · Anthropic crawler identities in the AI Bot Index · AI website readiness scanner