Why the purpose split matters
ClaudeBot is described by Anthropic as collecting public web content that can contribute to model utility and safety. Claude-SearchBot is described as navigating the web to improve search result quality. Claude-User is used when a person asks Claude to retrieve web content. The three categories can produce similar-looking HTTP requests while representing materially different site-owner decisions.
For a publisher, that makes policy design more expressive. A site can choose to support search discovery and user-directed access while taking a different position on model-development collection. Conversely, a private or operational system may decide that none of those automated paths should be available. The important point is that the provider already exposes enough semantic separation to avoid guessing.
robots rules should match the business intent
Anthropic says its bots honor industry-standard robots.txt directives. It also documents support for Crawl-delay, which is not part of the formal robots standard but can be used to request a slower crawl rate. Site operators should still treat Crawl-delay as a cooperative preference rather than a hard capacity control; application rate limits and edge controls remain the enforcement layer for resource protection.
Each relevant subdomain should publish the intended policy. A permissive rule on the root domain does not automatically define the policy for a different hostname. The deployed response should be tested publicly, because a source-control file that never reaches production has no effect.
Do not confuse access with downstream use
An observed Claude-SearchBot request can show that Anthropic's search crawler reached a page. It does not prove that the page was selected for an answer, cited to a user, or responsible for a conversion. A Claude-User request can support the inference that a user-directed retrieval occurred, but it still does not reveal the person's prompt or the model's private reasoning.
This is where request-sequence analysis is helpful. If a user-directed fetch requests an article and then several related documents in a short sequence, a site can reconstruct a probable path without pretending to know the private objective. That level of analysis is useful for information architecture and access policy while staying inside the evidence boundary.
A practical measurement plan
Measure the three tokens separately. Compare which paths each requests, how often they return, their error rates, and whether changes in robots policy alter those patterns. Then keep human referral analytics in a separate view. The result is a policy loop grounded in observable behavior rather than assumptions about the model.
Over time, this can answer a concrete business question: which Anthropic access classes are creating enough discoverability, support value, or licensing opportunity to justify continued access?
Original evidence: exact three-way Anthropic policy is uncommon in a 10-domain cohort
On September 8, 2026, AIWebSignals inspected the same fixed ten-domain technology cohort used for its OpenAI policy study. All ten permitted ClaudeBot, Claude-SearchBot, and Claude-User at the homepage path under named or inherited wildcard rules, while only one explicitly named all three current Anthropic identities. That is a policy-expression observation, not evidence of Anthropic crawler traffic or edge reachability.
Read the reproducible 10-domain Anthropic crawler policy study.
Related technical evidence
Anthropic three-way crawler policy study · ClaudeBot, Claude-SearchBot and Claude-User in the AI Bot Index · robots.txt for AI crawlers