Three purposes, three operational questions
OpenAI's crawler documentation and publisher guidance distinguish search discovery from model-development crawling. OAI-SearchBot is the relevant token for sites that want public pages eligible to be discovered for ChatGPT search experiences. GPTBot is separately documented for model-development use. OpenAI also documents OAI-AdsBot for validating advertiser landing pages. That separation matters because the business value and risk profile of each request class is different.
A publisher that treats every OpenAI user agent as one category loses policy precision. Search visibility may create human referral opportunities. Training-oriented crawling may be evaluated through a licensing or opt-out lens. Advertising validation matters only if the site is participating in an advertising workflow. The right robots policy starts with the business objective, not the provider name alone.
robots.txt is only the first gate
An Allow directive says a cooperative crawler is permitted to request a path. It does not guarantee that the CDN will serve the path, that bot management will avoid a challenge, that application middleware will permit the request, or that the returned page is useful. OpenAI's own troubleshooting guidance for advertiser crawling explicitly calls out robots.txt, web protection, bot mitigation, JavaScript challenges, CAPTCHAs, authentication, and rate limits as separate failure layers.
That means a site can have a perfectly permissive robots.txt file and still be effectively invisible. Testing should compare the final HTTP response for ordinary traffic and the intended crawler identity, then inspect whether the returned content is the real page rather than an interstitial. Edge access is part of AI discoverability.
A defensible split policy
If the goal is discovery without granting the same access for model-development crawling, a site can explicitly allow OAI-SearchBot while setting a different rule for GPTBot. The policy should be deployed at every hostname that matters and paired with a reachable sitemap, canonical URLs, useful internal links, and indexable page content. That still does not guarantee citation or ranking, but it removes an avoidable access contradiction.
Operators should also keep measurement separate from configuration. After a policy change, inspect actual first-party requests and human referral traffic. If OAI-SearchBot appears but referral sessions do not change, the evidence says access changed—not that commercial value changed. That is the correct point to keep observing rather than over-claiming.
Atlas adds two more evidence layers: indexing controls and interaction semantics
OpenAI's current publisher and developer FAQ documents a narrower outcome when OAI-SearchBot cannot crawl a page: ChatGPT Atlas may still surface the page's link and title when OpenAI obtains the URL through a third-party search provider or by crawling other pages and has signals that the page is relevant. OpenAI points publishers who want to suppress that limited exposure to a noindex meta tag, while also noting that its crawler must be able to access the page to read the tag. That makes robots permission and indexing or surfacing directives separate evidence layers.
The same FAQ says ChatGPT Agent in Atlas uses ARIA labels, roles, and states to interpret page structure and interactive elements such as buttons, menus, and forms. Those semantics answer a different question from crawler access. A page can be publicly reachable and crawlable while still being difficult for an agent to operate reliably if its interactive controls do not expose useful accessible names, roles, or states. AIWebSignals therefore treats agent interaction readiness as a distinct product opportunity rather than inferring it from robots.txt, sitemap availability, or ordinary HTTP reachability.
OpenAI also states that referral URLs from ChatGPT include utm_source=chatgpt.com. That parameter can support observable human-referral attribution in analytics, but it does not prove which crawler or agent caused the referral, that a particular page was cited, or that the visit produced an economic outcome.
What AIWebSignals would watch
For OpenAI traffic, useful metrics include request count by token, status-code distribution, repeat paths, discovery-resource requests such as robots.txt and sitemaps, and human sessions attributed to ChatGPT referrals. A site can then compare policy changes against both machine traffic and downstream human behavior.
The decision framework becomes simple: allow what creates measurable value or strategic reach, restrict what conflicts with the site's policy, and avoid assuming that a crawler request itself proves an economic outcome.
Original evidence: purpose-explicit policy is uncommon in a 10-domain cohort
On September 8, 2026, AIWebSignals inspected a fixed technical-observability cohort of ten public technology and marketing robots.txt files. All ten permitted OAI-SearchBot, OAI-AdsBot, and GPTBot at the homepage path under named or inherited wildcard rules, but none explicitly named OAI-AdsBot and none explicitly separated all three purposes. That is a policy-expression observation, not evidence of crawler traffic or edge reachability.
Read the reproducible 10-domain OpenAI crawler policy study.
Related technical evidence
OpenAI three-way crawler policy study · OAI-SearchBot, GPTBot and OAI-AdsBot in the AI Bot Index · robots.txt for AI crawlers