OpenAI Crawler Policy Cohort: Search, Ads, and Training Controls Across 10 Public Technology Sites

Aggregate result

• 10/10 cohort robots.txt files were reproducibly retrieved.
• 10/10 permitted OAI-SearchBot, OAI-AdsBot, and GPTBot at / under the observed named or wildcard rules.
• 1/10 explicitly named OAI-SearchBot.
• 0/10 explicitly named OAI-AdsBot.
• 2/10 explicitly named GPTBot.
• 0/10 explicitly separated all three OpenAI purposes in named crawler groups.

The defensible finding is not that these sites are "OpenAI-ready." It is that homepage permission was broadly permissive while purpose-specific policy expression was uncommon in this technical observability cohort. An absent named crawler stanza is not the same thing as a block because wildcard rules can govern the request.

Why the three-way distinction matters

OpenAI's current publisher guidance says OAI-SearchBot is the crawler site owners should avoid blocking when they want content eligible for ChatGPT search discovery. GPTBot is a separate model-development control. OpenAI's advertiser guidance says OAI-AdsBot is required for ChatGPT Ads landing-page validation and separately recommends OAI-SearchBot. Those are different operator decisions even when a site's robots.txt happens to give them the same inherited policy.

A purpose-explicit robots policy is useful when an operator wants to make different choices for search, advertising validation, and model-development access. A wildcard policy can still be valid; it simply expresses less purpose-specific intent.

Reproducible observations

Observed at 2026-09-08T10:04:25-07:00. Scope: homepage path /. "Named group" means the exact OpenAI crawler token appeared in a matching user-agent group. "Wildcard Allow" means the crawler inherited an explicit User-agent: * allowance. "Wildcard/default" means no matching root-path disallow was observed, so the homepage remained permitted under ordinary robots matching semantics.

Cohort methodology

Technical observability cohort of ten public technology/marketing platforms selected before OpenAI-policy inspection. Inclusion required only ordinary public retrieval of /robots.txt; the cohort is descriptive and not intended to estimate prevalence across the web.

The cohort consists of Cloudflare, Vercel, WordPress.com, GitHub, Stripe, Shopify, HubSpot, Zapier, Notion, and Wix. The sites were selected before inspecting their OpenAI-specific crawler rules. Inclusion was based on ordinary public robots.txt retrieval, not on whether the resulting policy was permissive or restrictive.

This is a technical observability cohort, not a statistically representative sample of the web. The counts describe only these ten public robots files at the timestamp above.

Evidence boundary

A named_group rule is purpose-explicit. wildcard_allow and wildcard_default are inherited policy. robots.txt permission does not prove crawler traffic, successful edge access, indexing, ad participation, citation, training use, or commercial outcome.

This study did not impersonate OAI-SearchBot, OAI-AdsBot, or GPTBot, did not bypass any block, and did not access authenticated surfaces. It observed public robots.txt configuration only. Edge/WAF treatment remains a separate evidence layer, and actual OpenAI request traffic requires first-party logs or provider-specific evidence.

Primary references

• OpenAI: Publishers and Developers FAQ
• OpenAI: Advertiser Guidance for Allowing OpenAI Web Crawlers
• RFC 9309: Robots Exclusion Protocol

Last reviewed September 8, 2026.

Related AIWebSignals evidence