robots.txt

robots.txt for AI Crawlers

robots.txt communicates crawl preferences to cooperative crawlers. It is public policy text, not authentication, a firewall, or proof that a crawler visited your site.

How matching works

A crawler selects the most specific matching user-agent group, then applies the most specific matching path rule. When Allow and Disallow rules are equally specific, Allow generally wins for major standards-following crawlers.

Use exact published user-agent tokens.
Keep one policy purpose per named group.
Use a wildcard group as a deliberate fallback.
Test the homepage and important content paths separately.

What robots.txt cannot prove

An allowed rule does not prove that a crawler can pass a security challenge, render client-only content, discover an unlinked page, or choose to use the content. Treat the file as one observable access signal.

Do not publish secrets in robots.txt.
Do not use robots.txt as an authorization system.
Check the returned content type and status.
Reference a valid XML sitemap when appropriate.

Common policy mistakes

Small formatting and scope errors can reverse the intended outcome. Review the deployed file, not only the source template, and confirm every hostname has the policy you expect.

Do not invent a crawler token; copy it from current provider documentation.
Do not assume a rule on one subdomain controls another.
Do not block CSS or scripts needed to understand public content without a reason.
Do not treat Allow as consent to train, rank, or cite.

Separate search visibility from training access

This illustrative policy allows OpenAI's search crawler while declining its training crawler. Provider names and purposes can change, so verify them against current documentation before publishing.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Verify the change

Run the AIWebSignals scanner against the exact public page, review the observation confidence, and compare the saved result after your update. A technical improvement should be visible in the evidence—not assumed from a deployment.

Scan and verify

Authoritative references

Use current primary documentation when a crawler token, platform policy, or web standard changes.

Last reviewed August 18, 2026.