Crawler access

AI Crawler Access Guide

AI systems cannot discover public content they cannot retrieve. Access depends on several independent layers, so a single robots.txt line is never the whole story.

Separate the access layers

Start with DNS and HTTP availability, then check security challenges, status codes, robots.txt rules, and page-level indexing directives. A crawler can be allowed in robots.txt and still receive an interstitial or error instead of the origin page.

Confirm the hostname resolves publicly.
Return a normal origin response without a challenge for intended crawlers.
Review named and wildcard robots.txt groups.
Check X-Robots-Tag and page-level robots metadata.

Use explicit, testable policy

Decide separately how search, user-requested retrieval, and training crawlers should be treated. Publish the exact policy you intend and retest from outside your authenticated browser session.

Document the purpose of each named crawler.
Keep sitemap declarations reachable.
Avoid contradictory wildcard and named groups.
Re-scan after CDN, WAF, or robots changes.

Diagnose in layers

A failed fetch is a symptom, not a diagnosis. Record the response at each layer so you can distinguish a missing page from a blocked crawler or an edge-security challenge.

Compare an ordinary request with the intended crawler user agent.
Inspect redirects until the final URL and status are known.
Confirm the returned body is the page—not a challenge template.
Repeat the check from a clean, signed-out session.

A compact external check

Run the same public checks from outside your administrator session. Replace example.com with the exact hostname you intend machines to reach.

curl -I https://example.com/
curl -s https://example.com/robots.txt
curl -I https://example.com/sitemap.xml

Verify the change

Run the AIWebSignals scanner against the exact public page, review the observation confidence, and compare the saved result after your update. A technical improvement should be visible in the evidence—not assumed from a deployment.

Scan and verify

Authoritative references

Use current primary documentation when a crawler token, platform policy, or web standard changes.

Last reviewed August 18, 2026.