URL discovery

Sitemaps and Discovery Surfaces

A sitemap is a machine-readable inventory of canonical public URLs. It supports discovery and refresh workflows, but it does not guarantee crawling, indexing, ranking, or AI citation.

Build a trustworthy inventory

Include canonical, indexable URLs that return successful responses. Split large inventories into sitemap indexes and keep each referenced file available without authentication.

Use absolute HTTPS URLs.
Exclude redirects, errors, and private routes.
Keep canonical tags aligned with sitemap locations.
Update lastmod only when content materially changes.

Connect the surfaces

Reference the sitemap from robots.txt and submit it to the webmaster tools you use. Link important pages internally so discovery does not depend on the sitemap alone.

Declare the sitemap in robots.txt.
Submit it to Google Search Console and Bing Webmaster Tools.
Monitor fetch and parsing errors.
Keep important content reachable through ordinary navigation.

Treat inclusion as inventory—not approval

A sitemap tells a crawler which URLs you consider important; it does not override access rules, canonical choices, noindex directives, or quality decisions.

Remove URLs that permanently redirect.
Do not list authenticated account pages.
Use separate sitemap files when operational ownership differs.
Compare submitted, discovered, crawled, and indexed counts over time.

A valid one-page sitemap

Use the standard sitemap namespace, absolute canonical URLs, and truthful modification dates. Then declare the public sitemap location in robots.txt.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/guide</loc>
    <lastmod>2026-08-18</lastmod>
  </url>
</urlset>

# In robots.txt
Sitemap: https://example.com/sitemap.xml

Verify the change

Run the AIWebSignals scanner against the exact public page, review the observation confidence, and compare the saved result after your update. A technical improvement should be visible in the evidence—not assumed from a deployment.

Scan and verify

Authoritative references

Use current primary documentation when a crawler token, platform policy, or web standard changes.

Last reviewed August 18, 2026.