Crawler access
Inspect robots.txt and known AI crawler policies to see which bots are allowed, blocked, or undefined.
AIWEBSIGNALS · OBSERVE THE MACHINE WEB
AI Web Signals: see what AI systems can actually observe.
AIWebSignals combines public technical readiness with privacy-minimized first-party AI traffic evidence. Start with a free scan. Connect your own traffic when you need operational visibility.
Public signals first.First-party evidence when connected.No provider account custody.
USE CASES · START WITH THE QUESTION
What do you need to know about machine traffic?
Choose the job you are trying to do. Each path stays explicit about what is observed, inferred, connected, or still gated.
TECHNICAL READINESS
Can AI crawlers reach and understand my site?
Check crawler policy, discovery files, structured data, metadata, and the public surfaces machines can actually observe.
/ Run readiness scanFIRST-PARTY TRAFFICWhich AI systems are actually requesting my pages?
Connect privacy-minimized request evidence from infrastructure you already operate without handing AIWebSignals your provider account credentials.
/ Connect AI trafficBEHAVIORWhere do machine visitors go next?
Turn normalized automated requests into agent journeys so repeated paths, loops, exits, and transitions are easier to inspect.
/ Explore Agent JourneysIMPACTWhich machine activity matters to my site?
Use connected evidence to inspect AI-related request activity alongside measured site outcomes without pretending that a request is a conversion.
/ Explore AI ImpactPOLICY & ACCESSWhat should machine traffic be allowed to do?
Observe and simulate policy decisions with explicit evidence and boundaries. Enforcement remains a separately gated production control.
/ Review Agent PolicyMACHINE ACCESSWhere could machine access become a product?
Define pricing and machine-access configuration without counting ordinary crawler requests as revenue or giving AIWebSignals custody of customer funds.
/ Explore AI RevenueExample report
See the result before you scan.
This illustrative preview shows how findings are grouped and prioritized. Your result reflects only what the scanner can observe from your site.
Example score72Good foundation
Crawler accessReady
Discovery surfacesNeeds work
Structured dataPartial
Start hereAdd a valid XML sitemap and reference it from robots.txt.
Get your own prioritized report →What gets checked
Four groups of observable signals.
Discovery surfaces
Check sitemaps, llms.txt, canonical signals, and machine-readable paths that help systems find content.
Structured data
Review JSON-LD and microdata so important entities and page meaning are exposed in structured form.
Metadata quality
Inspect titles, descriptions, canonical URLs, indexing directives, social metadata, and language signals.
Learn and verify
Practical guides for every signal the scanner checks.
Use the education library to understand crawler rules, discovery files, structured data, sitemaps, and metadata before you rescan.
AI crawler rules
Write robots.txt policies that are explicit and testable.
llms.txt
Understand the emerging discovery surface and its limits.
Structured data
Expose entities and page meaning in machine-readable form.
Sitemaps
Make important public URLs easier to discover and refresh.
Public discovery
Seek public agent activity across the web.
The AI Activity Radar finds official MCP registry entries and standardized A2A Agent Cards while keeping public evidence separate from measured traffic.
Open AI Activity Radar →Designed for trustworthy results
Clear boundaries are part of the product.
Analytics events use categorical funnel labels and never include scanned domains, URLs, report IDs, configuration IDs, or tokens.
The same observable inputs produce the same findings. Scores are withheld when access challenges make a fair result impossible.
Reports put the highest-value technical fixes first and preserve history for exact-target comparisons.
See what your website exposes today.
Run the scan, review the strongest next action, and use saved history to verify improvements.
Start your free scanAI TRAFFIC OPERATIONS
Go from public signals to first-party AI traffic intelligence.
Use AIWebSignals free for public research and readiness checks. Paid plans add connected traffic evidence, AI Impact, access policy, and machine-access monetization controls.
START WITH THE MODEL
Humans and machines do not experience the Internet the same way.
See how information becomes data, documents, network requests, identity, permission, and economic events—and why that creates a new visibility problem for website owners.
AIWEBSIGNALS RESEARCH
Understand the machine web with evidence, not folklore.
Read original research on AI crawler policy, machine traffic, generative search visibility, agent readiness, AI referrals, and paid machine access. Provider-specific claims link back to current primary documentation, and inference is labeled separately from observed evidence.
PUBLIC SCAN → FIRST-PARTY EVIDENCE
Readiness shows what machines can reach. First-party traffic shows what they actually requested.
The free scan inspects public technical signals. It can establish reachability and machine-readable discovery conditions, but it cannot turn a public request into evidence that a particular AI system actually visited your site.
Public HTTP responses, crawler policy surfaces, discovery files, structured data, and other machine-readable signals the scanner can retrieve directly.
Which AI systems requested your pages, how often they returned, which paths they followed, and whether that activity led to referrals or revenue.
Connect first-party request evidence from infrastructure you already operate. AIWebSignals normalizes that evidence so observed traffic can be separated from inference.
Connect first-party AI trafficSee the evidence methodology
The readiness scan remains free and requires no account. Connecting first-party traffic is a separate step.