Measuring the Machine Web Without Pretending We Can Read an Agent's Mind
A practical evidence model for separating observed machine requests, inferred journeys, referrals, policy effects, and revenue.
AIWEBSIGNALS RESEARCH
Research for the machine-readable web.
We study what website operators can actually observe as AI systems crawl, retrieve, refer, interact with, and eventually pay for web resources. Every brief separates measured evidence from inference and links back to the provider or protocol documentation behind the claim.
Machine Web snapshot
This panel loads the current Activity Radar sweep so the publication has a live observation layer alongside long-form analysis.
Research briefs
Provider policies change, protocols evolve, and crawler behavior is easy to overstate. These pieces are written to remain useful by making the evidence boundary explicit.
OpenAI Crawler Controls: Search Visibility and Model-Development Access Are Not the Same Decision
How to reason about OAI-SearchBot, GPTBot, OAI-AdsBot, robots.txt, and edge access without collapsing different purposes into one rule.
ClaudeBot, Claude-SearchBot, and Claude-User: Anthropic's Three-Way Access Model
A site-owner guide to separating model-development crawling, search indexing, and user-requested retrieval in Anthropic traffic policy.
PerplexityBot and robots.txt: What a Publisher Can Control, and What the Signal Does Not Prove
A current evidence-based guide to PerplexityBot access, indexing behavior, blocked-page limitations, and measurement.
Google-Extended Is Not a Crawler: Why That Distinction Changes AI Access Policy
Google-Extended is a robots.txt product token, not a separate HTTP user agent. Here's what it controls and what it does not change in Google Search.
The AI Crawler Policy Matrix: Search, Training, User Retrieval, Ads, and Paid Access
A purpose-first framework for deciding which machine traffic to allow, observe, block, rate-limit, or eventually charge.
How to Detect AI Bot Traffic Without Trusting the User-Agent String Too Much
An evidence hierarchy for AI bot identification using declared tokens, provider documentation, verified network signals, request behavior, and uncertainty labels.
AI Referral Traffic Is Not AI Crawler Traffic: The Funnel Most Analytics Dashboards Blur Together
How to measure machine requests, AI-origin human visits, conversions, and revenue as separate stages of one commercial funnel.
x402 and Paid Machine Access: What HTTP-Native Payments Actually Prove
A practical guide to 402 payment requirements, verification, settlement, replay safety, and evidence-based machine-access revenue.
An Agent-Ready Website Starts With Accessibility, Not an 'AI Optimization' Trick
Why semantic HTML, ARIA, stable controls, crawlable content, and predictable forms help both people and AI agents interact with a site.
Generative AI Search Readiness: What Google Actually Recommends in 2026
A practical reading of Google's current guidance for AI Overviews and AI Mode, including indexing, snippets, unique content, and measurement.
OpenAI crawler policy cohort: search, ads, and training
Ten reproducible robots.txt observations show how often OAI-SearchBot, OAI-AdsBot, and GPTBot are explicitly separated versus governed by wildcard policy.
Anthropic crawler policy cohort: model development, search, and user retrieval
Ten reproducible robots.txt observations show how often ClaudeBot, Claude-SearchBot, and Claude-User are explicitly named versus governed by wildcard policy.
Google-Extended policy cohort: product token, not crawler identity
Ten reproducible robots.txt observations measure explicit versus inherited Google-Extended policy while preserving Google's no-separate-HTTP-user-agent distinction.
Web Bot Auth at the edge: documented support across connector providers
Four of five managed edge providers in the AIWebSignals connector cohort explicitly document cryptographic Web Bot Auth support; one remains unknown from bounded public documentation review.
Verified AI agent evidence: does the proof survive your log pipeline?
Two provider paths can carry verification in request logs, one needs runtime instrumentation, and two require separate security streams for the strongest documented identity evidence.
AI bot purpose controls at the edge
Two providers now publish explicit three-way training/search/fetcher-agent taxonomies; the remaining cohort uses different purpose-aware control shapes.
Foundational guides
The research library is complemented by seven implementation guides covering crawler access, robots.txt, llms.txt, structured data, sitemaps, and metadata.
FOUNDATIONAL READING
The Internet has two audiences: humans and machines.
Before crawler rules, structured data, agent journeys, or machine payments, understand the basic difference between the web people experience and the networked resources machines actually request.