1. Separate provider documentation from observed traffic
Provider documentation tells us what a published crawler token is intended to do and which controls the provider says it respects. First-party request evidence tells a site owner what actually reached the site. We use provider documentation to classify purpose, but we do not convert a published purpose into a claim about a specific private prompt or downstream model action.
When documentation and observed behavior appear to conflict, we report the conflict instead of automatically treating either side as conclusive. A robots.txt rule may be correct while a CDN cache, alternate hostname, stale deployment, spoofed user agent, or unrelated automated client creates an apparently contradictory request. The next step is to reproduce the observation, verify the public configuration, and look for corroborating evidence before naming a provider as non-compliant.
2. Keep identity confidence visible
User-agent strings are requester-controlled. A recognized token is evidence of a declared identity, not a cryptographic signature. Where CDN-verified bot metadata or other provider-supported verification exists, confidence can increase. Where it does not, we preserve the uncertainty instead of assigning every automated request to a famous provider.
Behavioral patterns such as repeated crawling, sitemap discovery, or rapid path traversal can strengthen the conclusion that traffic is automated, but behavior alone does not identify the organization behind it. We prefer labels such as recognized token, corroborated bot, or unclassified automation over a false binary of “verified AI” and “not AI.”
3. Keep crawler, referral, and revenue data separate
Automated request counts come from server or edge traffic. Human AI referrals come from web analytics and campaign/referrer signals. Subscription revenue comes from the billing provider. Machine-access revenue comes from successful settlement evidence. These can be compared on a timeline, but they should not be collapsed into a single “AI value” number without a causal basis.
A useful report can show that crawler activity and AI-origin referral sessions moved in the same period. It should not claim that one caused the other unless the measurement design supports that conclusion. Likewise, a payment challenge, quote, or successful verification is not booked as revenue. Economic claims require an actual billing or settlement event that can be reconciled.
4. Define the time window before looking at the result
Machine traffic can be bursty. A single crawler sweep can make one day look dramatically different from the next, and a product launch or policy change can create temporary behavior that is not representative of the following month. We therefore define observation windows before drawing conclusions and state when a sample is too small or too short to support a durable trend.
Comparisons should use equivalent windows whenever possible: seven days before versus seven days after a policy change, or month-over-month periods with the same inclusion rules. If the collection method, classifier, retention policy, or source coverage changed during the comparison, that change belongs in the interpretation because the apparent trend may be partly methodological.
5. Prefer reproducible policy experiments
When evaluating robots.txt or access policy, record the exact policy, deployment time, affected hostname and paths, then compare observed requests before and after. A useful experiment has a defined question and a measurable outcome: did the crawler stop requesting the protected path, did errors fall, or did human referral traffic materially change?
We also record what the experiment cannot answer. Blocking a crawler and then seeing fewer requests shows an access effect; it does not establish whether a provider removed previously collected material, changed a model, or altered every downstream product. Allowing a search crawler and later receiving a referral does not prove that the crawler caused the referral. Reproducibility is valuable only when the claim stays inside the experiment's observable boundary.
6. Treat “not observed” as different from “did not happen”
An empty result can be meaningful, but only within the coverage of the measurement. If a site has no observed GPTBot requests during a 30-day window, the defensible statement is that none were observed in the connected sources during that window. It is not proof that OpenAI never accessed the content through another hostname, historical crawl, partner source, user-requested fetch, or system outside the measured path.
The same rule applies to referral and conversion data. Missing campaign parameters, privacy controls, browser behavior, or analytics exclusions can reduce attribution coverage. Research pages should say when a metric is unavailable or incomplete rather than turning missing telemetry into a zero with more certainty than the instrumentation supports.
7. Minimize retained traffic data
Machine-web research does not require a warehouse of visitor identifiers. AIWebSignals's connected-ingestion design intentionally excludes raw IP addresses, cookies, query strings, request bodies, raw user agents, and raw request IDs from normalized connected storage. Research should retain only the evidence needed to answer the stated measurement question.
Data minimization also improves research discipline. If a conclusion can be supported by a query-free path, status code, timestamp, source classification, and privacy-minimized linkage, collecting unrelated personal or request-level detail adds risk without improving the answer. We design measurements around the minimum sufficient evidence rather than collecting everything first and deciding why later.
8. Keep source freshness and versioning visible
Crawler identities, robots controls, search products, and payment protocols change. Provider documentation is therefore treated as time-sensitive source material. Research briefs link to the underlying documentation, carry publication and update dates, and should be revised when a provider changes the meaning of a crawler token or control surface.
When a later source contradicts an earlier description, the current article should be corrected and the interpretation updated. We do not preserve a stale statement merely because it was true when first published. Where historical behavior matters, it should be labeled as historical rather than blended with the current operating guidance.
9. Separate ranking eligibility from ranking guarantees
Technical eligibility is not a promise of visibility. Allowing a crawler, publishing a sitemap, adding structured data, or making a page indexable can remove barriers to discovery, but none of those actions guarantees a citation, ranking, AI answer inclusion, or referral. Provider ranking and retrieval systems can use additional signals that are not visible to the publisher.
Our research therefore describes controls in terms of what they make possible or restrict. When we discuss search or generative-AI visibility, we measure impressions and referrals where those metrics are available instead of presenting access configuration as proof of editorial selection.
10. Economic claims require economic evidence
A crawl is not revenue, a payment requirement is not revenue, and a verified payment authorization is not necessarily settled revenue. We count economic value only when there is a defensible billing or settlement event. This is especially important for x402 experiments, where retries and duplicated responses can otherwise inflate a dashboard.
Where we eventually report machine-access economics, the report should distinguish requests shown a price, payment attempts, successful verification, successful settlement, previously recorded transactions, and net revenue actually attributable to the resource. Estimated opportunity can be useful for planning, but it must remain labeled as an estimate rather than quietly entering measured revenue totals.
11. Publish corrections instead of defending a stale conclusion
Machine-web research is young enough that responsible conclusions will sometimes change. If a provider publishes new crawler guidance, a protocol revision changes semantics, or a reproducible observation disproves an earlier interpretation, the right response is to update the article and its modification date. Significant corrections should explain what changed and why.
This correction policy is part of the product's credibility. The goal is not to accumulate permanent claims; it is to give site operators the most defensible current view of what can be observed, controlled, and measured.
How to use this publication
Use the research briefs to understand the policy and measurement model, the AI Bot Index to verify provider-specific identities, the guides for implementation details, and Activity Radar for a current public-machine-web observation layer. Where a conclusion depends on your own site, measure it on your own traffic instead of assuming a web-wide trend applies locally.
If a research page does not have enough evidence to answer a question, we would rather state that limitation than fill the gap with a confident narrative. That standard is especially important when the subject is commercially valuable: AI referrals, crawler licensing, access enforcement, and machine payments should become more measurable over time, but measurement only creates value when readers can distinguish what the system knows from what it merely suspects.