{"slug":"policy-is-not-observation","title":"Robots.txt permission is not evidence of an AI visit","description":"A practical evidence model for declared crawler policy, actual responses, verified requests, and downstream outcomes.","sources":[{"name":"IETF: Robots Exclusion Protocol, RFC 9309","url":"https://www.rfc-editor.org/rfc/rfc9309.html"},{"name":"MachineRealms: published interoperability evidence","url":"https://machinerealms.com/api/v1/research/interoperability"}],"sections":[["Four records, four meanings","Consider four statements: a robots rule permits an article, an audit retrieved the article, a verified provider request reached the article, and a person arrived from an AI service. Each statement has a different source and denominator. Reporting them under a single visibility metric makes failures hard to diagnose and successes easy to exaggerate."],["Start with the declared rule","RFC 9309 describes cooperative crawler rules and explicitly distinguishes them from access authorization. Record the hostname, path, relevant crawler token, fetched policy version, and interpretation in the working evidence record. That record establishes what the policy declares. It does not tell you whether a provider scheduled a visit, whether a firewall challenged the request, or whether retrieved content was used in an answer."],["Then inspect the response","A bounded public test can establish that one observer received one response at a particular time. Preserve the observer and conditions. If the response is a challenge page, classify the result as challenge-limited rather than successful simply because the HTTP status was 200. If the test times out, the result is timeout from that vantage, not a universal outage."],["Ask for connected evidence when needed","Actual request history requires first-party or otherwise authorized logs. Identity claims require the method used to validate them. Even verified retrieval does not reveal training use, private prompts, model reasoning, citation, or commercial impact. Those remain different questions with different evidence requirements."],["Use the distinction in decisions","A policy-response contradiction warrants an infrastructure investigation. A healthy response with no observed provider traffic suggests a different question about demand, discovery, or observation coverage. A referral without a matching crawler event should remain a referral, not be assigned to a guessed bot. This separation makes the observatory useful even when the answer is unknown."]],"next":"ai-referrals-vs-crawlers","published":"2026-09-26","updated":"2026-09-26","status":"published","author":"AIWebSignals Research","url":"https://aiwebsignals.com/research/regulated-commerce/policy-is-not-observation","example":{"type":"synthetic","heading":"Allowed at one layer, unavailable at another","text":"A synthetic robots file allows /guides/, while the test observer receives a security challenge at /guides/overview. This demonstrates a difference between the declared rule and that observer response. It does not establish that every crawler is blocked or that the page was used in a model.","exercise":"Write two findings: the applicable declared rule and the actual response class. Add observer and time to each. Do not replace either record with a single yes/no visibility label."},"methodology":"https://aiwebsignals.com/research/regulated-commerce/methodology"}