{"slug":"benchmark-methods-and-unknowns","title":"How to benchmark regulated websites without inventing certainty","description":"A study design that preserves sample boundaries, confidentiality, unknowns, and reproducible observations.","sources":[{"name":"IETF: Robots Exclusion Protocol, RFC 9309","url":"https://www.rfc-editor.org/rfc/rfc9309.html"},{"name":"AIWebSignals: current public capability contract","url":"https://aiwebsignals.com/api/capabilities"}],"sections":[["Define the population first","A convenience sample of sites found during implementation is not a representative industry sample. Specify the inclusion criteria, sampling frame, exclusions, observation period, and unit of analysis before publishing a percentage. A hostname, company, location, and page are not interchangeable units."],["Separate pilots from benchmarks","The initial research focus is cannabis-sector technical interfaces. Confidential operator-managed pilots must remain separate from any external cohort and must not be presented as independent validation. If a future approved example is informed by operator-managed work, disclose that methodological relationship without identifying the underlying business. At launch, no industry prevalence result is asserted."],["Preserve unknown states","A challenge, timeout, required login, or missing authorization can leave a question unanswered. Preserve that uncertainty rather than assigning failure. A score based on only observable fields can still mislead when observation coverage differs sharply between sites. Report coverage alongside findings, and do not force a composite ranking where comparability has not been established."],["Make methods reproducible without exposing pilots","Authorized working records should preserve the tested URL, timestamp, observer, methodology version, result state, and sanitized evidence. Public releases use only approved records or clearly labeled synthetic fixtures. Confidential URLs, screenshots, stable identifiers, and distinctive timestamps stay private. A private pilot cannot substantiate a publicly reproducible industry claim merely by omitting its name."],["Use a publication gate","The planned study requires a frozen external cohort, collected observations, contradiction review, re-identification review, and a documented correction channel. A study page may publish its protocol before results exist, but it must not fill the empty result table with illustrative percentages. Unknown is useful when the alternative is a fabricated industry conclusion."]],"next":"methodology","published":"2026-09-26","updated":"2026-09-26","status":"published","author":"AIWebSignals Research","url":"https://aiwebsignals.com/research/regulated-commerce/benchmark-methods-and-unknowns","example":{"type":"synthetic","heading":"Do not turn an inaccessible page into a failed site","text":"In an invented five-site exercise, two responses are security challenges and three can be inspected. Reporting the challenged sites as failed would invent findings; reporting only the inspectable sites without the exclusion would hide coverage. This is a methodological example, not our dataset.","exercise":"Freeze the sample and unit of analysis before measurement. Publish observation coverage, unknowns and exclusions alongside any eligible findings. Keep operator-managed pilots outside external benchmark denominators."},"methodology":"https://aiwebsignals.com/research/regulated-commerce/methodology"}