The observability mistake
If an analyst searches server logs for a Google-Extended user-agent string, the expected result is nothing. Google documents that crawling is performed with existing Google user agents and that Google-Extended exists as a control token in robots.txt. This makes it fundamentally different from tokens such as OAI-SearchBot, ClaudeBot, or PerplexityBot, which can be associated with direct request identities.
That distinction matters for analytics. A site cannot count 'Google-Extended requests' because there is no separate request stream to count. The observable artifact is the published policy, while the underlying crawl traffic remains associated with Google's ordinary crawling infrastructure.
What the token controls
Google says Google-Extended can be used to manage whether content Google crawls may be used for training future generations of Gemini models and for certain grounding uses in Gemini products and Vertex AI. The company also states that the token does not affect a site's inclusion in Google Search and is not used as a Search ranking signal.
That creates a useful policy separation. A publisher can maintain ordinary Google Search crawl access while expressing a different preference about specified Gemini training or grounding uses. The robots policy is therefore a product-use control rather than a search-indexing switch.
AI Overviews and AI Mode follow Search fundamentals
Google's guidance for AI features in Search says the foundational SEO requirements remain the same: pages need to be indexable, eligible to appear in Search with a snippet, and useful to people. Google does not require a special AI file or a separate AI crawler for eligibility. Controls such as noindex, nosnippet, data-nosnippet, and max-snippet affect what can be surfaced from indexed pages.
For publishers, this is strategically important. Blocking Google-Extended should not be interpreted as opting out of Google Search AI features. Conversely, allowing Google-Extended is not a shortcut into AI Overviews or AI Mode. Search visibility still depends on ordinary indexing and quality systems.
How to represent this correctly in an AI traffic product
A machine-web dashboard should classify Google-Extended as a policy signal, not fabricate a crawler activity count. The product can report whether the token is allowed or disallowed in robots.txt and explain the affected product class. It should not imply that an absent log entry means the policy is being ignored.
This kind of semantic precision is exactly why bot observability needs provider-specific knowledge instead of a generic list of strings.
Original evidence: explicit Google-Extended control appeared on 2 of 10 sites
On September 8, 2026, AIWebSignals inspected the same fixed ten-domain technology cohort used for its OpenAI and Anthropic policy studies. All ten had a permissive Google-Extended robots policy at the homepage path, but only two explicitly named the product token; eight inherited broader wildcard/default policy. This is a robots policy-expression observation. Google-Extended has no separate HTTP request user-agent string, so the study does not claim distinct Google-Extended crawler traffic.
Read the reproducible 10-domain Google-Extended policy study.
Related technical evidence
Google-Extended robots.txt policy study · Google-Extended in the AI Bot Index · robots.txt for AI crawlers