Crawler directory

Google-Extended

AI training

About this crawler

A robots.txt name that controls whether Google may use your pages for Gemini.

You won't see this name in your traffic. Google reads it from your robots.txt to decide how your pages may be used, and its other crawlers do the fetching.

User agent

Google-Extended

DataFair never counts this name, because no request carries it.

IP addresses

This name has no addresses, because no request carries it.

Control it in robots.txt

User-agent: Google-Extended
Disallow: /

This asks the crawler to stay away. Compliance is up to the operator.

Sources

Google documentation

More from Google

Track crawlers with DataFair