AI crawlers and search bots
A directory of the AI crawlers and search bots that visit websites: what each one does, its user agent, and who runs it.
AI answers
Fetches a page because a person asked an assistant something.
- OpenAI ChatGPT-User: Opens a page while ChatGPT answers someone.
- Anthropic Claude-User: Opens a page while Claude answers someone.
- Perplexity-User: Reads a page to cite it in an answer.
- Google-Agent: Visits a page for an AI agent acting on a person's request.
- Google-NotebookLM: Fetches a page a person added as a source in NotebookLM.
- Google-Read-Aloud: Fetches a page so it can be read out loud to someone.
- GoogleAgent: Listed by third parties as a Google crawler. We could not confirm its purpose.
- MistralAI-User: Opens a page for a person using Le Chat.
- DuckDuckGo DuckAssistBot: Reads pages for AI-assisted search answers.
- Amazon Amzn-User: Fetches a page on request for Amazon assistants.
- meta-externalfetcher: Fetches a link a person gave to Meta AI.
- Moonshot AI Kimi-User: Opens a page while Kimi answers someone.
- xAI Grok-DeepSearch: Reported to read pages during a Grok research request. We found no xAI page that confirms it.
- xAI-SearchBot: Listed by third parties as a xAI crawler. We could not confirm its purpose.
- Alibaba Qwen-User: Listed by third parties as a Alibaba crawler. We could not confirm its purpose.
Search indexing
Builds the index behind a search engine or AI search product.
- Googlebot: Crawls pages for Google Search.
- Google-InspectionTool: Fetches a page for Google's search testing tools, such as URL Inspection.
- Google-CloudVertexBot: Crawls a site when its owner asks for it while building a Vertex AI agent.
- Microsoft bingbot: Crawls pages for Bing and Copilot search.
- Microsoft msnbot: An older Microsoft crawler name that still appears in some logs.
- OpenAI OAI-SearchBot: Indexes pages for ChatGPT search results.
- Anthropic Claude-SearchBot: Indexes pages for Claude search results.
- PerplexityBot: Indexes pages for Perplexity results.
- MistralAI-Index: Indexes pages for Le Chat search.
- Amazon Amzn-SearchBot: Indexes pages for Amazon search experiences.
- meta-webindexer: Indexes pages to improve search inside Meta AI.
- Moonshot AI Kimi-SearchBot: Indexes pages for Kimi search.
- Applebot: Crawls pages for Spotlight, Siri and Safari.
- DuckDuckGo DuckDuckBot: Crawls pages for DuckDuckGo search.
- YandexBot: Crawls pages for Yandex search.
- Baiduspider: Crawls pages for Baidu search.
- ByteDance TikTokSpider: Listed by third parties as a ByteDance crawler. We could not confirm its purpose.
- You.com YouBot: Listed by third parties as a You.com crawler. We could not confirm its purpose.
AI training
Collects pages that may be used to train models.
- OpenAI GPTBot: Collects pages that may train OpenAI models.
- Anthropic ClaudeBot: Collects pages that may train Anthropic models.
- GoogleOther: A general crawler Google teams use for research and development.
- Google-Extended: A robots.txt name that controls whether Google may use your pages for Gemini.
- Applebot-Extended: A robots.txt name that controls whether Apple may use your pages to train its models.
- meta-externalagent: Collects pages for Meta AI models.
- Moonshot AI KimiBot: Collects pages for Kimi models.
- ByteDance Bytespider: Reported to collect pages for ByteDance AI products. We found no ByteDance page that confirms it.
- Common Crawl CCBot: Builds an open web archive that is widely used for training.
- Amazonbot: Collects pages for Amazon AI and search products.
- cohere-ai: Listed by third parties as a Cohere crawler. We could not confirm its purpose.
- cohere-training-data-crawler: Listed by third parties as a Cohere crawler. We could not confirm its purpose.
- Allen AI AI2Bot: Collects pages for open AI research at the Allen Institute.
- Diffbot: Extracts structured data from pages for its knowledge graph.
- Baidu ERNIEBot: Listed by third parties as a Baidu crawler. We could not confirm its purpose.
- Alibaba QwenBot: Listed by third parties as a Alibaba crawler. We could not confirm its purpose.
- DeepSeekBot: Listed by third parties as a DeepSeek crawler. We could not confirm its purpose.
- Zhipu AI ChatGLM-Spider: Listed by third parties as a Zhipu AI crawler. We could not confirm its purpose.
Other bots
Ad checks, SEO research, link previews, and bots whose purpose is not documented.
- OpenAI OAI-AdsBot: Checks pages that are submitted as ads.
- meta-externalads: Fetches pages for Meta's advertising products.
- AhrefsBot: Maps backlinks for an SEO tool.
- SemrushBot: Gathers data for an SEO tool.
- Majestic MJ12bot: Maps backlinks for an SEO tool.
- Moz DotBot: Gathers link data for an SEO tool.
- Meta facebookexternalhit: Draws the preview when a link is shared.
- X Twitterbot: Draws the preview when a link is posted.
- LinkedInBot: Draws the preview when a link is posted.
- Slackbot: Draws the preview when a link is sent.
- Discordbot: Draws the preview when a link is sent.
- Meta WhatsApp: Draws the preview when a link is sent.
- TelegramBot: Draws the preview when a link is sent.
- Meta FacebookBot: Listed by third parties as a Meta crawler. We could not confirm its purpose.
- xAI GrokBot: Listed by third parties as a xAI crawler. We could not confirm its purpose.
- xAI-Bot: Listed by third parties as a xAI crawler. We could not confirm its purpose.
- xAI-Grok: Listed by third parties as a xAI crawler. We could not confirm its purpose.
- xAI-Web-Crawler: Listed by third parties as a xAI crawler. We could not confirm its purpose.
- ByteDance Doubaobot: Listed by third parties as a ByteDance crawler. We could not confirm its purpose.
- Baidu YiyanBot: Listed by third parties as a Baidu crawler. We could not confirm its purpose.
- Alibaba TongyiBot: Listed by third parties as a Alibaba crawler. We could not confirm its purpose.
- Alibaba AliyunBot: Listed by third parties as a Alibaba crawler. We could not confirm its purpose.