AI crawlers and search bots

A directory of the AI crawlers and search bots that visit websites: what each one does, its user agent, and who runs it.

AI answers

Fetches a page because a person asked an assistant something.

Search indexing

Builds the index behind a search engine or AI search product.

AI training

Collects pages that may be used to train models.

Other bots

Ad checks, SEO research, link previews, and bots whose purpose is not documented.