Crawler directory

Common Crawl CCBot

AI training

About this crawler

Builds an open web archive that is widely used for training.

Your page is being collected, and it may be used to build AI models. No person is waiting on this visit.

User agent

CCBot

DataFair counts a request as CCBot when its user agent contains that name, in any letter case. Requests like this stay out of your visitors and your monthly allowance.

IP addresses

Common Crawl publishes the addresses this crawler uses. DataFair does not compare requests against them yet, so a count means a request used this name, not proof that Common Crawl sent it.

https://index.commoncrawl.org/ccbot.json

Control it in robots.txt

User-agent: CCBot
Disallow: /

This asks the crawler to stay away. Compliance is up to the operator.

Sources

Common Crawl documentation

More ai training crawlers

Track crawlers with DataFair