Crawler directory

DuckDuckGo DuckDuckBot

Search indexing

About this crawler

Crawls pages for DuckDuckGo search.

Your page is being read for a search index. Being in that index is what lets the product show or cite your page later.

User agent

DuckDuckBot

DataFair counts a request as DuckDuckBot when its user agent contains that name, in any letter case. Requests like this stay out of your visitors and your monthly allowance.

IP addresses

DuckDuckGo publishes the addresses this crawler uses. DataFair does not compare requests against them yet, so a count means a request used this name, not proof that DuckDuckGo sent it.

https://duckduckgo.com/duckduckbot.json

Control it in robots.txt

User-agent: DuckDuckBot
Disallow: /

This asks the crawler to stay away. Blocking it can also keep you out of its answers or results, so decide with care. Fetches a person triggers may not follow robots.txt, so check the operator's documentation.

Sources

DuckDuckGo documentation

More from DuckDuckGo

Track crawlers with DataFair