Crawler directory

meta-webindexer

Search indexing

About this crawler

Indexes pages to improve search inside Meta AI.

Your page is being read for a search index. Being in that index is what lets the product show or cite your page later.

User agent

meta-webindexer

DataFair counts a request as meta-webindexer when its user agent contains that name, in any letter case. Requests like this stay out of your visitors and your monthly allowance.

IP addresses

We did not find an address list for this crawler. A count means a request used this name, not proof of who sent it.

Control it in robots.txt

User-agent: meta-webindexer
Disallow: /

This asks the crawler to stay away. Blocking it can also keep you out of its answers or results, so decide with care. Fetches a person triggers may not follow robots.txt, so check the operator's documentation.

Sources

Meta documentation

More from Meta

Track crawlers with DataFair