AI crawlers

The dashboard's crawler panel shows which AI assistants, search engines and training crawlers read your site. Hits are counted per crawler and per day, and per page when the page is known. They aren't visitors, and they don't use your event allowance.

Why it needs a line on your server

Most AI crawlers download your HTML and never run JavaScript, so the tracking script only sees the few that do, such as Googlebot. To count the rest, report each page request from your server or edge.

Report a request

Create an API key with the crawlers:write scope under Integrations, and keep it on the server. Then send the visitor's user agent, and the page's URL if you want to see which pages each crawler reads. Only the path is kept, never the query string, and page detail is kept for 90 days. Anything DataFair doesn't recognize as a listed crawler is ignored, so you can send every page request.

POST https://datafair.dev/api/v1/crawler-hits
Authorization: Bearer YOUR_API_KEY
Content-Type: application/json

{
  "website_id": "YOUR_WEBSITE_ID",
  "user_agent": "<the request's User-Agent>",
  "url": "<the requested URL or path, optional>"
}

Example: Next.js middleware

export function middleware(req, event) {
  event.waitUntil(fetch('https://datafair.dev/api/v1/crawler-hits', {
    method: 'POST',
    headers: { Authorization: `Bearer ${process.env.DATAFAIR_KEY}`, 'Content-Type': 'application/json' },
    body: JSON.stringify({ website_id: process.env.DATAFAIR_SITE, user_agent: req.headers.get('user-agent'), url: req.nextUrl.pathname }),
  }).catch(() => {}));
}

What it can't tell you

User agents can be faked, and new crawlers appear all the time. These are requests that identify themselves, not a verified list.

Documentation