5 Ways to Check If AI Crawlers Are Scraping Your Website (With Commands)

AI crawlers read websites every day, but almost no site owner can see them. The reason is simple: analytics tools rely on JavaScript running in the browser to count visitors, while AI crawlers only download HTML and execute no JavaScript. To see them, you have to look at where requests arrive — the server or the CDN.

Here are five methods, starting from zero cost.

First, figure out which crawlers to watch

AI crawlers fall into three categories, and they mean completely different things:

Type	Examples	What a request means
Training crawlers	GPTBot, ClaudeBot, CCBot	The page was collected for model training
AI search crawlers	OAI-SearchBot, Claude-SearchBot, PerplexityBot	The page entered the index that assistants retrieve from
Fetching on behalf of a user	ChatGPT-User, Claude-User, Perplexity-User	Someone asked an assistant a question right now, and it opened your page

The third category is the most worth watching. It is closest to the signal "my page was used in an answer."

user-agent (UA for short) is a field in the HTTP request header that a client uses to identify itself, usually in a format like Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot). Every identification method in this article — grep regex, middleware matching, WordPress hooks — is essentially just comparing keywords in the user-agent. Two things are key to understand: first, user-agent can be completely spoofed, and any script can claim to be GPTBot, so it can only be used for "categorical statistics," not "identity verification"; second, major AI vendors document their crawlers' user-agent names and IP ranges in official docs, and the proper approach is to combine user-agent matching with source IP verification. In addition, some crawlers use multiple names (for example, ChatGPT-User and OAI-SearchBot both belong to OpenAI), so for statistics it is often more meaningful to group by company than by a single UA.

Method 1: Server access logs (free)

Nginx and Apache already write every request to a log. First, count the requests per AI crawler:

grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User" /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn

Then look at which pages a given crawler requested (in the default log format, the path is the 7th field):

grep "ChatGPT-User" /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

And the requests that returned 404 — the crawler thought a page existed there but did not find it (the status code is the 9th field):

grep -E "GPTBot|ClaudeBot|PerplexityBot" /var/log/nginx/access.log | awk '$9 == 404 {print $7}' | sort | uniq -c | sort -rn | head

Use case: one-off troubleshooting on a VPS. Limitations: logs rotate after days to weeks, a CDN in front of the server hides requests by serving from cache, and what you get is a number, not a trend.

The commands in the article rely on the "combined" log format for Nginx/Apache, whose field order is: $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent". After splitting on spaces, the 1st field is the client IP, the 7th field is the request path (including the query string), the 9th field is the HTTP status code, and the last field is the user-agent. That is why the scripts use $7 to get the path and $9 to detect 404s — they correspond to "which field after splitting on spaces." If you have customized log_format in Nginx (for example, by removing referer or using JSON format), the field numbers change, and you need to run head -1 access.log first to check the actual structure and then adjust. The same is true for Apache: the standard combined format is identical to Nginx, but if you enable custom fields like %D, things shift. Only by understanding this can you migrate these grep/awk commands to your own environment.

Method 2: CDN console (free)

The CDN sees every request earlier than the server does, including those it answers directly from cache. The Cloudflare dashboard has an AI crawlers section that lists the AI crawlers requesting your site and lets you allow or block them individually. Other CDNs and hosts (Vercel, Netlify, Fastly) show requests in their logs, and you can filter by user agent.

Use case: quickly answering "which AI companies are crawling me, and how much." Limitations: you usually cannot see time trends by page, nor what happened afterward (did anyone visit, did anything convert).

Method 3: A few lines of middleware (for Next.js and similar frameworks)

If your framework supports running code on every request, log crawlers there. In Next.js, for example:

// middleware.ts (proxy.ts in Next.js 16)
import { NextResponse, type NextRequest } from 'next/server'

const AI = /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User/i

export function middleware(req: NextRequest) {
  const bot = req.headers.get('user-agent')?.match(AI)?.[0]
  if (bot) console.log(JSON.stringify({ bot, path: req.nextUrl.pathname }))
  return NextResponse.next()
}

export const config = { matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'] }

These logs go into your host's function logs. The same idea applies to Express, Cloudflare Workers, or any server-side framework.

Limitations: you still need to store, count, and chart everything yourself, and middleware runs before the page, so it cannot get the response status code.

Middleware is a function that runs before a request reaches the specific page handler. It can read request headers, URL, and cookies, and decide whether to pass through, rewrite, or directly return a response. Next.js has supported placing middleware.ts in the project root since version 12, executing that code uniformly for matching paths — which makes "logging crawlers on every request" very lightweight. Three characteristics are worth noting: first, middleware runs in the Edge Runtime and cannot use Node.js fs, and console.log only writes to the hosting platform's function logs (such as Vercel's Logs panel); second, config.matcher uses a regex to exclude static assets, preventing logs from being flooded by _next/static; third, middleware executes before the page renders, so it can only get request information and not the response status code — this explains the "limitation" mentioned in the original article. To log 404s, you need to intercept at the page or API layer instead.

Method 4: WordPress

WordPress runs PHP on every uncached request, so a small snippet placed in a plugin or child theme can log AI crawlers:

add_action('template_redirect', function () {
    $ua = $_SERVER['HTTP_USER_AGENT'] ?? '';
    if (preg_match('/GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User/i', $ua, $m)) {
        error_log('AI bot ' . $m[0] . ' ' . ($_SERVER['REQUEST_URI'] ?? '/'));
    }
});

Pitfall: page caching (WP Rocket, LiteSpeed, CDN) returns cached pages directly without executing PHP, so cached pages will not be counted. Either exclude crawler user agents from caching, or count at the CDN layer instead.

Method 5: Automated analytics tools that read crawlers from the server side

The first four methods all give you raw requests. To turn them into something actionable, you need to store them, group them by page and company, and correlate them with what users do afterward. I built this logic into PageLens: a small piece of code on the server reports crawler requests, and a dashboard shows which pages AI crawlers have read, which pages assistants opened to answer questions, the 404s they hit, and the visits and revenue coming from AI assistants.

Verify whether a crawler is real

Anyone can send a request claiming to be GPTBot. OpenAI, Anthropic, and Perplexity all publish the IP ranges their crawlers use, so a request carrying a crawler name but coming from a different address is a fake. One study examined 740 top websites and found that 9% of sites reject requests carrying AI crawler names, even though their robots.txt actually allows those crawlers.

The specific method for IP range verification is: first filter out requests that "claim to be GPTBot" by user-agent, then extract their source IP ($remote_addr or the CF-Connecting-IP header returned by the CDN), and check whether it falls within OpenAI's officially published IP list. OpenAI, Anthropic, and Perplexity all provide CIDR-format network ranges in their documentation, or offer a JSON file that can be pulled periodically (such as OpenAI's https://openai.com/gptbot.json). In engineering practice, you usually write a script that periodically pulls these ranges and caches them locally, then performs an ip-in-CIDR check on each request. The reverse is also valuable: if you find a large number of "GPTBot" requests coming from cloud providers (AWS, Hetzner) rather than official ranges, they are most likely spoofed crawlers and should be blocked outright. The "9% of sites mistakenly rejecting real crawlers" mentioned in the article is precisely because this verification step was skipped, and blocking based only on UA matching or intuition ends up shutting out legitimate crawlers too.

What to do once you have the data

• Read a lot but never opened to answer questions: AI knows the page exists but thinks it is not useful for answering. Put the question in the title and the answer in the first two sentences.
• Opened to answer questions but never visited: you were used as a source but were not clicked. Check how the page describes you, and whether it gives a reason to click.
• Important pages have never been read: link to them from the homepage and sitemap, and check robots.txt and CDN configuration.
• Crawlers are hitting 404s: they expect a page there, usually /pricing, /docs, or /llms.txt. Create it or redirect it.

The field numbers in the commands correspond to the default "combined" log format for Nginx and Apache. If your format is different, adjust accordingly.