How to Detect AI Crawlers in Your Server Logs: The Complete Method

In short — AI crawlers do not execute JavaScript, so they are completely absent from Google Analytics and any tag-based tool. Your server logs are the only source that records them. Detecting them takes two steps: identify the user-agent string (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot…), then verify that the IP address belongs to the operator's official ranges — a user-agent alone can be forged with a single command-line flag. OpenAI, Google, Perplexity and Apple publish their ranges in JSON files; Anthropic does not, which makes its crawlers unverifiable by this method. In our own logs, eight days of observation were enough to show that three crawlers presented as equivalent behave very differently.

An SEO lead who opens their analytics tool, sees no AI crawler, and concludes that these engines have no interest in their site is making an understandable mistake — and a structural one. AI crawlers trigger no JavaScript tag, so no tool built on that mechanism can see them. They arrive, they read, they leave, and the dashboard stays empty.

This article sets out the complete method for detecting them at the source, in your server request log: which fields to read, how to recognise each crawler, and above all how to tell a genuine crawler from a scraper impersonating one. Every point is illustrated with figures from our own logs.

Why AI crawlers are invisible in Google Analytics

Consumer analytics tools work through a tag: a JavaScript snippet runs in the visitor's browser and sends a signal. That mechanism assumes a client that executes code. Nearly all training crawlers simply fetch the raw HTML and never interpret it.

The consequence is stark: a site can receive several hundred AI crawler visits per week without a single one appearing in its analytics report. A server log, by contrast, records every request received, whether it comes from a human, a search engine or a script. This is a difference in kind, not in accuracy.

This invisibility has an underrated practical consequence: it makes any informed decision about blocking AI crawlers impossible. Choosing to allow or refuse a crawler without knowing what it actually consumes amounts to legislating blind.

Where to look: the useful fields in a log line

A log line in the combined format, the most common on Apache and Nginx, contains everything you need. Four fields matter for detection: the client IP address, the timestamp, the requested URL with its status code, and the user-agent at the end of the line.

The user-agent handles identification, the IP address handles verification, the status code handles diagnosis, and the URL tells you what the crawler is interested in. None of these fields is sufficient alone; it is their combination that produces reliable information. If you are unsure where to find these files, our guide covers how to retrieve your logs depending on your hosting.

One point deserves attention on modern stacks: some crawlers that render JavaScript generate framework-internal requests, recognisable by technical query parameters. In our logs, 86.2% of GoogleOther's requests were not pages but internal payloads. Counting them as page visits distorts any reading of volume.

Recognising an AI crawler by its user-agent string

Every operator publishes the exact string used by its crawlers. Detection means searching for a characteristic substring, case-insensitively, in the user-agent field.

On the OpenAI side, three distinct agents cover three roles: GPTBot collects content for model training, OAI-SearchBot builds the ChatGPT search index, and ChatGPT-User fetches a page live when a user clicks a citation. These are three separate robots.txt tokens and three independent decisions — the detail for each is on the GPTBot, OAI-SearchBot and ChatGPT-User pages.

On the Anthropic side the split is comparable: ClaudeBot for training, Claude-SearchBot for search result quality, Claude-User for user-triggered fetches. On the Google side, Googlebot crawls for Search, Google-Extended governs how content is used by generative AI products, and GoogleOther covers other internal uses. PerplexityBot and Applebot complete the list of common agents.

We maintain a page per crawler in our crawler reference, each with the full string, the robots.txt token, the official source and the visibility stake attached to it.

Why the user-agent is never enough: impersonation

The user-agent field is declarative. It is set by the client, never verified by the server, and anyone can announce themselves as Googlebot or GPTBot by adding a flag to a curl command. Detection based on that field alone is therefore structurally unreliable.

This is not a theoretical risk. Commercial scrapers routinely impersonate legitimate crawlers, precisely because most sites let them through unchecked. The direct consequence: an unverified crawl report overstates how much search engines actually care about your site, and can lead you to decisions built on noise.

The stake is defensive too. A site that decides to block a training crawler gains nothing by filtering on the user-agent: the impersonator changes its string, the legitimate crawler obeys. The block then hits exactly the wrong actor — the one following the rules.

Verifying crawler authenticity by IP range

The reliable method is to check that the request's IP address belongs to the ranges officially published by the crawler's operator. Unlike a user-agent, an IP address cannot be forged in a completed HTTP request: the response has to come back somewhere.

OpenAI publishes its ranges in dedicated JSON files, separate for each of its agents. Google publishes its own, as do Perplexity and Apple. Verification means loading those lists, then testing whether each observed address falls inside the declared blocks.

Anthropic does not publish official IP ranges, and its documentation states that IP-based blocking is unreliable. ClaudeBot, Claude-SearchBot and Claude-User are therefore not verifiable by this method: a real limitation, better known than papered over with false certainty.

For Googlebot a second, authoritative method exists: reverse DNS lookup. You resolve the IP address to a hostname, check that the hostname belongs to a Google domain, then resolve that name forward again to confirm it points back to the original address. The round trip is essential — a one-way lookup can be spoofed.

→ Check your own logs in the Crawl Lab log analyzer: it isolates each crawler, tests every request against official IP ranges, and separates verified hits from likely impersonations. Everything runs in your browser, no file is ever uploaded.

The CDN trap: when the IP in your logs is not the crawler's

There is one configuration in which IP-range verification fails silently, and it is extremely common: the site is served behind a CDN or reverse proxy. In that case the connection your origin server sees does not come from the crawler, but from the CDN node relaying the request.

The symptom is distinctive: every request appears to originate from a handful of addresses belonging to your infrastructure provider, and no crawler comes out as authentic, because no observed address falls inside OpenAI's or Google's ranges. The report concludes that impersonation is rampant, when the problem is purely a logging configuration issue.

The fix is to log the origin address forwarded by the proxy rather than the connecting address. CDNs add this information in a dedicated header — X-Forwarded-For in the general case, CF-Connecting-IP on Cloudflare — and you need to adapt your Apache or Nginx log format to write it into the file. Without that step, authenticity verification is worthless.

This CDN dependency has a second consequence, developed in our article on the default AI crawler block taking effect on 15 September 2026: a crawler refused at the network layer never reaches your server, and therefore leaves no line in your logs at all. It does not produce an error, it produces an absence — the hardest signal to spot without an earlier point of comparison.

Telling the three families apart: training, citation, agent

Detecting a crawler is not enough; you need to know what it does with your content. Three families are distinguished by purpose, and that distinction governs every decision that follows.

Training crawlers collect text to feed the corpora used to train models. The content is absorbed into the model's weights: there is no citation mechanism and no referral traffic. GPTBot, ClaudeBot and Google-Extended belong to this category.

Citation crawlers build an index used to answer questions while citing sources, with a clickable link. These are the ones that can send you readers: OAI-SearchBot, Claude-SearchBot and PerplexityBot. Blocking them means opting out of sourced AI answers.

Agents fetch a page in real time because a human asked for it, such as ChatGPT-User. Their volume is low but their signal is strong: each request maps to a real user intent. This three-way split mirrors the categories Cloudflare has made configurable, a change whose consequences we cover in our article on the default block taking effect on 15 September 2026.

What volume reveals: reading the signals in the numbers

Once crawlers are identified and verified, their crawl profile becomes interpretable. In our own logs — a site launched thirteen days earlier, 1,722 crawler requests over eight days — three clearly distinct behaviours emerged.

GPTBot issued 56 requests from three IP addresses, covering 49 distinct URLs: a breadth-first sweep, almost one page per request. ClaudeBot produced 224 requests across only 56 URLs over the same period: insistent revisiting of a narrow perimeter. Two training crawlers, two opposite strategies.

OAI-SearchBot issued 57 requests from fourteen IP addresses, a third of which carried a framework-internal parameter — the signature of a crawler that executes the page's JavaScript. GPTBot produced no such request. If your content depends on client-side rendering, OpenAI's two crawlers do not see the same version of your site.

Claude-SearchBot did not appear once. We draw no rule from that: the absence of a citation crawler on a two-week-old domain is expected behaviour, not a symptom. The full detail of these measurements is in our case study on a new site's crawl budget.

Common false positives to rule out

Three misreadings come up constantly. The first counts infrastructure file requests — robots.txt, sitemaps, llms.txt — as wasted crawl budget. Those files guide crawling: a crawler is supposed to fetch them, and their presence in your logs is a sign things are working.

The second treats framework-internal requests as page views. On a modern stack, a crawler that renders JavaScript can multiply technical requests for a single real page, artificially inflating its apparent volume.

The third interprets a volume change without looking at status codes. A crawl drop may reflect waning engine interest — or simply the fact that your server returned errors. In our logs, during three days of 500 errors, one crawler's activity fell by 81% while another lost only 16%. Reading status codes is covered in our guide to 404 and 5xx errors as crawlers see them.

Turning detection into a routine

A one-off detection is worth little; it is the series that informs. Recording monthly volume per crawler, the share of verified requests, and the distribution of status codes lets you catch an anomaly as it happens rather than six weeks later.

Once the process is established this takes a few minutes, and it is the only defensible basis for deciding whether to block AI crawlers. The KPIs to track and how to interpret them are covered in a dedicated guide.

Three numbers are worth writing down each month. The first is the share of requests that pass IP verification: a sharp drop suggests either a scraping campaign impersonating a known crawler, or a logging configuration that stopped forwarding the origin address. The second is the ratio of citation-crawler to training-crawler traffic, which tells you whether your content is being indexed for answers or merely absorbed. The third is the error rate served to verified crawlers, the single figure most likely to explain a crawl decline that looks like disinterest but is in fact self-inflicted.

Read next

• Server Log Analysis for SEO: The Complete 2026 Guide
• Cloudflare Blocks AI Crawlers by Default on 15 September 2026
• A new site's crawl budget: what our logs show
• SEO log analysis: which KPIs to track and how to read them
• Getting your server logs: Apache, Nginx, cPanel, CDN