AI crawlers and indexing bots: the reference profiles
GPTBot, ClaudeBot, PerplexityBot, Googlebot… every robot visiting your site has an operator, a purpose and rules of its own. These reference profiles document, for each crawler, the verifiable facts: the user-agent string, the robots.txt token, the official IP ranges when the operator publishes them, and what its visits mean for your SEO and your visibility in AI answers (GEO).
Every technical fact links to its primary source — the operator's official documentation. The IP ranges shown are refreshed automatically on every site deployment, from the files published by the operators themselves.
Every crawler in one table: robots.txt token, purpose and the cost of blocking it
This table brings together the twelve crawlers documented on this site. The last column is the one that matters for a decision: what you actually lose by blocking each agent. Blocking a training crawler and blocking a citation crawler do not have remotely the same consequences.
| Crawler | Operator | Robots.txt token | Purpose | IP-verifiable | What you lose by blocking it |
|---|---|---|---|---|---|
| GPTBot | OpenAI | GPTBot | Training | Yes | No effect on ChatGPT search. Your content leaves OpenAI's future training corpora. |
| OAI-SearchBot | OpenAI | OAI-SearchBot | Citation | Yes | Your site disappears from ChatGPT search: it can no longer be cited there with a link. |
| ChatGPT-User | OpenAI | ChatGPT-User | Agent (real time) | Yes | A ChatGPT user following a link to your page can no longer load it. |
| ClaudeBot | Anthropic | ClaudeBot | Training | No | No effect on Claude's answers. Your content leaves Anthropic's training corpora. |
| Claude-SearchBot | Anthropic | Claude-SearchBot | Citation | No | Your content is no longer indexed for Claude's search results. |
| Claude-User | Anthropic | Claude-User | Agent (real time) | No | A Claude user asking to read your page no longer gets it. |
| PerplexityBot | Perplexity | PerplexityBot | Citation | Yes | Your site no longer appears as a cited source in Perplexity's answers. |
| Google-Extended | Google-Extended | Training | No | No effect on Google Search or your ranking. Your content no longer feeds Google's generative AI products. | |
| GoogleOther | GoogleOther | Search | Yes | No documented effect on Google Search indexing. Covers internal uses (R&D, products). | |
| Googlebot⚠ multi-purpose | Googlebot | Search | Yes | Gradual removal from Google Search: pages stop being recrawled and rankings decay. | |
| Bingbot⚠ multi-purpose | Microsoft | Bingbot | Search | Yes | Removal from Bing, and loss of the source feeding Copilot. |
| Applebot⚠ multi-purpose | Apple | Applebot | Search | Yes | Disappearance from Siri, Spotlight and Apple Intelligence. |
Watch out for multi-purpose crawlers. Googlebot, Applebot and Bingbot crawl to index but also collect data that may serve training. Cloudflare handles them under the most restrictive rule: since its announcement of 1 July 2026, a site blocking the “Training” category may stop serving Googlebot — and gradually drop out of Google's index.
What the 15 September 2026 switch changes →The robots.txt token states what you ask for. Only your server logs show what actually happened — and a user-agent can be forged with a single command-line flag.
How to detect AI crawlers in your server logs →The detailed profiles, by operator
OpenAI — ChatGPT
GPTBot
Training crawler
Official GPTBot documentation. User-agent string, verified IP ranges, robots.txt rules, how to detect OpenAI's crawler in server logs and block it.
OAI-SearchBot
ChatGPT search index
Official OAI-SearchBot documentation. User-agent string, verified IP ranges, robots.txt rules — the OpenAI crawler that makes you visible in ChatGPT search.
ChatGPT-User
Live browsing (user-triggered)
ChatGPT-User fetches a page when a ChatGPT user asks for it. Not a bulk crawler: every hit is a potential reader. IP ranges, robots.txt, measurement.
Anthropic — Claude
ClaudeBot
Training crawler
ClaudeBot collects web content to train Anthropic's Claude models. User-agent, robots.txt compliance, and why IP verification of this bot is limited.
Claude-SearchBot
Search result quality
Official Claude-SearchBot documentation. User-agent string, robots.txt rules, how to detect Anthropic's search crawler in your server logs and allow it.
Claude-User
Live browsing (user-triggered)
Claude-User fetches a page when a Claude user asks for it. Every hit is a real human read through AI. Detection, robots.txt, measurement in your logs.
Google — Search, AI Overviews & Gemini
Google-Extended
robots.txt token (Gemini training)
Google-Extended is not a crawler: it's a robots.txt token controlling whether your content trains Gemini — without touching your SEO or rankings.
GoogleOther
Generic crawler (R&D, products)
GoogleOther is Google's “generic” crawler, used outside Search indexing: R&D, products, internal uses. Detection, IP ranges, robots.txt token.
Googlebot
Google Search indexing
Googlebot feeds the Google Search index — and AI Overviews. User-agent, official IP ranges, anti-spoofing verification, robots.txt best practices.
Perplexity — answer engine
Microsoft — Bing & Copilot
Apple — Siri & Apple Intelligence
To find out which of these bots actually visit your site — and verify their authenticity by IP — drop your server logs into our analyzer: everything happens in your browser, no upload.
Analyze my logs