ClaudeBot: Anthropic's training crawler
ClaudeBot is Anthropic's web crawler collecting content to train the Claude models. Like OpenAI's GPTBot, it builds a training corpus: it maintains no search index and cites no sources. Anthropic documents that it honors robots.txt — but, notably, publishes no official IP ranges, which limits verification.
Technical profile
- Operator
- Anthropic
- Role
- Training crawler
- robots.txt token
ClaudeBot- User-agent (example)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)- Respects robots.txt
- Yes (documented by Anthropic)
- Official IP ranges
- Not published by the operator (IP verification impossible)
- Official documentation
- Anthropic crawling documentation
ClaudeBot's role in the Anthropic ecosystem
Anthropic distinguishes three agents: ClaudeBot (training collection), Claude-SearchBot (search result quality) and Claude-User (live fetches when a Claude user asks). Blocking ClaudeBot is therefore a training opt-out that doesn't remove Claude's ability to fetch and cite your pages on demand.
In the logs of young sites, ClaudeBot is often among the most active AI crawlers by volume — its regular visits indicate your content is considered relevant for Anthropic's corpora.
The specificity: no published IP ranges
Unlike OpenAI, Google, Microsoft, Apple or Perplexity, Anthropic publishes no official list of IP ranges for its agents. Direct consequence: a “ClaudeBot” hit can be neither formally confirmed nor denied by IP. Serious analyses label it “unverifiable” rather than “verified” — which is exactly what our analyzer does, out of methodological honesty.
Verifying it's really ClaudeBot
Without official IP ranges, verifying ClaudeBot relies on weak signals: volume consistency, plausible cloud origin, regular crawl behavior. Our analyzer displays this “doubt” status explicitly instead of faking certainty — a “ClaudeBot” behaving aggressively or erratically deserves suspicion.
Blocking or allowing ClaudeBot (robots.txt)
Anthropic documents that ClaudeBot honors robots.txt, including targeted Disallow rules. Blocking it removes your content from Claude's future training corpora, with no effect on Claude-User (live browsing) or Claude-SearchBot, which have their own tokens.
To block it:
User-agent: ClaudeBot Disallow: /
To explicitly allow it:
User-agent: ClaudeBot Allow: /
Frequently asked questions
Should I block ClaudeBot?
Same trade-off as GPTBot: it's an editorial training opt-out. Blocking it prevents neither Claude from fetching your pages on a user's request (Claude-User), nor search result improvements (Claude-SearchBot). If your priority is visibility in AI answers, letting it through costs nothing.
How do I verify a ClaudeBot hit is genuine?
You can't prove it formally: Anthropic publishes no official IP ranges, unlike OpenAI or Google. Best practice is to label these hits “unverifiable” and watch for abnormal behavior (sudden volume, aggressive crawling), the likely signs of an impostor.
Does ClaudeBot respect robots.txt?
Yes, per Anthropic's official documentation: ClaudeBot reads and honors robots.txt directives, and Anthropic states blocked pages are excluded from collection. An agent claiming to be ClaudeBot while ignoring your directives is most likely fake.
Why is ClaudeBot crawling my site so much?
High ClaudeBot volume is common, especially after a site is discovered or fresh content is published: training crawls are periodic and broad. If it becomes disproportionate, Crawl-delay or targeted Disallow rules in robots.txt are honored according to Anthropic.
Sources
Every fact on this page is backed by a primary source — the operator's official documentation and, where it exists, its published IP list.
See what this bot does on your site
Drop your access.log files into the Crawl Lab log analyzer: it isolates ClaudeBot, verifies its authenticity and lists the pages it requests — entirely in your browser, no upload.
Analyze my logs