AI crawlers · Anthropic

ClaudeBot

ClaudeBot is a crawler operated by Anthropic, and it collects text to train a model — the fact that decides everything else on this page.

What does ClaudeBot actually do?

ClaudeBot is Anthropic's training crawl, and the registry lists two user-agent strings for it: the current name and the older anthropic-ai token. Robots files written a while ago often name only the legacy string, or a Claude-Web token that no longer matches anything that arrives. If your rule predates the renaming, check it still matches real requests.

The block is cheap and one-directional. No live conversation depends on this crawl, so nothing user-facing breaks. But text already collected does not come back out of a model that has been trained on it, which means the decision only ever governs future runs - there is no version of this rule that reaches backwards.

There is no partial setting available. Google gives you snippet and preview directives that let you stay indexed while limiting how much of a page can be shown; Anthropic gives you path-level robots rules and nothing finer. You decide per URL prefix whether the training crawl may read a page, and that is the entire vocabulary.

There is one dial beyond allow and deny, and it governs rate rather than scope: Anthropic documents support for the non-standard Crawl-delay directive, which Google has never honoured and Amazon explicitly declines. For a training crawl that is the most useful control in the whole file, because nobody is waiting on any individual fetch, so slowing it down costs you nothing that anyone will notice.

CrawlClick files ClaudeBot as a training crawl. What the three crawl purposes mean.

A robots rule that names Anthropic rather than ClaudeBot also lands on Claude-User (query time), Claude-SearchBot (search index), which does a different job. Blocking the wrong one of a vendor's agents is the commonest mistake in this whole subject.

Should I block ClaudeBot?

A robots rule naming only anthropic-ai is assumed to still be doing its job, when the crawl now arrives under ClaudeBot and the old token matches nothing.

To block it, add this to robots.txt (how the rules resolve):

User-agent: ClaudeBot
Disallow: /

Replace Disallow: / with Allow: / to permit it explicitly. A crawler that ignores robots.txt is not blocked by it either way — that is what verification and edge rules are for.

How do I verify ClaudeBot is genuine?

Anthropic lets you check a request claiming to be ClaudeBot against published IP ranges. A user-agent header on its own proves nothing.

Its address list lives at https://claude.com/crawling/bots.json.

The verification procedure explains why a reverse lookup has to be forward-confirmed, and why a published range list has to be re-fetched rather than pinned.

How do I find ClaudeBot in my logs?

Match the user-agent against "claudebot" or "anthropic-ai", case-insensitively. Nothing comes back from a training fetch, so keep ClaudeBot out of any count you use to judge Anthropic as a source of readers.

Crawl-to-referral gap defines the ratio, and the rate-limiting guide covers what to do when the volume itself is the problem.

Registry last reviewed 2026-09-13.