AI crawlers · Anthropic

Claude-SearchBot

Claude-SearchBot is a crawler operated by Anthropic, and it builds the index an assistant answers from — the fact that decides everything else on this page.

What does Claude-SearchBot actually do?

Claude-SearchBot exists so that Claude's web search has something to search. It crawls to build and refresh the index; Claude-User then opens individual pages mid-conversation. If this agent has never seen a page, the assistant has no reason to know the URL exists, so the query-time fetch that might have cited you never gets triggered.

It is the youngest of Anthropic's three tokens and the one most often missing from robots files. A rule that names ClaudeBot and stops there leaves this agent unaddressed - fine if allowing it was the intention, a silent gap if it was not. Check the file against all three names rather than against your memory of writing it.

Treat it operationally like a search-engine crawler: sitemaps, stable canonicals, the same response for every user agent. Serving this crawler a different page from the one Claude-User later fetches produces citations that do not match what the reader sees, which damages you more than never being cited would have.

Anthropic is unusually direct about the cost of a block, which most vendors leave you to infer: its documentation describes disallowing this agent as preventing indexing and reducing visibility in search results. All three Anthropic tokens are covered by that one page, so there is no separate reference to go and miss - which makes a robots file naming only one of them harder to excuse than it looks.

CrawlClick files Claude-SearchBot as a search index crawl. What the three crawl purposes mean.

A robots rule that names Anthropic rather than Claude-SearchBot also lands on Claude-User (query time), ClaudeBot (training), which does a different job. Blocking the wrong one of a vendor's agents is the commonest mistake in this whole subject.

Should I block Claude-SearchBot?

The name contains Claude, so it gets the same rule as ClaudeBot - trading away search visibility to stop training that this agent never did.

To block it anyway, add this to robots.txt (how the rules resolve):

User-agent: Claude-SearchBot
Disallow: /

Replace Disallow: / with Allow: / to permit it explicitly. A crawler that ignores robots.txt is not blocked by it either way — that is what verification and edge rules are for.

How do I verify Claude-SearchBot is genuine?

Anthropic lets you check a request claiming to be Claude-SearchBot against published IP ranges. A user-agent header on its own proves nothing.

Its address list lives at https://claude.com/crawling/bots.json.

The verification procedure explains why a reverse lookup has to be forward-confirmed, and why a published range list has to be re-fetched rather than pinned.

How do I find Claude-SearchBot in my logs?

Match the user-agent against "claude-searchbot", case-insensitively. Set the Claude-SearchBot fetch count beside the visits Anthropic actually returns; that ratio is your crawl-to-referral gap for this engine.

Crawl-to-referral gap defines the ratio, and the rate-limiting guide covers what to do when the volume itself is the problem.

Registry last reviewed 2026-09-13.