AI crawlers · Anthropic

Claude-User

Claude-User is a crawler operated by Anthropic, and it fetches your page while a person waits for an answer — the fact that decides everything else on this page.

What does Claude-User actually do?

Claude-User is what a fetch looks like when somebody in a conversation needs your page: a link they pasted, a search result the assistant opened, a page it reads as one step of a task. It targets URLs rather than sites, so it shows up as scattered individual requests rather than anything resembling a crawl.

Anthropic publishes one address file covering all three of its crawlers. That means an address match confirms the request came from Anthropic but not which agent sent it. If you want to treat live fetches differently from training, the range check alone cannot do it - you have to combine it with the user-agent string and accept that the string is the weaker half of the pair.

One conversation can produce several fetches in quick succession while Claude works through a task, reading a page, following a reference, checking a second source. That resembles a small crawl in your logs without being one, and the same user-agent covers both a person chatting and an automated workflow built on the API.

Anthropic is the exception among vendors running a user-triggered agent. OpenAI, Perplexity and Meta all state in writing that their user-initiated fetchers may not be bound by robots.txt; Anthropic's documentation gives this one an ordinary robots token and claims no exemption for it. Of the live-fetch agents in this registry, this is the one where the standard mechanism is also the documented one.

CrawlClick files Claude-User as a query time crawl. What the three crawl purposes mean.

A robots rule that names Anthropic rather than Claude-User also lands on Claude-SearchBot (search index), ClaudeBot (training), which does a different job. Blocking the wrong one of a vendor's agents is the commonest mistake in this whole subject.

Should I block Claude-User?

It gets swept up in a blanket no-AI rule, which stops the assistant reading a page for somebody who explicitly asked for it and changes nothing at all about training.

To block it anyway, add this to robots.txt (how the rules resolve):

User-agent: Claude-User
Disallow: /

Replace Disallow: / with Allow: / to permit it explicitly. A crawler that ignores robots.txt is not blocked by it either way — that is what verification and edge rules are for.

How do I verify Claude-User is genuine?

Anthropic lets you check a request claiming to be Claude-User against published IP ranges. A user-agent header on its own proves nothing.

Its address list lives at https://claude.com/crawling/bots.json.

The verification procedure explains why a reverse lookup has to be forward-confirmed, and why a published range list has to be re-fetched rather than pinned.

How do I find Claude-User in my logs?

Match the user-agent against "claude-user", case-insensitively. Every Claude-User fetch had somebody waiting on it, so this count maps onto real questions being asked about your pages rather than onto crawl budget.

Crawl-to-referral gap defines the ratio, and the rate-limiting guide covers what to do when the volume itself is the problem.

Registry last reviewed 2026-09-13.