AI crawlers

22 known agents from 14 companies. The column that matters is what the crawl is for, because that decides what blocking it costs you.

Why does the purpose of a crawl matter?

Because publishers routinely block the wrong one. 4 of these agents collect training data, 11 build the index an assistant answers from, and7 fetch a page live while someone waits on an answer. Blocking a training crawler keeps your text out of the next model and costs you nothing in AI search. Blocking a search or query-time crawler removes you from the answers themselves — and prevents no training at all, because that is a different agent with a different token.

Most companies here run more than one. OpenAI runs three, and they do three different jobs. A single blanket block is almost never the trade a publisher actually wants.

What do the three purposes mean?

Training — collects text to train a model: Blocking it keeps your text out of the next training run. It does not affect whether you appear in AI answers, because answers are built from a different crawl.
Search index — builds the index an assistant answers from: Blocking it removes you from that engine’s answers. Your pages stop being retrievable as a source, so the engine cites someone else.
Query time — fetches your page while a person waits for an answer: Blocking it means the assistant cannot read your page to answer a question someone is asking right now. This is the highest-intent traffic any AI engine sends at your server.

Which crawler belongs to which company?

Anthropic

Claude-Userquery time

Claude-SearchBotsearch index

ClaudeBottrainingnot monetizable

Google

Google-CloudVertexBotquery time

GoogleOthersearch index

Googlebotsearch index

OpenAI

ChatGPT-Userquery time

OAI-SearchBotsearch index

GPTBottrainingnot monetizable

How do I tell a real crawler from a fake one?

You check it rather than believe it. A user-agent header is free to forge, so every agent above is only as trustworthy as the evidence behind it: a signed request, a forward-confirmed reverse DNS name, or membership of a published address range. Each page lists which of those its vendor actually supports — the procedure is here.

Which of these cannot be verified at all?

Five, and they are worth naming because no amount of care at your edge changes their status. MistralAI-User, YouBot, cohere-ai, Bytespider and both Meta agents publish no address file, no hostname suffix and no request signature. A request carrying one of those names is a claim with nothing behind it, and the registry entries for several of them point at a company home page rather than a crawler reference, because there is no crawler reference to point at.

That absence has a practical consequence rather than a rhetorical one. Any rule that treats verified traffic differently — a higher rate allowance, preferential serving, anything billable — has to place these agents in the unverified tier and leave them there. It also means a publisher cannot tell a genuine fetch from a scraper wearing the same name, in either direction, which is the vendor's omission rather than the publisher's.

How current is this list?

It is generated from CrawlClick's own registry rather than hand-copied, so a change in the data plane cannot leave this page quietly wrong, and every entry carries a link to the operating vendor's own documentation so you can check a claim at its source. Vendors still add agents, rename tokens and retire strings without notice — Anthropic's older anthropic-ai token and OpenAI's newer advertising agent are both examples — so the review date below is part of the information rather than a footnote.

Registry last reviewed 2026-09-13.