How to write a robots.txt for AI crawlers

Most robots files written for AI crawlers do something slightly different from what their author intended. The reason is almost never a typo. It is that robots.txt resolves by rules nobody reads twice: a crawler picks exactly one group, specific groups never merge with the wildcard, and several of the agents people most want to control have published in advance that they will not obey the file at all.

How does a crawler choose which group to obey?

RFC 9309 is short and unambiguous on this point. A crawler compares its own product token against every User-agent line, case-insensitively. If one or more groups name that token, those groups and only those groups apply, merged together. If none do, the * group applies. If there is no * group either, nothing applies and the crawler may fetch anything.

The consequence that catches people is the middle step. A specific group replaces the wildcard rather than adding to it. Google states this directly in its own documentation: user-agent specific groups and global groups are not combined. So a file that says Disallow: /private/ under * and then adds a courteous User-agent: GPTBot / Allow: / group has just granted GPTBot access to /private/, because GPTBot stopped reading at its own group and never saw the wildcard.

What does “most specific match” mean for paths?

Within the chosen group, every Allow and Disallow line is compared against the request path and the longest match by octet count wins. Length, not order — a rule written last does not override one written first. Where an Allow and a Disallow match at exactly the same length, the RFC says the Allow should be preferred, which is why Disallow: /docs/ plus Allow: /docs/ leaves the path reachable rather than blocked.

Two more limits are worth knowing before you write anything long. Parsers are required to read at least 500 kibibytes of the file, and content past that point may simply be ignored. And a crawler is allowed to reuse a cached copy for up to 24 hours, so nothing you write takes effect the moment you save it.

Which tokens actually exist?

Not every name in this space is a crawler. Some are control tokens with no fetcher behind them: no request ever arrives carrying the name, and the rule you write governs how content already collected may be used. Blocking one of these will never show up in your logs, which is exactly why people conclude the rule “did not work” and delete it.

Token	Operator	What a rule against it governs
GPTBot	OpenAI	Collection of text for model training
OAI-SearchBot	OpenAI	Presence in ChatGPT’s search index
ChatGPT-User	OpenAI	Pages fetched during a conversation
ClaudeBot	Anthropic	Collection of text for model training
Claude-SearchBot	Anthropic	Presence in Claude’s search index
Claude-User	Anthropic	Pages fetched during a conversation
Google-Extended	Google	Whether crawled content may train Gemini — not a crawler
Applebot-Extended	Apple	Whether crawled content may train Apple’s models — not a crawler
GoogleOther	Google	Fetches for Google teams outside Search
CCBot	Common Crawl	Inclusion in the open Common Crawl archive

Google is explicit that Google-Extended does not affect a site’s inclusion in Search and is not used as a ranking signal. Apple says the same shape of thing about Applebot-Extended: pages that disallow it can still be included in search results. Both exist precisely so that the training question can be answered separately from the visibility question, and both are wasted on a publisher who blocks the crawler instead.

Why does one rule sometimes cover several crawlers?

Because a crawler can declare more than one product token, and the fallback is to the wildcard rather than to anything that looks like a parent. Google documents Googlebot as a second token for Googlebot-Image, Googlebot-News, Googlebot-Video and — much less obviously — for Google-CloudVertexBot. A group headed User-agent: Googlebot therefore governs all of them unless a more specific group names one directly.

GoogleOther-Image and GoogleOther-Video behave the same way with respect to GoogleOther. Storebot-Google does not declare a Googlebot token at all, and Google’s own worked example shows it falling through to the * group. There is no way to infer this from the names; it has to be read off the vendor’s table.

Cohere’s entry is the same trap in miniature. The registry lists two strings for it, cohere-ai and cohere-training-data-crawler, and a group naming only the first leaves the training crawl entirely unaddressed.

Which agents say robots.txt does not bind them?

Three vendors have published, in advance, that their user-triggered fetchers are not governed by the file the way an indexing crawler is. This is not a rumour about misbehaviour — it is documentation.

• ChatGPT-User — OpenAI writes that because these actions are initiated by a user, robots.txt rules may not apply.
• Perplexity-User — Perplexity writes that this fetcher generally ignores robots.txt rules, for the same reason.
• meta-externalfetcher — Meta writes that it may bypass robots.txt rules because the fetch is user-initiated. Meta adds that its link-preview agent may also bypass the file during security or integrity checks.

If you genuinely want those fetches stopped, the rule has to live somewhere you can enforce it: a user-agent or address rule at your CDN, applied after verification. Writing it in robots.txt and considering the matter closed is the single commonest mistake in this whole subject.

What about the smaller live-fetch agents?

They are the ones most often left out, and the omission is an allow decision rather than a neutral one. Two are worth naming because they sit at opposite ends of the documentation spectrum.

DuckAssistBot is unusually well specified for a small engine. DuckDuckGo publishes the addresses it fetches from, states that a robots change takes effect within 72 hours, says blocking it does not affect organic rankings, and says the content is not used to train AI models. Every part of the decision is documented, which makes it one of the few entries where you can write a rule and know what you traded.

MistralAI-User is the opposite. There is no crawler reference, no address file, no hostname suffix, and — importantly — no statement either way about whether robots.txt binds it. That is a weaker position than an explicit exemption: OpenAI, Perplexity and Meta at least tell you their user-triggered fetchers may ignore the file, which is a signal to go and enforce the rule somewhere you control. Silence leaves you with a lever and no information about whether it is connected to anything.

How long before a rule takes effect?

Longer than you think, and the delay is vendor-specific rather than standard. RFC 9309 permits a 24-hour cache, and several operators document intervals well past that.

Operator	Documented delay before a robots change is respected
OpenAI	About 24 hours from the update
DuckDuckGo	Within 72 hours
Amazon	Fetches host-level robots.txt or uses a cached copy from the last 30 days

That last one deserves a moment. A rule you add for Amazonbot today may go on being ignored for a month, entirely legitimately, before it starts to bite. A publisher who adds a rule, watches the logs for a week and concludes the crawler is defying them has drawn the wrong conclusion from real evidence.

What about crawl-delay?

Crawl-delay is not part of RFC 9309, and support for it is genuinely split. Anthropic documents that it respects the directive as a non-standard extension. Amazon documents the opposite in as many words: its agents do not support it. Google has never supported it. So a single Crawl-delay line does not mean what a single Disallow line means — it is a request that one vendor honours, another has declined in writing, and most simply drop. If the volume rather than the reading is your problem, rate limiting is the mechanism that actually applies.

What can robots.txt not do?

It cannot enforce anything. Every one of these behaviours is voluntary, the file has no authentication, and a crawler that ignores it is not blocked by it in any sense. It cannot express “index me but do not summarise me” — that is the job of page-level snippet directives, which are a different mechanism with a different syntax. It cannot reach content already collected: a block is prospective, and nothing you write removes text from a model that has already trained on it or from an archive already published.

It also cannot be reasoned about from names alone. Two of the tokens in the table above are not crawlers, three of the agents publishers most want to control have said the file does not bind them, and one vendor’s flagship token silently governs four other agents. The file is worth writing carefully, and worth re-reading against the crawler reference once a quarter, because the vendors keep changing what the names mean.

What should a publisher actually do?

Decide the training question and the visibility question separately, because they are controlled by different tokens and the trade is different in each direction. Write a * group that states your baseline. Then write a named group for every agent whose treatment differs from that baseline — and remember that each named group has to restate everything, because it replaces the wildcard rather than extending it. Finally, check the file against a list of tokens rather than against your memory of writing it. The tokens that are missing are never the ones you were thinking about.