robots.txt

robots.txt is a text file at your site root that tells crawlers what they may fetch. AI vendors generally honour it and increasingly publish separate agent names so it can be used selectively.

Can I allow AI search but block AI training?

Usually yes, because vendors use different agents for the two jobs. Disallowing GPTBot while allowing OAI-SearchBot and ChatGPT-User blocks training collection while keeping you present in answers. Check each vendor’s documentation, since the names change.

Is robots.txt enforceable?

Not technically. It is a request that well-behaved crawlers honour voluntarily. Some crawlers ignore it entirely, which is one reason verification at the edge matters more than the file alone.

Does blocking crawlers protect my traffic?

It protects your content from being read, which is not the same thing. If your audience increasingly arrives via AI answers, being absent from those answers can cost more than being read for free.

How does a crawler decide which rules apply to it?

It matches its own product token against the User-agent lines, case-insensitively, and obeys the groups that name it. If none do, it falls back to the wildcard group. The consequence that catches people is that a named group replaces the wildcard rather than adding to it, so every restriction you wrote under * has to be restated inside each agent-specific group or it does not apply to that agent at all.

Within the chosen group, the longest matching path wins by character count rather than by the order the lines were written in, and a tie between an allow and a disallow resolves in favour of the allow.

Which names in a robots file are not crawlers?

Some tokens are controls with no fetcher behind them. Google-Extended governs whether already crawled content may train Gemini, and Applebot-Extended does the same for Apple’s models. No request ever arrives carrying either name, so a rule against one will never appear in your logs — which is exactly why publishers conclude it did not work and delete it.

How long does a change take to bite?

Longer than a deploy. Crawlers are permitted to reuse a cached copy of the file, and vendors document intervals from about a day to a month. A rule added this morning and judged by this week’s logs will usually look as though it is being ignored when it is simply not in force yet.

What cannot be expressed here at all?

How much of a page may be reproduced. This file governs access, not display, and the directives that limit snippet length or exclude an element from a preview live in a page-level meta tag instead. It also reaches nothing already collected: a rule added today is prospective, and no wording removes a page from a model that has already trained on it.