How to rate-limit AI crawlers without blocking them

There are two very different complaints hiding behind “the AI bots are hammering us”. One is about consent — you object to the reading. The other is about capacity — you do not mind being read, you mind the origin falling over at 3am. Only the first is a job for a disallow rule. The second has its own set of levers, most of which are in HTTP rather than in robots.txt, and several of which have documented side effects worth knowing before you pull them.

Is the volume actually the problem?

Measure before you act, because bot traffic is routinely misattributed. Split the request count from the byte count and from the origin-hit count. A crawler taking 40,000 requests a month that all resolve at your CDN edge is costing you almost nothing; four hundred requests a month that each miss cache and trigger a database query may be costing you a great deal more. Rate is the wrong unit — origin work is the right one.

Then split by agent, because the mix decides which lever is appropriate. Bulk index crawls arrive in long sequences across many URLs and nobody is waiting on any individual fetch. A query-time agent like Claude-User or ChatGPT-User opens one page inside somebody’s question and the latency is felt by a person. Throttling the first group costs you patience; throttling the second costs you the citation, because a slow response loses to whatever else the assistant can read in time.

What does robots.txt offer here?

Less than its reputation suggests. Crawl-delay is not part of RFC 9309, and support is genuinely split rather than merely inconsistent. Anthropic documents that it honours the directive as a non-standard extension. Amazon documents, in as many words, that its agents do not support it. Google has never supported it. So the same line in the same file is obeyed, declined and ignored by three different readers, and none of them is misbehaving.

There is a second-order use for the file, though, that has nothing to do with rate. A disallow on the paths that are expensive and worthless — faceted search URLs, calendar pagination, session-parameter variants, print views — removes work rather than deferring it. Most crawl-volume problems turn out to be a crawler enthusiastically exploring a combinatorial URL space nobody meant to publish.

Which vendors let you set a rate directly?

Only one of any size, and not the one people expect.

Operator	Documented control	Where it lives
Microsoft	Crawl Control: an hourly crawl-rate pattern for bingbot, from a preset or a custom curve	Bing Webmaster Tools
Google	No rate setting; an emergency HTTP mechanism plus a report form that takes days and cannot request an increase	Search Console and your own server
Anthropic	Crawl-delay, honoured as a non-standard extension	robots.txt
Amazon	None — crawl-delay explicitly unsupported	—

Bing’s is the genuinely useful one if your load is cyclical: you can tell bingbot to crawl slowly during business hours and make up the ground overnight. Nothing in the AI-native set of agents offers an equivalent dashboard, which is the honest state of the art rather than an oversight you can work around.

What does Google actually tell you to do?

Return errors, briefly. Google’s documented mechanism for urgently reducing crawl rate is to serve 500, 503 or 429 instead of 200 to crawl requests, and it is blunt about the time limit: a couple of hours, or one to two days. Beyond that, Google warns that URLs returning those codes over multiple days may be dropped from the index outright.

That is a real emergency valve and a terrible steady state. Treat it the way you would treat pulling a circuit breaker: correct during an incident, evidence of a design problem if it is still in place next week. For anything ongoing, Google’s other route is a request form that takes several days to process and cannot be used to ask for more crawling later.

Which HTTP mechanisms work on everybody?

The ones that are part of the protocol rather than part of a vendor’s goodwill.

• 429 Too Many Requests with Retry-After. This is the standard way to say “not now, come back in N seconds”. It is a first-class status code with a defined meaning, unlike a silently dropped connection, which a crawler can only interpret as an outage.
• Conditional requests. Serve strong ETag and Last-Modified headers and answer If-None-Match and If-Modified-Since with a 304. The crawler still gets its answer; you send no body. On an archive where most pages have not changed since the last sweep, this is the single largest saving available and it costs you nothing in visibility.
• Cache headers the edge can act on. A crawler request that never reaches your origin is not a capacity problem at all. Most AI agents fetch plain HTML and execute nothing, so they are unusually cache-friendly compared with a logged-in human session.
• An accurate lastmod in your sitemap. Wrong or absent modification dates make a crawler re-fetch a stable archive to find out whether anything changed. Correct ones let it skip.

How should an edge rule be keyed?

On verified identity, never on the claimed user-agent string. A throttle keyed on the header GPTBot punishes whoever sends that string, and the scrapers borrowing the name are precisely the traffic you wanted to slow down — while a forged header lets them evade any allowance you grant. Run the verification check first, then apply a generous budget to the verified agent and a tight one to everything merely claiming to be it. That single ordering change is usually worth more than any number you pick for the limit.

Give the query-time agents their own bucket, set well above the bulk crawlers. The crawler reference marks which agents fetch live, and those are the requests where a delay turns into a lost citation rather than a slightly later index update.

What are the limits of all this?

None of it is enforcement. Crawl-delay is a request, Retry-After is advice, and an agent that ignores robots.txt will ignore a polite status code too. Rate limiting also cannot distinguish between a crawl that will earn you a citation and one that will not, so a blunt limit applied to everything slows down the traffic you wanted along with the traffic you did not.

And a throttle does not answer the consent question at all. If your real objection is that the reading itself is uncompensated, a slower version of the same reading changes nothing. That is a different decision with three genuine options: block, charge for access, or monetize the crawl so that being read is worth something. Capacity is an engineering problem. Compensation is not.

What should you do first?

Cache properly, return 304s, and disallow the expensive URL spaces nobody meant to publish. Those three cost nothing in visibility and usually remove most of the load. Only then reach for a limit, key it on verified identity, and keep the live-fetch agents out of the strictest bucket. Save Google’s error-code valve for an actual incident, and write down the date you opened it.