How to verify an AI crawler is genuine

Any script can send User-Agent: GPTBot. Treating that header as proof means your crawl statistics include scrapers, and any monetization built on it can be defrauded.

Method one: published address ranges

Most vendors publish the IP ranges their crawlers use as JSON. Check the request’s source address against the current list. This is simple and fast, but the lists rotate, so a cached copy goes stale and starts producing false negatives.

Method two: forward-confirmed reverse DNS

Reverse-resolve the source IP to a hostname, confirm the hostname ends in the vendor’s domain, then forward-resolve that hostname and confirm it maps back to the original IP. Both directions are required. Reverse DNS alone can be set by whoever controls the address.

1.2.3.4  →  crawl-1-2-3-4.googlebot.com  →  1.2.3.4   ✓ verified
1.2.3.4  →  host.attacker.example                     ✗ wrong domain
1.2.3.4  →  crawl.googlebot.com  →  5.6.7.8           ✗ does not map back

Method three: signed requests

Newer vendors sign requests cryptographically using HTTP message signatures. This is the strongest proof because it does not depend on network-level identity at all, and it is the direction the industry is moving.

What should happen when verification fails?

Serve the ordinary page and record the request as unverified. Do not block outright, since a false negative would remove you from a legitimate engine’s index, and never bill anyone for it.

Which method should I use?

All three, strongest first. Signature if present, then reverse DNS, then address ranges. Treat the result as a confidence level rather than a boolean, and reserve anything commercially meaningful for the verified tier.

Which vendors publish what?

The three methods are not evenly available, and the gaps decide how much weight a verified label can carry for a given agent.

Operator	Published address list	Reverse DNS namespace
Google	Five JSON files split by crawler category	googlebot.com, google.com, googleusercontent.com
Microsoft	bingbot.json, plus a single-address lookup tool	search.msn.com
OpenAI	One file per agent	Not published
Anthropic	One file covering all three agents	Not published
Apple	applebot.json	applebot.apple.com
Common Crawl	ccbot.json	crawl.commoncrawl.org
Amazon	A list of individual addresses	Not documented
DuckDuckGo	duckassistbot.json, plus an enumerated address list	Not published
Meta, Mistral, You.com, Cohere	None	None

That last row is the important one. MistralAI-User, YouBot, cohere-ai and the two Meta agents have no verification route at all: no address file, no hostname suffix, no signature. A request carrying one of those user-agent strings is a claim with nothing behind it, and Mistral’s registry entry does not even point at a crawler reference — it points at the general developer documentation, because there is no crawler page to point at. Any rule that treats verified traffic differently should put these agents in the unverified tier and leave them there until a vendor publishes something to check against.

Why does one file per agent matter?

Because it lets you enforce a distinction that robots.txt only requests. OpenAI publishes a separate address file for each of its three agents, so you can allow the search crawler’s ranges and drop the training crawler’s at the edge, and neither decision depends on a header anyone can forge. Anthropic publishes one file covering all three of its crawlers, which confirms that a request came from Anthropic but not which agent sent it — so separating live fetches from training there still rests on the user-agent string, which is the weaker half of the pair.

What exactly does Google’s procedure say?

Four steps, and skipping the last one is the usual mistake. Run a reverse DNS lookup on the address from your logs. Check that the name ends in googlebot.com, google.com or googleusercontent.com. Run a forward lookup on that name. Confirm it returns the address you started with. The third and fourth steps exist because a reverse record can be set by whoever controls the address block; only the round trip proves the vendor controls the name.

How often should the lists be refetched?

Often enough that a rotation does not read as an intrusion. Address ranges change without notice, and a pinned copy produces false negatives that look exactly like a scraper: familiar user-agent, unfamiliar address. Fetch on a schedule, keep the previous copy until the new one parses, and alert on a file that stops resolving rather than silently falling back to an old one.

Common Crawl is worth singling out here, because it says the quiet part aloud: its documentation warns that crawlers falsely identifying themselves as CCBot exist. Every vendor could write that sentence. Most do not.

DuckAssistBot is the other outlier, in the opposite direction from what its size would suggest. DuckDuckGo publishes the addresses its AI-answer fetcher crawls from both as a JSON file and as an enumerated list, states a 72-hour window for a robots change to take effect, and says the content is not used to train AI models. Three vendors many times its size document none of those things.

What does verification not tell you?

Whether the fetch was useful. A verified agent may be a training crawler that will never send you a reader, or a query-time fetch that leads to a citation, and the verification step treats them identically. It also says nothing about whether the page it received was complete — if your content assembles itself in the browser, a perfectly verified crawler may have received an empty shell. Identity and value are separate questions, and only the first one has a clean answer.