AI crawler

The label covers three quite different jobs, and conflating them is the most common mistake publishers make. One page can be visited for all three reasons inside a single week.

What are the three kinds?

Training crawlers such as GPTBot and ClaudeBot gather text for a model that ships months later. Nothing you publish today changes an answer today, and the value returns to you slowly if at all.

Search-index crawlers such as OAI-SearchBot and PerplexityBot maintain the corpus an answer engine draws on. One visit can shape what gets said about your topic for weeks afterwards.

Query-time crawlers such as ChatGPT-User, Perplexity-User and Claude-User arrive while somebody waits. Latency is the constraint, the fetch is usually shallow, and the worth is immediate.

Why does the distinction matter commercially?

Because the three are priced nothing alike. CrawlClick never monetizes training collection, prices index maintenance modestly, and treats a fetch made while a human waits as premium inventory.

Does one robots.txt rule cover all of them?

No, and assuming it does is how publishers end up half-blocked. Directives attach to agent names, so disallowing GPTBot leaves ChatGPT-User entirely free to keep fetching — either exactly what you intended, or an unpleasant surprise.

How do I tell them apart?

By user-agent first, then verification. A -User suffix usually signals a live request, though the header is just a string anyone can send, so the claim has to be confirmed by reverse DNS, published address ranges or a signed request before you count it or bill it.

What do these visits cost me?

Bandwidth and origin CPU, billed to you at your hosting rates whether or not anything comes back. An impostor consumes the same resources with none of the upside, which is why unverified bots get the ordinary page from CrawlClick and are never billed to anyone.