How to see which AI crawlers visit your site
Your analytics tool counts visitors and ignores crawlers entirely, which is why AI traffic loss looks like an unexplained decline. Here is how to measure it.
Most publishers cannot answer a simple question: how often does ChatGPT read my site? Analytics tools count visitors, and a crawler is not a visitor, so the entire category is invisible.
Why doesn’t Google Analytics show this?
Because JavaScript analytics only fire when a browser executes a page. AI crawlers fetch the HTML and leave without running scripts, so they never register. A site can be read a hundred thousand times a month and show nothing at all.
Where does the data actually live?
In your server or CDN request logs. Every crawler request appears there with a user-agent string identifying the bot. The information exists; it is simply not in the place people look.
Which agents should I look for?
Group them by what they do rather than by vendor, because the jobs have different value:
- Training:
GPTBot,ClaudeBot,CCBot,Bytespider - Search index:
OAI-SearchBot,PerplexityBot,Applebot,Amazonbot - Query-time:
ChatGPT-User,Perplexity-User,Claude-User,DuckAssistBot
The query-time group matters most, because each of those fetches means a person was waiting.
How do I count them from a log file?
If you can export raw logs, a single pass gets you the shape of it:
grep -oiE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User" \
access.log | sort | uniq -c | sort -rnOn Cloudflare, the same breakdown is available through Logpush without touching your origin.
Why should I not trust the user-agent alone?
Because anyone can set one. A meaningful count requires confirming the claim by reverse DNS or the
vendor’s published address ranges. Without verification, a scraper pretending to be GPTBot
inflates your numbers and distorts every conclusion you draw.
What should I do with the numbers?
Compare them against referrals from the same provider over the same period. That ratio, rather than the raw crawl count, is what shows whether the exchange is still working.
What do I need from each log line?
Six fields, and most access-log formats already carry them: timestamp, source address, user-agent, request path, response status and bytes sent. Add the cache status if your CDN exposes it, because that single field tells you whether the crawl is costing you origin work or being served for nothing at the edge, which changes the answer to almost every follow-up question.
The source address matters as much as the user-agent, because it is the only half of the pair that can be checked. Logging the agent string and discarding the address leaves you with a count you cannot verify, and an unverifiable count of a name anyone can forge is not a measurement.
How do I normalise the agent names?
Carefully, because several of them overlap. Googlebot as a substring also matches Googlebot-Image,
Googlebot-News and Googlebot-Video, so a naive count folds four crawls into one. GoogleOther has
image and video variants under the same family name. Anthropic’s training crawl also arrives under
the older anthropic-ai string on some traffic. Cohere has two separately named agents in one
registry entry. Meta writes its agents with internal capitals while the strings arriving in logs are
usually lowercase, so a case-sensitive match silently misses most of them.
Match case-insensitively, match the most specific name first, and keep a bucket for anything that looks like a bot and matches nothing — that bucket is where next quarter’s new agent shows up.
What if I cannot get raw logs?
Most CDN dashboards expose a bot or crawler breakdown that is good enough to see the shape, even when it will not let you export individual requests. It is usually sampled and usually groups agents by its own taxonomy rather than yours, so treat the numbers as proportions rather than totals. That is still enough to answer the first question, which is whether the volume is large enough to be worth a proper measurement.
What should I look at besides the totals?
Distinct URLs, and the ratio of origin hits to edge hits. A thousand fetches of your home page is a crawler checking for changes; a thousand fetches across a thousand URLs is an archive being read. And if most of those requests miss cache, you have a capacity question as well as a compensation one, which is worth separating before you decide what to do about either.