How to measure your crawl-to-referral gap

A crawl count on its own is not persuasive. Engines have always crawled. What changed is how little they return, and that only appears when you measure both directions at once.

What is the gap exactly?

Crawls divided by visitors returned, calculated for each engine separately. Traditional search typically sits near five. AI answer engines routinely exceed one hundred, and some return nothing measurable at all.

Why measure per engine rather than in aggregate?

Because the aggregate hides the finding. Google’s healthy ratio averages out Perplexity’s absent one, and the resulting middle number tells you nothing. Separated, the contrast is the whole story.

How do I attribute a referral to an engine?

By the referrer on the human visit. chatgpt.com, perplexity.ai, gemini.google.com and claude.ai all identify themselves this way. Map them onto the same provider as the crawler so GPTBot crawls and chatgpt.com visits appear on one row.

What if the referrer is missing?

Some AI clients strip it. Where they do, a UTM parameter on the outbound link is often present instead. Where neither exists the visit is uncountable, which means most gap measurements understate the problem rather than overstating it.

What period should I measure?

At least a month. Crawl volume is uneven, and index crawls arrive in bursts that make a single week misleading in either direction.

What does a bad result look like?

An engine crawling tens of thousands of times and returning single-digit visitors. At that point the content is doing the work of answering while receiving nothing back, and the question becomes what to do about it rather than whether it is happening.

How do I build the table?

Two columns joined on one key, and the key is the provider rather than the agent. Count verified crawler fetches per provider for the period, count human sessions whose referrer belongs to that same provider, and divide. GPTBot, OAI-SearchBot and ChatGPT-User all roll up to OpenAI on the crawl side; chatgpt.com referrals roll up to OpenAI on the visit side. Anything finer than provider produces rows you cannot interpret, because a training fetch was never going to send a visitor and counting it as a miss overstates the gap.

Then keep one extra column: distinct URLs touched. A thousand fetches of one page is a very different fact from a thousand fetches across an archive, and the raw total hides which of the two you are looking at.

Why does verification belong in this measurement?

Because unverified traffic inflates the numerator and nothing else. Scrapers borrow the well-known agent names precisely because they are well known, so an unfiltered count of requests claiming to be GPTBot measures your popularity with scraper authors rather than with OpenAI. Run the verification check first, and report the verified and unverified counts as separate lines — the ratio between those two is independently interesting.

What should I do about the pages that are not being crawled?

Look at them before you look at the ratio. A gap calculated over an archive an engine has barely touched is a statement about coverage, not about compensation. Cross-reference the crawled URLs against your sitemap: if a whole section is missing, the question is why it was not fetched, and the answer is usually a robots rule, a redirect chain, or a page whose substance only exists after JavaScript runs.

What are the honest limitations of this number?

Three, and they all push in the same direction. Referrers are stripped by some clients, so visits are undercounted. Answers that satisfy the reader produce no visit by design, so the engine may have served you well and sent nothing. And no engine publishes how often an answer citing you was shown, so the denominator you actually want does not exist. The figure is a floor on the problem rather than a measurement of it.

What is it good for, then?

Comparison and argument. The ratio is consistent enough over time that a change in it means something, and different enough between providers that the contrast is the finding. It is also the one number that turns a vague complaint into a specific one when you are deciding whether to block, to charge, or to monetize the crawl.