Crawler verification

Crawler verification confirms that a bot is what it says it is. A user-agent header is a claim anyone can make, so it cannot be trusted on its own.

How is a crawler verified?

Three methods, in increasing strength. Published address ranges, where the vendor lists the IPs its crawler uses. Forward-confirmed reverse DNS, where the IP resolves to a vendor hostname that resolves back to the same IP. And signed requests, where the crawler cryptographically proves its identity.

What happens when verification fails?

The traffic is treated as unverified. CrawlClick serves it the ordinary page and never bills anyone for it, because a spoofed crawler must not be able to manufacture a charge.

Why does this matter for advertisers?

It is the structural defence against click fraud’s equivalent in this market. If anyone could set a user-agent header and generate billable impressions, the inventory would be worthless.

What does verification cost in practice?

A DNS round trip, or a lookup against a cached address list. Neither is free on the critical path of a request, which is why verdicts are cached against the source address rather than recomputed per fetch: a crawler arrives many times from the same address, and the identity of that address does not change between one request and the next. Refreshing the published lists on a schedule and caching the result is what keeps the check from becoming the latency it was meant to protect you from.

Should an unverified request be blocked?

Usually not. A false negative — a genuine crawler whose address rotated out of a list you cached too long — looks exactly like a scraper, and blocking it removes you from an engine’s index for reasons you will never see in a report. Serving the ordinary page and recording the request as unverified keeps the mistake cheap. Reserve blocking for volume problems and anything commercially meaningful for the verified tier.

Is verification a yes or a no?

Better treated as a confidence level. A signed request is stronger evidence than a forward-confirmed hostname, which is stronger than membership of a list you fetched this morning, which is stronger than a user-agent header that matches. Collapsing that into a boolean throws away the information that tells you how much weight a given fetch can carry.