CDN edge

The CDN edge is where requests arrive before reaching your origin. It is also the only place where crawler traffic can be separated from human traffic without touching your application.

Why route crawler traffic at the edge?

Because it is the one point that sees every request and can branch on it. A small piece of logic there can send verified AI crawlers to a monetization service while ordinary human requests continue untouched to origin.

Does this slow the site down?

For ordinary visitors, no. Their requests take the normal path, keeping caching, DDoS protection and firewall rules exactly as configured. Crawler requests take the additional hop, and no person is waiting on those. A publisher may also route readers arriving from an AI answer through the same hop, which is off by default; that is a small, high-value slice of traffic rather than the bulk of it, so the cache behaviour the CDN depends on is unchanged either way.

What happens if the monetization service is unavailable?

The request falls through to the origin and the crawler receives the real page. Failing open is essential: a monetization outage must never become an availability outage.

What can an edge runtime actually do?

Less than an application server and more than a cache rule. Edge workers run under tight CPU and memory limits, close to the visitor, with no persistent local state and no long blocking calls. That shape suits request classification precisely: read a header, check a cached verdict, decide where the request goes. It does not suit anything that needs a database round trip on the critical path, which is why verification results have to be computed elsewhere and cached rather than looked up per request.

Why is this the only place the decision can be made?

Because it is the only layer that sees every request before the application does and can branch without a code change in the application. Doing the same work in your CMS means a plugin per platform, a deployment per site and a performance cost on every page render. Doing it in DNS is too coarse: DNS resolves a name, not a request, so it cannot tell a crawler from a reader at all.