Guides
How to measure what AI search is doing to your traffic, and what you can do about it.
Where should a publisher start?
Begin with measurement. You cannot decide what to do about AI crawlers until you know how often they read you and how little they return.
The four guides below run in order, and each one produces something the next one needs: a list of agents hitting your origin, a ratio between fetches and visits, pages written so a fragment survives being lifted out, and finally a decision about payment. Skipping to the last one without the first two means negotiating without knowing what you are selling.
- How to see which AI crawlers visit your site
Your analytics tool counts visitors and ignores crawlers entirely, which is why AI traffic loss looks like an unexplained decline. Here is how to measure it.
- How to write a robots.txt for AI crawlers
Group selection, token fallback and propagation delay decide what your rules actually do. Here is how the file resolves, and which agents say it does not bind them.
- Why your traffic dropped when AI search arrived
A decline with steady rankings and no algorithm update usually has one cause analytics cannot show you. Here is how to confirm it from your own request logs.
- How to rate-limit AI crawlers without blocking them
When crawl volume is the problem, a Disallow is the wrong tool. Here are the mechanisms that slow a crawler while keeping you readable, and what each costs.
- How to measure your crawl-to-referral gap
Crawl counts alone prove nothing. Comparing crawls against referrals from the same engine turns a vague sense of decline into a number you can act on.
- How to make your content AI-search friendly
Answer engines retrieve fragments, not pages. Five structural changes make your writing far more likely to be quoted and cited.
- Why an AI engine cited a competitor instead of you
Citation is decided in four stages, and losing at any one of them looks identical from outside. Here is how to find the stage you lost, and how to fix it.
- How to handle AI crawlers on paywalled content
Serving a crawler the full article while a reader hits a wall is what search engines call cloaking. Here is the documented line, and where AI vendors are silent.
- How to monetize AI crawler traffic
Blocking crawlers ends the exchange rather than repairing it. Monetizing the crawl keeps you present in answers while restoring compensation.
- Will sponsored passages for AI crawlers hurt SEO?
The search risk in serving crawlers anything is deception, not advertising. Here is what search engines actually penalise, and how to watch for trouble.
What should an advertiser read?
Start with how brands get named in answers at all, since that determines what is worth buying.
Budget questions come second and resolve quickly: campaigns are funded up front, clear at a second-price auction, and stop dead when the money runs out.
- How to appear in AI answers as an advertiser
There is no ad slot inside an AI answer. Here is what actually determines whether a brand gets named, and what can be bought.
- How to measure your brand's presence in AI answers
AI answers have no impressions log and no click, so brand presence has to be sampled by asking the engines. Here is the method, and what it cannot tell you.
Which guide do both sides need?
Verification, because every other number on this site rests on it. A publisher who counts forged headers inflates their own reporting; an advertiser who pays against them funds scrapers.
- What AI crawlers see when your page needs JavaScript
Two vendors document that their crawler renders. The rest say nothing at all. Here is how to find out what an agent actually got, using only your own request logs.
- How to verify an AI crawler is genuine
User-agent strings are trivially forged. Here are the three methods that actually prove a crawler is who it claims to be, in order of strength.
What do the guides assume you have?
Access to your own server or CDN logs, and nothing else. No account with us, no signature in a template, no change to how your pages are written. The first three can be worked through by a reader who never intends to sell anything, and they are worth reading on that basis alone.
Where do the definitions live?
Two reference sections sit beside these walkthroughs and answer different questions. The glossary explains a concept once, in a paragraph you can quote. The crawler directory is a lookup table: one entry per known agent, naming the company behind it, the job it does, and where that vendor publishes the addresses it fetches from.
How current is any of this?
Every guide carries the date it was last checked. Vendors rename agents, publish fresh address files and add new products without notice, so a guide naming specific bots goes stale faster than one describing a method. Where the two conflict, trust the method.