AI search glossary
The vocabulary of AI search, defined plainly. 23 terms covering who reads your site, how it is measured, and how the money moves.
Why does this vocabulary need writing down?
Because most of it was coined in the last two years by companies with no reason to agree with each other, and several of the words already mean two things. A crawler, an agent and a fetcher are used interchangeably by vendors who classify their own products differently. Cloaking has a precise definition in a search engine's spam policy and a much looser one in ordinary conversation. Share of answer is measured by whoever is quoting it, using a question set they chose themselves.
Each entry below states one thing in a paragraph you could quote without the rest of the page. Where a term describes something a vendor documents, the entry says what that documentation actually says; where it describes something nobody has standardised, it says that too, because a definition that hides the disagreement is worse than none.
Which of these should I read first?
Start with the three that decide every other question: what an answer engine is, what RAG does to your page on the way to becoming an answer, and how a document is broken into passages by chunking. Almost every practical recommendation in the guides follows from those three mechanisms, and the recommendations are much easier to argue with once you know which one they rest on.
Who is reading my site?
- AI crawler
- A bot operated by an AI company that fetches web pages to train a model, build a search index, or answer a user's question in real time.
- Crawler verification
- Confirming that a bot claiming to be a particular AI crawler genuinely is one, since user-agent strings are trivially forged.
- GPTBot
- OpenAI's training crawler, which collects web content used to improve its models rather than to answer any specific user question.
- Query-time crawl
- A page fetch made by an AI engine while a user waits for an answer, as opposed to bulk indexing done in advance.
How is any of this measured?
- AI referral signal
- A directional indication that a human browser may have reached a publisher from an AI answer; useful for measurement, but it cannot prove identity or bill.
- Answer engine
- A system that responds to a question with a synthesised answer rather than a list of links, typically by retrieving live web pages and summarising them.
- Crawl-to-referral gap
- The ratio between how often an AI engine reads a site and how many visitors that engine sends back, measured per engine.
- Share of answer
- How often a brand appears in AI-generated answers across a fixed set of questions, measured relative to the competitors named in the same answers.
- Surfacing
- When content placed on a page actually appears in an AI engine's answer, whether quoted directly or paraphrased.
- Traceable marker
- A unique token placed inside sponsored content so its appearance in an AI answer can be attributed unambiguously.
What makes content usable by an answer engine?
- Chunking
- Splitting a document into smaller passages so a retrieval system can fetch and quote the relevant part rather than the whole page.
- Cloaking
- Serving different content to a crawler than to a human visitor, usually to obtain ranking or placement that the real page would not earn.
- Generative engine optimization
- The practice of structuring and writing content so that AI answer engines are more likely to retrieve it, quote it and cite the source.
- Prompt injection
- Text placed in a document to manipulate a language model's behaviour rather than to inform the reader, such as instructions hidden in the copy.
- Self-contained content
- Writing where each passage makes sense on its own, without depending on earlier paragraphs for context a retrieved fragment will not carry with it.
How does the money work?
- Disclosed sponsorship
- Paid content that states plainly that it is paid, with the label bound into the same sentence as the claim so the two cannot be separated.
- Payout threshold
- The minimum accrued balance a publisher must reach before a payment is issued; anything below it carries forward rather than being lost.
- Prepaid budget
- An advertising balance funded in advance, where delivery stops when the funded amount is exhausted regardless of the configured budget.
- Revenue share
- The proportion of an advertiser's payment passed to the publisher whose page carried the placement, with the remainder retained by the marketplace.
- Second-price auction
- An auction where the winning bidder pays just enough to have beaten the runner-up, rather than the full amount they bid.
What sits in the request path?
- CDN edge
- The layer of a content delivery network closest to the visitor, where requests are handled before reaching the origin server.
- RAG
- Retrieval-augmented generation, where a model fetches relevant documents and uses them as context when composing an answer.
- robots.txt
- A file at the root of a site that tells crawlers which paths they may fetch, and which named agents are allowed or disallowed.
What is deliberately not defined here?
Anything that would need a number nobody can check. There is no entry giving a typical crawl-to-visit ratio, a going rate for a placement, or a percentage for how much traffic AI search has taken, because those figures do not exist in a verifiable form. Where a number would help and we do not have one, the entry describes the mechanism instead and leaves the arithmetic to your own logs.
How do the categories above relate to each other?
They follow the path a page takes. Something crawls it, which is the first group. What it did is measured, which is the second. Whether the passage it took was usable depends on how the content was written, which is the third. The fourth covers what changes hands when that passage carries a disclosed sponsored claim, and the fifth covers the layer in the request path where the decision is made at all. Read in that order the vocabulary is one process rather than five lists.
Where do the crawler names live?
In the crawler referencerather than here. A glossary entry explains a concept once; the reference is a lookup table with one page per agent, naming the operator, the job the crawl does, the robots token, and where that vendor publishes the addresses it fetches from. Two terms above —GPTBot andAI crawler — sit in both places on purpose, because they are asked about as concepts as often as they are looked up as entries.