Query-time crawl

A query-time crawl happens because someone just asked a question. The engine fetches your page, reads it, and composes an answer while the person waits.

Why is query-time inventory more valuable?

Because the outcome is immediate and attributable. An index crawl might influence answers at some point; a query-time crawl influences the answer being written this second, for a person with an active question. Intent is at its highest.

How can I recognise one?

Vendors typically use a distinct agent string, often with a -User suffix: ChatGPT-User, Perplexity-User, Claude-User. Some also pass a referrer that reveals the originating query, which allows the content served to be matched to what was actually asked.

Are query-time crawls priced differently?

Yes. Query-time crawls clear at a higher floor than index crawls, and training crawls are never monetized. Pricing follows the value of the moment rather than treating all bot traffic alike.

What does a query-time crawl look like in a log?

Isolated single requests rather than sequences. There is no traversal, no sitemap fetch and no predictable schedule, because nothing chose the URL except the question being asked. Arrivals cluster around whatever is being discussed rather than spreading evenly across an archive, so a sudden group of fetches on three related pages is the normal shape of this traffic rather than an anomaly.

Why does latency matter more here than anywhere else?

Because a person is waiting and the engine has alternatives. Every other crawl in the registry can be slow, deferred or retried tomorrow with no visible consequence. A query-time fetch that times out does not produce an error anyone sees; it produces an answer assembled from somebody else’s page. A bot challenge, a consent interstitial or a cold cache all have the same effect as a block at this moment, usually without the publisher intending any of them.

Are these fetches bound by robots.txt?

Several vendors have published that they are not, on the grounds that a person initiated the request. That is documented rather than alleged, and it changes what a rule can achieve: stopping these fetches has to happen where you can enforce it rather than where you can request it.

How does this change what a page should contain?

It puts a premium on being complete at the moment of asking. An index crawl can afford to catch up with a page later; a query-time fetch reads whatever is served in that second. Pages that assemble themselves after load, or that hide their substance behind an interaction, are read as whatever the first response contained.