AI crawlers · Google

Googlebot

Googlebot is a crawler operated by Google, and it builds the index an assistant answers from — the fact that decides everything else on this page.

What does Googlebot actually do?

Googlebot is the AI crawler nobody lists as one. What it indexes is what AI Overviews and AI Mode draw on, and there is no separate agent to block for those surfaces. The same fetch serves the ordinary results and the generated summary sitting above them, which is why the usual block-it-or-not framing does not fit here.

The controls that do fit are page-level rather than robots-level. Google's documented levers - nosnippet, max-snippet and data-nosnippet - limit how much of a page may be shown in a preview, and that is the mechanism constraining what can surface in a generated answer. Blocking Googlebot itself removes you from Search entirely, which almost nobody actually wants.

One log caveat: the string you match on also appears in Googlebot-Image, Googlebot-News and Googlebot-Video, so a naive substring count folds several crawls into one figure. Split them before you compare crawls against referrals, or heavy image crawling of a photo archive will read as interest in your articles.

The Googlebot token also reaches further than the Googlebot name does. Google lists it as a second user-agent token for Googlebot-Image, Googlebot-News, Googlebot-Video and, far less obviously, for Google-CloudVertexBot, so a group headed User-agent: Googlebot governs every one of them unless a more specific group names that crawler directly. Agents that do not declare the token - GoogleOther and Google-Extended among them - fall through to the wildcard group instead. One rule can cover more than you meant and less than you assumed at the same time.

CrawlClick files Googlebot as a search index crawl. What the three crawl purposes mean.

A robots rule that names Google rather than Googlebot also lands on Google-CloudVertexBot (query time), GoogleOther (search index), which does a different job. Blocking the wrong one of a vendor's agents is the commonest mistake in this whole subject.

Should I block Googlebot?

Publishers block Googlebot to stay out of AI Overviews, which removes them from Google Search altogether while the snippet directives that actually govern AI previews sit unused.

To block it anyway, add this to robots.txt (how the rules resolve):

User-agent: Googlebot
Disallow: /

Replace Disallow: / with Allow: / to permit it explicitly. A crawler that ignores robots.txt is not blocked by it either way — that is what verification and edge rules are for.

How do I verify Googlebot is genuine?

Google lets you check a request claiming to be Googlebot against reverse DNS ending in .googlebot.com or .google.com, published IP ranges. A user-agent header on its own proves nothing.

Its address list lives at https://developers.google.com/static/crawling/ipranges/common-crawlers.json, and its reverse-DNS names end in .googlebot.com or .google.com.

The verification procedure explains why a reverse lookup has to be forward-confirmed, and why a published range list has to be re-fetched rather than pinned.

How do I find Googlebot in my logs?

Match the user-agent against "googlebot", case-insensitively. Set the Googlebot fetch count beside the visits Google actually returns; that ratio is your crawl-to-referral gap for this engine.

Crawl-to-referral gap defines the ratio, and the rate-limiting guide covers what to do when the volume itself is the problem.

Registry last reviewed 2026-09-13.