AI crawlers · Google

Google-CloudVertexBot

Google-CloudVertexBot is a crawler operated by Google, and it fetches your page while a person waits for an answer — the fact that decides everything else on this page.

What does Google-CloudVertexBot actually do?

Google-CloudVertexBot is not part of Search and is not driven by Google's own products. It fetches pages so that a Vertex AI customer can ground their agent or search application on them, and Google documents it as crawling at a site owner's request. The traffic exists because somebody configured a data store pointing at your URLs.

That makes an unexplained appearance worth investigating. If nobody at your organisation set up a Vertex data store, then a third party added your site to theirs, and your content is being read into somebody else's product rather than into Google's index or an answer that carries your name.

Verification is awkward in a specific way. It shares reverse DNS namespaces with Googlebot and appears in Google's published crawler ranges, so hostname and address evidence prove only that the request came from Google infrastructure. Neither tells you which product made it or whose project it serves. The user-agent string is the only thing separating this from the search crawler.

The remedy is also unusual, because this is the one entry in the whole registry whose traffic exists because somebody deliberately pointed a product at your URLs. Everywhere else the decision is whether to let a vendor read you; here the fetches were configured by a customer, and the customer may be you. Check your own Vertex AI data stores first. If none of them name your domain, a robots rule is the only lever you have over a third party's grounding corpus, and it is worth writing before you go looking for who set it up.

CrawlClick files Google-CloudVertexBot as a query time crawl. What the three crawl purposes mean.

A robots rule that names Google rather than Google-CloudVertexBot also lands on GoogleOther (search index), Googlebot (search index), which does a different job. Blocking the wrong one of a vendor's agents is the commonest mistake in this whole subject.

Should I block Google-CloudVertexBot?

The google.com reverse DNS makes it look like a Search crawler, so it gets allowed or blocked on ranking grounds when it has no bearing on Google Search either way.

To block it anyway, add this to robots.txt (how the rules resolve):

User-agent: Google-CloudVertexBot
Disallow: /

Replace Disallow: / with Allow: / to permit it explicitly. A crawler that ignores robots.txt is not blocked by it either way — that is what verification and edge rules are for.

How do I verify Google-CloudVertexBot is genuine?

Google lets you check a request claiming to be Google-CloudVertexBot against reverse DNS ending in .google.com or .googlebot.com, published IP ranges. A user-agent header on its own proves nothing.

Its address list lives at https://developers.google.com/static/crawling/ipranges/common-crawlers.json, and its reverse-DNS names end in .google.com or .googlebot.com.

The verification procedure explains why a reverse lookup has to be forward-confirmed, and why a published range list has to be re-fetched rather than pinned.

How do I find Google-CloudVertexBot in my logs?

Match the user-agent against "google-cloudvertexbot", case-insensitively. Every Google-CloudVertexBot fetch had somebody waiting on it, so this count maps onto real questions being asked about your pages rather than onto crawl budget.

Crawl-to-referral gap defines the ratio, and the rate-limiting guide covers what to do when the volume itself is the problem.

Registry last reviewed 2026-09-13.