AI crawlers · You.com

YouBot

YouBot is a crawler operated by You.com, and it builds the index an assistant answers from — the fact that decides everything else on this page.

What does YouBot actually do?

YouBot crawls for You.com's index, which serves both its consumer assistant and its API and enterprise products. A fetch can therefore end up supporting an answer for a member of the public or a request made through somebody's paid integration, and nothing in the request itself tells you which of the two you are contributing to.

You.com publishes no address ranges, no reverse DNS suffix and no signed requests, so the monetizable-when-verified position is theoretical for this agent - there is currently no route to the verified state. Any billing or preferential-serving rule keyed on this user-agent is unenforced until the vendor publishes something to check against.

The documentation is a product page rather than a crawler reference with tokens, ranges and behaviour notes. If you want certainty about what this crawl does with your content, the honest answer is that the public record does not supply it, and defaulting to the conservative choice is a defensible position rather than a paranoid one.

It also breaks the usual measurement. Comparing fetches against visits returned works when the crawl serves one visible surface, and here it serves at least two: a consumer assistant that can send a reader, and paid integrations that never will. Nothing in the request distinguishes them, so a ratio calculated for this agent mixes a referral channel and a licensing channel into one figure that means neither.

CrawlClick files YouBot as a search index crawl. What the three crawl purposes mean.

You.com runs one agent in this registry, which makes YouBot the whole of the decision. Vendors that split the work across several tokens let a publisher decline one job and keep another; You.com publishes no second name here, so there is nothing to separate. Whatever you settle on for YouBot governs every use You.com makes of your pages until another token appears — which is a coarser instrument than it looks, and worth weighing before the search index label decides it for you.

Should I block YouBot?

The search-index label suggests an engine you can measure yourself against, but this crawl also feeds API and enterprise products, so allowing it is not the visibility trade it looks like.

To block it anyway, add this to robots.txt (how the rules resolve):

User-agent: YouBot
Disallow: /

Replace Disallow: / with Allow: / to permit it explicitly. A crawler that ignores robots.txt is not blocked by it either way — that is what verification and edge rules are for.

How do I verify YouBot is genuine?

You.com publishes no way to check this one, so a request claiming to be YouBot cannot be confirmed. Treat it as unverified and never bill anything against it.

The verification procedure explains why a reverse lookup has to be forward-confirmed, and why a published range list has to be re-fetched rather than pinned.

How do I find YouBot in my logs?

Match the user-agent against "youbot", case-insensitively. Set the YouBot fetch count beside the visits You.com actually returns; that ratio is your crawl-to-referral gap for this engine.

Crawl-to-referral gap defines the ratio, and the rate-limiting guide covers what to do when the volume itself is the problem.

Registry last reviewed 2026-09-13.