AI crawlers · Common Crawl

CCBot

CCBot is a crawler operated by Common Crawl, and it collects text to train a model — the fact that decides everything else on this page.

What does CCBot actually do?

CCBot is the only agent here run by a non-profit and the only one whose output you can go and inspect. Common Crawl publishes its archives openly, so what was captured from your site is a matter of public record rather than something you have to infer from your own access logs.

That openness is also why the block decision is broader than it looks. Those archives are a standard input to model training across the industry, but they are equally what researchers, archivists and small projects rely on because they cannot afford to crawl the web themselves. Blocking CCBot removes you from all of that, not only from the model training.

It is retrospective in one direction only. A block stops future captures; archives already published stay published, and stay inside whatever derived datasets were built from them. If your aim is to undo past inclusion, robots.txt is not the mechanism - this is a decision about what happens from now on.

Common Crawl now documents both ways of proving a fetch is real, which is more than several commercial vendors in this registry manage. Genuine addresses reverse-resolve into the crawl.commoncrawl.org namespace over IPv4, forward-confirm back to the same address, and are also published as JSON. The foundation warns in the same breath that crawlers falsely identifying themselves as CCBot exist, so the check is not ceremonial.

It is also the only entry here whose documentation points at an opt-out registry running alongside robots.txt, rather than treating the file as the whole of the interface. That is a small thing and a revealing one: an organisation publishing an archive has to think about withdrawal in a way that an operator training a private model has never been obliged to.

CrawlClick files CCBot as a training crawl. What the three crawl purposes mean.

Common Crawl runs one agent in this registry, which makes CCBot the whole of the decision. Vendors that split the work across several tokens let a publisher decline one job and keep another; Common Crawl publishes no second name here, so there is nothing to separate. Whatever you settle on for CCBot governs every use Common Crawl makes of your pages until another token appears — which is a coarser instrument than it looks, and worth weighing before the training label decides it for you.

Should I block CCBot?

CCBot gets blocked as an AI training crawler, when Common Crawl is an open archive that academic and non-commercial projects depend on as well.

To block it, add this to robots.txt (how the rules resolve):

User-agent: CCBot
Disallow: /

Replace Disallow: / with Allow: / to permit it explicitly. A crawler that ignores robots.txt is not blocked by it either way — that is what verification and edge rules are for.

How do I verify CCBot is genuine?

Common Crawl lets you check a request claiming to be CCBot against reverse DNS ending in .crawl.commoncrawl.org, published IP ranges. A user-agent header on its own proves nothing.

Its address list lives at https://index.commoncrawl.org/ccbot.json, and its reverse-DNS names end in .crawl.commoncrawl.org.

The verification procedure explains why a reverse lookup has to be forward-confirmed, and why a published range list has to be re-fetched rather than pinned.

How do I find CCBot in my logs?

Match the user-agent against "ccbot", case-insensitively. Nothing comes back from a training fetch, so keep CCBot out of any count you use to judge Common Crawl as a source of readers.

Crawl-to-referral gap defines the ratio, and the rate-limiting guide covers what to do when the volume itself is the problem.

Registry last reviewed 2026-09-13.