AI crawlers · Cohere

cohere-ai

cohere-ai is a crawler operated by Cohere, and it builds the index an assistant answers from — the fact that decides everything else on this page.

What does cohere-ai actually do?

This entry covers two user-agent strings doing two different jobs, and the second one says exactly what it is for: it is a training-data crawler. A robots group naming only the first string leaves the training crawl entirely unaddressed, which is the opposite of what most people writing that rule believe they have done.

Cohere sells models and retrieval tooling to businesses rather than running a consumer search engine. A fetch here is more likely to be serving somebody's enterprise deployment than a public answer with a citation next to your name. There is no results page to check afterwards, and so no obvious visibility benefit to weigh against allowing it.

No address list, no reverse DNS suffix, no signed requests, and a documentation link that lands on a company front page rather than a crawler reference. Verification is not possible today, which leaves the robots tokens as the entire control surface - and makes getting both of them right the whole of the job.

Treat the single registry entry as a limit of the public record rather than a claim that the two strings do the same work. One is named for training data collection and the other is not, which suggests two crawls with two purposes; Cohere documents neither, so there is no basis for giving them separate pages, separate advice or separate rules beyond naming both.

CrawlClick files cohere-ai as a search index crawl. What the three crawl purposes mean.

Cohere runs one agent in this registry, which makes cohere-ai the whole of the decision. Vendors that split the work across several tokens let a publisher decline one job and keep another; Cohere publishes no second name here, so there is nothing to separate. Whatever you settle on for cohere-ai governs every use Cohere makes of your pages until another token appears — which is a coarser instrument than it looks, and worth weighing before the search index label decides it for you.

Should I block cohere-ai?

A rule is written for cohere-ai and considered complete, missing the separately named training-data crawler that shares this registry entry.

To block it anyway, add this to robots.txt (how the rules resolve):

User-agent: cohere-ai
Disallow: /

Replace Disallow: / with Allow: / to permit it explicitly. A crawler that ignores robots.txt is not blocked by it either way — that is what verification and edge rules are for.

How do I verify cohere-ai is genuine?

Cohere publishes no way to check this one, so a request claiming to be cohere-ai cannot be confirmed. Treat it as unverified and never bill anything against it.

The verification procedure explains why a reverse lookup has to be forward-confirmed, and why a published range list has to be re-fetched rather than pinned.

How do I find cohere-ai in my logs?

Match the user-agent against "cohere-ai" or "cohere-training-data-crawler", case-insensitively. Set the cohere-ai fetch count beside the visits Cohere actually returns; that ratio is your crawl-to-referral gap for this engine.

Crawl-to-referral gap defines the ratio, and the rate-limiting guide covers what to do when the volume itself is the problem.

Registry last reviewed 2026-09-13.