AI crawlers · ByteDance

Bytespider

Bytespider is a crawler operated by ByteDance, and it collects text to train a model — the fact that decides everything else on this page.

What does Bytespider actually do?

Bytespider is ByteDance's training crawler and the one with a reputation problem. There are no published address ranges, no reverse DNS suffix and no crawler documentation page - the registry's link points at a corporate site. That means no way to confirm a request is genuine and no vendor procedure to escalate to when the crawl misbehaves.

Its robots compliance has been widely questioned in public reporting, and requests attributed to it are routinely described as heavy. Whether every one of those accounts is fair or not, the operational conclusion is identical: a Disallow line here is not enforcement. If you want this crawl stopped, it has to be stopped somewhere you control.

The upside is that blocking costs you nothing measurable. This agent is not the retrieval path for any product that cites sources back to you, so there is no visibility to trade away and no referral stream to lose. Of everything in this registry it is the easiest block to justify to a colleague who asks why.

Whatever rule you write here has to be self-enforcing, because there is nobody to escalate to. The other vendors in this registry publish a crawler reference and, in most cases, an address file and a contact route; ByteDance publishes a corporate site. A substring match on the user-agent is the entire identification available, and an edge rule you enforce yourself is the entire enforcement available.

CrawlClick files Bytespider as a training crawl. What the three crawl purposes mean.

ByteDance runs one agent in this registry, which makes Bytespider the whole of the decision. Vendors that split the work across several tokens let a publisher decline one job and keep another; ByteDance publishes no second name here, so there is nothing to separate. Whatever you settle on for Bytespider governs every use ByteDance makes of your pages until another token appears — which is a coarser instrument than it looks, and worth weighing before the training label decides it for you.

Should I block Bytespider?

It is blocked in robots.txt and assumed handled, when this is the agent least likely to treat that file as binding.

To block it, add this to robots.txt (how the rules resolve):

User-agent: Bytespider
Disallow: /

Replace Disallow: / with Allow: / to permit it explicitly. A crawler that ignores robots.txt is not blocked by it either way — that is what verification and edge rules are for.

How do I verify Bytespider is genuine?

ByteDance publishes no way to check this one, so a request claiming to be Bytespider cannot be confirmed. Treat it as unverified and never bill anything against it.

The verification procedure explains why a reverse lookup has to be forward-confirmed, and why a published range list has to be re-fetched rather than pinned.

How do I find Bytespider in my logs?

Match the user-agent against "bytespider", case-insensitively. Nothing comes back from a training fetch, so keep Bytespider out of any count you use to judge ByteDance as a source of readers.

Crawl-to-referral gap defines the ratio, and the rate-limiting guide covers what to do when the volume itself is the problem.

Registry last reviewed 2026-09-13.