GPTBot
OpenAI's training crawler, which collects web content used to improve its models rather than to answer any specific user question.
Also called: OpenAI crawler
Whatever GPTBot reads today feeds a model released months from now. No answer changes because of the visit, and no reader arrives because of it.
Is GPTBot the same as ChatGPT-User?
No, and the difference decides what a visit is worth. ChatGPT-User fetches a page because somebody
has just asked something and is sitting there waiting. OAI-SearchBot maintains the index behind
ChatGPT Search. Three agents, three jobs, one vendor.
How do I block it precisely?
Two lines in robots.txt at your site root: User-agent: GPTBot followed by Disallow: /. Naming
the agent that way leaves the other two untouched, so you stay present in answers while dropping out
of the training corpus.
Does blocking undo what was already taken?
No. The exclusion protocol governs future requests only, and text gathered before you added the rule sits inside whatever was trained on it. Publishers expecting a retroactive effect are usually disappointed.
Can I close off part of the site instead?
Yes. Disallow: takes a path, so an archive can be shut while the reference pages you want credited
in answers stay open. Blanket rules are a habit rather than a requirement.
How is a claimed GPTBot verified?
By network address, not by header. OpenAI publishes the ranges its crawler uses as a JSON file, and a request from outside those ranges wearing the same user-agent string is a scraper in costume.
Is GPTBot traffic monetizable?
Never. A disclosed sponsored passage is written to be read by a person, and burying one in a training corpus puts the claim somewhere the label cannot follow it.
What else should I read?
- AI crawler
A bot operated by an AI company that fetches web pages to train a model, build a search index, or answer a user's question in real time.
- Query-time crawl
A page fetch made by an AI engine while a user waits for an answer, as opposed to bulk indexing done in advance.
- robots.txt
A file at the root of a site that tells crawlers which paths they may fetch, and which named agents are allowed or disallowed.