AI crawlers · OpenAI

OAI-SearchBot

OAI-SearchBot is a crawler operated by OpenAI, and it builds the index an assistant answers from — the fact that decides everything else on this page.

What does OAI-SearchBot actually do?

OAI-SearchBot decides whether you exist inside ChatGPT's search at all. It builds the retrieval index; ChatGPT-User then opens individual pages during conversations. Absence here costs you more than citations, because the live fetch is unlikely to happen either - the model has to know a URL exists before it can decide to open it.

It behaves like a conventional search crawler: breadth across a site, internal links, sitemaps, repeat visits as pages change. The levers are the ordinary ones. Stable canonical URLs, a sitemap that reflects what is actually published, and identical responses regardless of user agent will do more here than anything AI-specific.

OpenAI publishes a separate address file per bot rather than one combined list, which is more useful than it first appears. It lets you enforce the split at the edge instead of trusting robots.txt: allow this agent's ranges, drop the training crawler's, and neither decision depends on a header that anyone can forge.

OpenAI's crawler documentation now lists a fourth agent that this registry does not yet carry: OAI-AdsBot, described as validating the safety of advertising landing pages, with its own robots token and its own address file. Whatever you decide about it, decide rather than omit - a token nobody considered is an allow made by accident, and this is the newest one in circulation.

CrawlClick files OAI-SearchBot as a search index crawl. What the three crawl purposes mean.

A robots rule that names OpenAI rather than OAI-SearchBot also lands on ChatGPT-User (query time), GPTBot (training), which does a different job. Blocking the wrong one of a vendor's agents is the commonest mistake in this whole subject.

Should I block OAI-SearchBot?

The commonest error is treating OAI-SearchBot and GPTBot as one thing because both are OpenAI crawlers, then writing a single rule against whichever name came to mind first.

To block it anyway, add this to robots.txt (how the rules resolve):

User-agent: OAI-SearchBot
Disallow: /

Replace Disallow: / with Allow: / to permit it explicitly. A crawler that ignores robots.txt is not blocked by it either way — that is what verification and edge rules are for.

How do I verify OAI-SearchBot is genuine?

OpenAI lets you check a request claiming to be OAI-SearchBot against signed request (Web Bot Auth), published IP ranges. A user-agent header on its own proves nothing.

Its address list lives at https://openai.com/searchbot.json.

The verification procedure explains why a reverse lookup has to be forward-confirmed, and why a published range list has to be re-fetched rather than pinned.

How do I find OAI-SearchBot in my logs?

Match the user-agent against "oai-searchbot", case-insensitively. Set the OAI-SearchBot fetch count beside the visits OpenAI actually returns; that ratio is your crawl-to-referral gap for this engine.

Crawl-to-referral gap defines the ratio, and the rate-limiting guide covers what to do when the volume itself is the problem.

Registry last reviewed 2026-09-13.