AI crawlers · Meta

meta-externalfetcher

meta-externalfetcher is a crawler operated by Meta, and it fetches your page while a person waits for an answer — the fact that decides everything else on this page.

What does meta-externalfetcher actually do?

meta-externalfetcher opens individual links in support of a product action: a page Meta AI needs in order to answer, a URL somebody put in front of the assistant. It works one URL at a time rather than crawling, so it appears sporadically and around specific content rather than steadily across your archive.

It is not facebookexternalhit. That is the separate agent behind link previews when your URL is shared on Facebook, Instagram, Messenger or WhatsApp, and it is not in this registry at all. Blocking Meta by network range, or writing a robots rule against a wildcard you assume covers anything Meta-shaped, is how publishers lose their share cards by accident.

Because it fetches on behalf of somebody mid-task, a block here is felt immediately and in public: the assistant reports that it could not open your link, in the conversation, to the person who asked. That is a different kind of cost from being absent from an index, and worth deciding separately from whatever you decided about the bulk crawl.

Meta documents that this agent may bypass robots.txt rules, on the grounds that the fetch was initiated by a person rather than scheduled. A disallow line is therefore a request here rather than a control, and the same documentation notes that the separate link-preview agent may also bypass the file during security or integrity checks. Neither of those is an allegation about misbehaviour - both are published in advance.

CrawlClick files meta-externalfetcher as a query time crawl. What the three crawl purposes mean.

A robots rule that names Meta rather than meta-externalfetcher also lands on meta-externalagent (search index), which does a different job. Blocking the wrong one of a vendor's agents is the commonest mistake in this whole subject.

Should I block meta-externalfetcher?

People assume blocking Meta means one rule, then discover the block landed on link previews as well because the preview fetcher answers to a name they never checked.

To block it anyway, add this to robots.txt (how the rules resolve):

User-agent: meta-externalfetcher
Disallow: /

Replace Disallow: / with Allow: / to permit it explicitly. A crawler that ignores robots.txt is not blocked by it either way — that is what verification and edge rules are for.

How do I verify meta-externalfetcher is genuine?

Meta publishes no way to check this one, so a request claiming to be meta-externalfetcher cannot be confirmed. Treat it as unverified and never bill anything against it.

The verification procedure explains why a reverse lookup has to be forward-confirmed, and why a published range list has to be re-fetched rather than pinned.

How do I find meta-externalfetcher in my logs?

Match the user-agent against "meta-externalfetcher", case-insensitively. Every meta-externalfetcher fetch had somebody waiting on it, so this count maps onto real questions being asked about your pages rather than onto crawl budget.

Crawl-to-referral gap defines the ratio, and the rate-limiting guide covers what to do when the volume itself is the problem.

Registry last reviewed 2026-09-13.