AI crawlers · Meta

meta-externalagent

meta-externalagent is a crawler operated by Meta, and it builds the index an assistant answers from — the fact that decides everything else on this page.

What does meta-externalagent actually do?

meta-externalagent is Meta's bulk crawler, the one that works across a site rather than fetching what somebody just asked about. Meta describes its output as feeding both product indexes and model training, so filing it purely as a search crawler understates what allowing it agrees to. Read the vendor's own description before deciding this one is harmless.

In logs the two Meta agents look nothing alike. This one arrives in sustained sequences across many URLs, following links as it goes, while the fetcher shows up as isolated single requests. If Meta's crawling has become a bandwidth problem, this is the agent responsible, and a crawl-rate rule may be the proportionate answer before a block.

Meta publishes no address list and no reverse DNS name, so a genuine fetch and a forged header are indistinguishable. That cuts both ways. You cannot allowlist or bill this agent with any confidence, but you also give up nothing verifiable by dropping the user-agent at your edge, because there was never any evidence to weigh.

Watch the casing when you go looking for it. Meta's documentation writes these names as Meta-ExternalAgent and Meta-ExternalFetcher, while the strings arriving in logs are commonly lowercase. Robots matching is case-insensitive under RFC 9309, so a group naming either form still applies; a case-sensitive grep in your log parser is not, and will quietly report a fraction of the real volume.

CrawlClick files meta-externalagent as a search index crawl. What the three crawl purposes mean.

A robots rule that names Meta rather than meta-externalagent also lands on meta-externalfetcher (query time), which does a different job. Blocking the wrong one of a vendor's agents is the commonest mistake in this whole subject.

Should I block meta-externalagent?

It gets filed under AI training and blocked, while meta-externalfetcher - the agent that reads pages for Meta AI during conversations - is left untouched.

To block it anyway, add this to robots.txt (how the rules resolve):

User-agent: meta-externalagent
Disallow: /

Replace Disallow: / with Allow: / to permit it explicitly. A crawler that ignores robots.txt is not blocked by it either way — that is what verification and edge rules are for.

How do I verify meta-externalagent is genuine?

Meta publishes no way to check this one, so a request claiming to be meta-externalagent cannot be confirmed. Treat it as unverified and never bill anything against it.

The verification procedure explains why a reverse lookup has to be forward-confirmed, and why a published range list has to be re-fetched rather than pinned.

How do I find meta-externalagent in my logs?

Match the user-agent against "meta-externalagent", case-insensitively. Set the meta-externalagent fetch count beside the visits Meta actually returns; that ratio is your crawl-to-referral gap for this engine.

Crawl-to-referral gap defines the ratio, and the rate-limiting guide covers what to do when the volume itself is the problem.

Registry last reviewed 2026-09-13.