Meta lists four other crawlers with separate user agents, and one of them, Meta-ExternalFetcher, may bypass robots.txt, which matters for crawler controls as one part of AI training data licensing and acquisition.
Every statement here comes from Meta's "Meta Web Crawlers" page (checked 2026-11-09, re-read 2026-11-16), and the page recommends no rule. A site addresses each crawler by its own robots.txt token, and a Meta-ExternalAgent disallow leaves link previews, Meta AI citations and user-requested fetches running.
Every figure and statement carries its stating party and check date, and how we source, date and check every figure sets out the method.
What Meta-ExternalAgent collects, in Meta's words
Meta states that Meta-ExternalAgent crawls the web "for use cases such as training foundation AI models or improving products by indexing content directly", which makes it the only crawler on Meta's page tied to AI model training.
The words "such as" mean that the list of use cases is not closed, and Meta's page names no model, no dataset and no product that receives the collected content (checked 2026-11-16).
One crawler serves two uses here, training and indexing, so Meta's documentation offers no token that stops training alone. Cloudflare measured Meta-ExternalAgent at 7.7 % of crawler traffic to its North American customers in its October 2025 data, which is Cloudflare's measurement and not Meta's.
The section hub on AI crawlers and how to allow, block or charge them places Meta's crawler among those of other operators.
Meta's five documented crawlers as Meta lists them
Meta's crawler page lists five crawlers, FacebookExternalHit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher, each with two example user-agent strings. The table below compares them in Meta's order, as of 2026-11-16, under the caption "Meta's five documented web crawlers".
| Crawler | Purpose (Meta's words) | Example user-agent strings | robots.txt (Meta's statement) | Training (Meta's statement) | Checked | Source |
|---|---|---|---|---|---|---|
| FacebookExternalHit | crawls "the content of an app or website that was shared on one of Meta's family of apps"; gathers link title, description and thumbnail | facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php); facebookexternalhit/1.1 |
"might bypass robots.txt when performing security or integrity checks" | not stated | 2026-11-16 | developers.facebook.com |
| Meta-WebIndexer | "navigates the web to improve Meta AI search result quality for users" | meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers); meta-webindexer/1.1 |
allowing it "helps us cite and link to your content in Meta AI's responses" | not stated | 2026-11-16 | developers.facebook.com |
| Meta-ExternalAds | "use cases such as improving advertising and other business-related products and services" | meta-externalads/1.1 (+/documentation/sharing/webmasters/web-crawlers); meta-externalads/1.1 |
own group in Meta's example | not stated | 2026-11-16 | developers.facebook.com |
| Meta-ExternalAgent | "use cases such as training foundation AI models or improving products by indexing content directly" | meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers); meta-externalagent/1.1 |
blocked by a disallow for it | "training foundation AI models" | 2026-11-09, re-checked 2026-11-16 | developers.facebook.com |
| Meta-ExternalFetcher | "fetches individual links at a user's request" | meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers); meta-externalfetcher/1.1 |
"may bypass robots.txt rules" | not stated | 2026-11-09, re-checked 2026-11-16 | developers.facebook.com |
Reading key:
- Meta calls these its "most common web crawlers", so the page is not stated to be a complete list.
- Log strings may be "similar to" the examples, and the bracketed path is Meta's own relative form.
- Only Meta-ExternalAgent is tied to training in Meta's wording.
robots.txt lines for Meta-ExternalAgent and Meta's other crawlers
Meta tells site owners to stop its crawlers by adding "a disallow for the relevant crawler" to robots.txt, so each Meta bot is addressed by its own user-agent token.
Meta's example gives meta-externalagent and meta-externalads separate groups, each with Allow: / and Disallow: /private/, and it writes the tokens in lowercase, a form copied here (checked 2026-11-16).
The table below compares five goals as of 2026-11-16, each token in its own User-agent group, with what Meta's statements say the lines do and what keeps running.
| Goal | Lines | What it does (Meta's statement) | What keeps running | Source |
|---|---|---|---|---|
| Stop Meta's training and indexing crawler, keep previews and Meta AI citations | User-agent: meta-externalagent then Disallow: / |
Meta-ExternalAgent stops crawling the site, both its training and its indexing use | FacebookExternalHit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalFetcher | developers.facebook.com |
| Keep a directory out (Meta's own example) | User-agent: meta-externalagent, Allow: /, then Disallow: /private/ |
the named directory is disallowed for Meta-ExternalAgent | the rest of the site for all five | developers.facebook.com |
| Also leave Meta AI search | the first row plus User-agent: meta-webindexer then Disallow: / |
Meta-WebIndexer stops; Meta says allowing it helps Meta cite and link content in Meta AI | FacebookExternalHit, Meta-ExternalAds, Meta-ExternalFetcher | developers.facebook.com |
| Also stop the advertising crawler | User-agent: meta-externalads then Disallow: / |
Meta-ExternalAds stops | the other four | developers.facebook.com |
| Ask user-requested fetches to stop | User-agent: meta-externalfetcher then Disallow: / |
a request Meta says the crawler "may bypass" | depends on Meta | developers.facebook.com |
User-agent: meta-externalagent
Disallow: /
These are whole-site lines and Meta's directory example only. Syntax, precedence and other operators' tokens are covered on the page about robots.txt for AI crawlers.
Meta-ExternalFetcher and FacebookExternalHit: the Meta bots that may bypass robots.txt
Meta states that Meta-ExternalFetcher "may bypass robots.txt rules" because it fetches links a user requested, and that FacebookExternalHit "might bypass robots.txt when performing security or integrity checks". Robots.txt is therefore not a dependable control against these two, whatever a site writes for the others (checked 2026-11-16).
Meta describes Meta-ExternalFetcher as a crawler that "fetches individual links at a user's request and supports product functions such as evaluating and improving agentic AI capabilities", and adds "including helping AI navigate websites to complete tasks for users".
Meta ties no training use to it on its page, so it is not called a training crawler here. For FacebookExternalHit, Meta says the bypass can occur "such as checking for malware or malicious content".
- Meta-ExternalFetcher: may bypass robots.txt "because it performs fetches that were requested by the user".
- FacebookExternalHit: crawls content shared on Facebook, Instagram or Messenger, "gathers, caches, and displays" its title, description and thumbnail, and might bypass robots.txt for security or integrity checks.
FacebookBot and other Meta names missing from Meta's crawler page
FacebookBot does not appear on Meta's crawler page as checked on 2026-11-16, and the documentation address once used for FacebookBot now shows the same five-crawler page. The address developers.facebook.com/docs/sharing/bot/ serves "Meta Web Crawlers" on that date.
Meta's page tells owners to name "the relevant crawler", and FacebookBot is not one of the five it lists. What FacebookBot did, whether it still runs and whether a FacebookBot group binds any listed crawler are not stated on Meta's page, so none is claimed here.
A site that still carries a FacebookBot group keeps it at no cost, but it is not one of the tokens Meta documents.
Recognizing Meta-ExternalAgent in server logs
A request identifies itself as Meta-ExternalAgent when its user agent carries the meta-externalagent/1.1 token, and Meta's page gives the strings but no IP list for checking them. Meta states no IP list, ASN, reverse-DNS method, crawl-delay statement or crawl-rate setting on the page checked 2026-11-16, and the page itself is marked as updated on 2026-05-21.
- Look for the meta-externalagent token in the user-agent field, in the form of Meta's examples.
- Tell it apart from meta-externalfetcher and facebookexternalhit, whose robots.txt treatment differs.
- Write to webmasters@meta.com with questions, the route Meta names for "questions or concerns".
For FacebookExternalHit, Meta advises allow-listing "either the user agent strings or the IP addresses (more secure)" without giving the addresses on the page, and no IP method is described here. Meta states no rate limit, so the page names none.
What a Meta-ExternalAgent disallow does and does not change
A Meta-ExternalAgent disallow asks Meta's training and indexing crawler to stop collecting the site, and by Meta's account it takes effect within up to 24 hours, but it does not stop link previews, Meta-WebIndexer or user-requested fetches.
Meta writes: "Please allow up to 24 hours for changes to robots.txt to take effect because crawlers may cache the contents of robots.txt for up to 24 hours" (checked 2026-11-16).
RFC 9309 (2022) says a cached copy should not be used for more than 24 hours, so Meta's statement and the protocol agree, and no claim is made about how Meta treats a server error.
Meta says it prefers "industry-standard practices like robots.txt rather than non-standard formats like NoAI tags", so a NoAI tag is not a signal Meta documents, and the page on machine-readable TDM reservations and meta tags compares the page-level tags other developers say they read.
A Meta-ExternalAgent disallow leaves four things running:
- FacebookExternalHit link previews for pages shared on Meta's apps.
- Meta-WebIndexer indexing for Meta AI search and citations, which a Meta-ExternalAgent group does not name.
- Meta-ExternalFetcher fetches that a user requested.
- Content collected before the rule, since no Meta statement about deleting it is recorded here.
Content that people post on Facebook or Instagram is not governed by a site's robots.txt. Meta states that it trains on EU adults' public content subject to an objection form, and the case page on Meta AI training on EU user data and the objection route covers that separate question.
Meta's EU position behind a Meta-ExternalAgent disallow
In the EU, a Meta-ExternalAgent disallow can act as the machine-readable reservation of DSM Directive Article 4(3), and Meta does not appear among the 22 signatories listed for the GPAI Code of Practice, whose Copyright chapter commits signatories to crawlers that follow robots.txt as specified in RFC 9309.
Article 4(3) of Directive (EU) 2019/790 (in force 2019-06-06) allows the text and data mining exception only where use has not been reserved "in an appropriate manner, such as machine-readable means in the case of content made publicly available online", as set out on the page about the EU text and data mining exception and its opt-out.
The European Commission's Code page, last updated 2026-10-07 (checked 2026-11-09), lists 22 signatories and Meta is not among them. The Code is voluntary and was published on 2025-07-10, and its Copyright chapter, Measure 1.3, says signatories employ web crawlers that read and follow robots.txt as specified in RFC 9309.
Regulation (EU) 2024/1689, Article 53(1)(c), applies from 2025-08-02 and requires general-purpose AI providers, signatory or not, to keep a copyright policy that identifies and complies with Article 4(3) reservations. No ruling on whether a NoAI tag counts as such a reservation is recorded here, and no US rule that makes robots.txt binding is recorded here.
Blocking Meta-ExternalAgent: benefits, risks, downsides and alternatives
Disallowing Meta-ExternalAgent costs nothing and Meta states that a disallow blocks it, but it earns nothing, reaches neither Meta-ExternalFetcher nor content collected earlier, and also ends the crawler's indexing use. Licensing content to Meta is a separate decision from crawler rules, and the profile of Meta as a data buyer covers that side.
Benefits:
- The rule is free, a few lines in a plain text file.
- It follows Meta's own instruction to add a disallow for the relevant crawler.
- Link previews and Meta AI citations stay if FacebookExternalHit and Meta-WebIndexer stay allowed.
- In the EU, the disallow can act as a machine-readable reservation under DSM Directive Article 4(3).
Risks:
- Meta-ExternalFetcher may still fetch pages, because Meta says it may bypass robots.txt.
- A spoofed string cannot be checked against a Meta IP list taken from the crawler page.
- A change may take up to 24 hours to reach Meta's crawlers.
Downsides:
- The disallow also ends Meta-ExternalAgent's indexing use, since Meta documents no training-only token.
- It brings no payment.
- It is prospective only, and Meta names no deletion of earlier collection.
Alternatives:
- Allow everything.
- Disallow Meta-ExternalAgent and keep Meta-WebIndexer and FacebookExternalHit allowed.
- Charge or license crawling through intermediaries, covered under the border below.
- License content directly, or decide on whether to sell or license your data to AI companies at all, where not licensing is a valid outcome.
Blocking, allowing and licensing can be combined, and none is recommended here.
Other AI crawlers, opt-out signals and the law around Meta's crawling
Meta-ExternalAgent is one crawler among many, and the pages below cover the others, the other ways to signal an opt-out and the law around crawling.
- AI crawlers and how to allow, block or charge them: every AI crawler and control method in one place.
- GPTBot and OpenAI's other crawlers: OpenAI's GPTBot and its search and user bots.
- ClaudeBot: Anthropic's ClaudeBot, Claude-User and Claude-SearchBot.
- Amazonbot: Amazon's crawler, whose robots.txt cache runs up to 30 days against Meta's 24 hours.
- Google-Extended: opting out of Gemini training.
- Applebot-Extended: Apple's training opt-out token.
- Pay-per-crawl and crawler licensing: charging crawlers instead of blocking them.
- The shared AI training opt-out: the draft shared opt-out, for which no Meta statement is recorded on Meta's crawler page (re-read 2026-11-16).
- Opting out of AI training through platform settings: opt-outs for content hosted on platforms, where robots.txt is not the owner's to edit.
- EU AI Act Article 53 copyright policy: the copyright policy GPAI providers must keep under Article 53.
- Is web scraping for AI training legal: copyright, contract and computer-misuse law on scraping.
This page is information, not legal advice.
Frequently asked questions
Does blocking Meta-ExternalAgent stop Facebook link previews?
Not by Meta's account: link previews come from FacebookExternalHit, which crawls links shared on Facebook, Instagram or Messenger and has its own user agent. A Meta-ExternalAgent group does not name it (checked 2026-11-16).
Which Meta models does Meta-ExternalAgent collect content for?
Meta's crawler page names no model. It says Meta-ExternalAgent crawls for use cases "such as training foundation AI models or improving products by indexing content directly" (checked 2026-11-09 and 2026-11-16).
Does Meta honor NoAI meta tags?
Meta's crawler page documents none. It says Meta uses "industry-standard practices like robots.txt rather than non-standard formats like NoAI tags", so robots.txt is the signal Meta describes (checked 2026-11-16).
How long does a robots.txt change take to reach Meta's crawlers?
Up to 24 hours by Meta's statement, because its crawlers "may cache the contents of robots.txt for up to 24 hours". RFC 9309 sets the same 24-hour limit for a cached copy.
Is FacebookBot still a Meta crawler?
Meta's crawler page does not list FacebookBot as of 2026-11-16. It lists FacebookExternalHit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher as its most common crawlers.