# Meta documents five crawler tokens, and we published that it documents one > Meta's web crawler documentation, read on 8 August 2026, names facebookexternalhit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher. Lantad's crawler registry carries one Meta token, and a post we published on 1 August 2026 stated that Meta has no second published name to allow separately. The name most worth having is Meta-WebIndexer, which Meta describes as serving Meta AI search. - Canonical page: https://lantad.co/blog/meta-documents-five-crawler-tokens - This file: https://lantad.co/blog/meta-documents-five-crawler-tokens.md - Last substantive update: 2026-08-08 ## Key facts - **Published:** 2026-08-08 - **Category:** Findings - **Author:** Lantad - **Length:** 3003 words - **Takeaway 1:** Meta's web crawler documentation, read on 8 August 2026, names five crawlers: facebookexternalhit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher. The page carries no last updated date. - **Takeaway 2:** Meta describes Meta-WebIndexer as navigating the web to improve Meta AI search result quality, and Meta-ExternalAgent as crawling for training foundation AI models, so the training and search separation is writable for Meta and we reported that it was not. - **Takeaway 3:** A Lantad post published on 1 August 2026 stated that Meta publishes one token and no second name to allow separately, and Lantad's crawler registry carried one Meta entry when this post was written. Both understated what Meta documents. - **Takeaway 4:** Meta's page states that Meta-ExternalFetcher may bypass robots.txt because it performs fetches requested by the user, and that facebookexternalhit might bypass robots.txt when performing security or integrity checks. - **Takeaway 5:** The robots.txt example printed on Meta's own documentation page addresses meta-externalagent and nothing else, so a site owner copying it leaves the other four tokens to the wildcard group or, under RFC 9309, to no rules at all. ## Summary On 1 August 2026 this blog published a count of how many robots.txt product tokens each [AI crawler](https://lantad.co/glossary/ai-crawler) vendor makes available, and used it to argue that the standard advice to block training and allow search is unwritable for most of them. Meta was one of the vendors we placed on the wrong side of that line. The post stated that Meta publishes one name, Meta-ExternalAgent, and no second published name to allow separately. Meta's own documentation names five. Read on 8 August 2026, the page at developers.facebook.com/docs/sharing/webmasters/web-crawlers lists facebookexternalhit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher, with a stated purpose for each. One of the five is a search crawler. That makes the separation we said was unavailable for Meta available, and it makes [our count of vendors publishing a single token](https://lantad.co/blog/six-vendors-publish-one-crawler-token) wrong about at least one of them. What follows is what the page says, quoted, what the extra names change for a robots.txt written before today, and what we can and cannot establish about when the gap opened. The last part is short, because the answer is that we cannot date it. ## What Meta's crawler documentation actually lists The page gives each crawler a name, a user agent string and a sentence of purpose. Quoting the sentences matters here, because the difference between two of them is the whole of this post. Meta-ExternalAgent, it says, crawls the web for use cases such as training foundation AI models or improving products by indexing content directly. Meta-WebIndexer, it says, navigates the web to improve Meta AI search result quality for users, and allowing it helps Meta cite and link to your content in Meta AI's responses. Meta-ExternalAds crawls for use cases such as improving advertising and other business related products and services. Meta-ExternalFetcher fetches individual links at a user's request and supports product functions such as evaluating and improving agentic AI capabilities. facebookexternalhit crawls content that was shared on one of Meta's family of apps, to gather the title, description and thumbnail. Those are five distinct jobs with five distinct names, which is the condition a per purpose robots.txt rule needs. It is worth being precise about why the names are what carries the weight. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), the Robots Exclusion Protocol, says that crawlers must use case insensitive matching to find the group that matches the product token and then obey the rules of that group. Nothing in the format addresses a purpose, a product or a company. It addresses a token. Five tokens is five addressable things; one token is one. The user agent strings are worth a note of their own, because four of the five are printed on the page with a relative path inside the parentheses, which is the documentation rendering a link to itself rather than anything a crawler would transmit. The bare forms above are the parts that are documented without ambiguity. Our own [AI crawlers reference](https://lantad.co/tools/ai-crawlers) lists the tokens we evaluate and the user agent each one sends where a vendor publishes it, and the same caution applies there: a user agent is a string a client chooses to send. ## Meta-WebIndexer is a search crawler, which changes the advice The reason a second name matters is not tidiness. It is that the common instruction on AI crawler control, which is to disallow the crawler that trains and allow the crawler that retrieves, needs two published names before it can be typed into a file. Where a vendor publishes one, the instruction collapses into an on or off switch and the choice a site owner thought they were making was never on offer. OpenAI is the reference case for the shape. [OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots), read on 8 August 2026, names four bots and separates them by job: GPTBot for the generative models, OAI-SearchBot for surfacing websites in ChatGPT's search features, ChatGPT-User for certain user actions, and OAI-AdsBot for validating the safety of pages submitted as ads. Anthropic publishes a similar separation in [its crawler support article](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler). Those splits are why our guidance for [getting cited in ChatGPT](https://lantad.co/how-to-get-cited/chatgpt) can tell a reader to keep the search token allowed while blocking the training one, and mean it. Meta's page supports the same sentence. Meta-ExternalAgent is the training name. Meta-WebIndexer is the retrieval name, and Meta's own description ties it to citation: allowing it helps Meta cite and link to your content in Meta AI's responses. A site that wants to stay out of the training set while remaining quotable in Meta AI now has two names to write, and until today this blog told its readers it had one. The practical consequence sits in existing files rather than in new ones. A robots.txt written any time in the past year against the name Meta-ExternalAgent expresses a decision about training and expresses nothing at all about Meta-WebIndexer. Under RFC 9309 that crawler then falls to the wildcard group if the file has one, and if it does not, the specification is explicit that no rules apply. This is the same quiet failure as [a group headed by a name no crawler answers to](https://lantad.co/blog/renamed-crawler-tokens-leave-stale-robots-rules): the file parses, the rule is honoured, and the scope is not the one the author had in mind. It is also why counting names in vendor documentation is a different exercise from reading your logs, since [several tokens never appear in a log at all](https://lantad.co/blog/the-crawler-tokens-that-never-appear-in-your-logs). ## Two of the five may bypass robots.txt, and Meta says so The same page carries a sentence that most vendor crawler documentation does not. It states that the Meta-ExternalFetcher crawler may bypass robots.txt because it performs fetches that were requested by the user, and that the facebookexternalhit crawler might bypass robots.txt when performing security or integrity checks. Read plainly, that is a vendor telling site owners in advance that two of its five named clients are not bound by the file. It deserves credit for being written down rather than discovered, and it deserves to be taken literally. A disallow rule aimed at meta-externalfetcher is a request that Meta has already said it may decline, so a site treating that rule as a control has misread what it bought. This is the practical content of the line in RFC 9309 that robots.txt rules are not a form of access authorization, which is the same reason [robots.txt is the weaker of the two layers](https://lantad.co/blog/two-layers-decide-if-ai-can-read-your-site) that decide whether a client reaches your pages. The layer that actually answers is the one that returns the status code. There is a second reason not to treat the file as the last word here, and it is one we have measured rather than read. When we looked at what sites serving a robots.txt ban actually return to a named AI crawler, [a large share of them served a 200 anyway](https://lantad.co/blog/gptbot-bans-served-a-200-anyway). The file said no and the server said yes. A bypass that a vendor documents and a bypass that a server accidentally permits produce the same outcome for the page, and neither is visible from reading the robots.txt alone. Identity is the third caveat and it applies to all five names. Everything above assumes the client sending meta-webindexer in its user agent is Meta. Nothing in the header proves it, which is the argument in [a user agent is a claim, not an identity](https://lantad.co/blog/a-user-agent-is-a-claim-not-an-identity). Meta's page does not publish IP ranges or a signature scheme for these crawlers, so a site wanting to verify rather than trust has nothing on this page to verify against. That is a real limitation of the documentation and it is separate from the token count. ## Our own registry and our own post named one The correction is worth stating without softening it. Our crawler registry, the list of product tokens a scan evaluates a robots.txt against, held fifteen tokens across nine vendors when this post was written, and exactly one of them was Meta's. The post of 1 August built a finding on that list and rendered a chart with Meta at one token, captioned as a record of what each vendor documents. It was not a record of what Meta documents. It was a record of what we had read. We cannot tell you when the gap opened, and we tried. Meta's page carries no last updated date, and we could not retrieve an archived copy of it from this environment to compare against, so the two readings we can put a date on are our own: a post on 1 August that said one, and a note in our registry source recording that the Meta user agent string was confirmed on 4 August. There are two explanations. Either Meta added names between early August and today, or we read the page incompletely and repeated the result. We have no evidence that separates them, and the second is not the kinder reading, so we are not going to pick the one that flatters us. What we can say is that the previous post's method was sound and its execution on this vendor was not. It told readers to open each vendor's documentation and count the names, which is exactly the check that finds this. The failure was doing it thoroughly for the vendors we link and less thoroughly for the two whose documentation sits on hosts this site does not link to, Meta and ByteDance, where the URL had to be written as plain text. The old URL we cited, developers.facebook.com/docs/sharing/webmasters/crawler, answers 301 to the page above rather than serving it, which is a small thing that makes a page easier to half read. Our [methodology](https://lantad.co/methodology) states what a scan measures and what it does not before it states any result, and [the research page](https://lantad.co/research) carries the same rule for anything aggregated. This post is the version of that rule applied to a published finding rather than to a scan. Separately, note that Meta's page asks for up to 24 hours for a robots.txt change to take effect, matching the 24 hour ceiling RFC 9309 puts on cached copies, so any edit you make from this post is not instantaneous. We looked at what that delay means in practice when [a robots.txt edit takes time to reach a crawler](https://lantad.co/blog/when-a-robots-txt-edit-reaches-a-crawler). ## What to check on your own robots.txt this week Four checks follow from this, and the first one is the only one that is urgent. Open your robots.txt and search it for the string meta. If the only Meta token in the file is meta-externalagent, then whatever decision that group expresses applies to Meta's training crawler and to nothing else Meta sends. Decide deliberately what you want Meta-WebIndexer to do, given that Meta ties it to being cited and linked in Meta AI answers, and write a group for it either way. The same applies to Meta-ExternalAds if advertising crawls matter to you. Reading your file the way one named crawler reads it is what [a robots.txt tester](https://lantad.co/tools/robots-txt-tester) is for, and it is a per token check rather than a per file one. Second, do not copy the example. The robots.txt sample printed on Meta's own documentation page addresses meta-externalagent, and a reader who pastes it has covered one of five names while feeling covered for all of them. Vendor examples are written to demonstrate syntax, not to express your policy. The related trap is the wildcard: a single group applied to every crawler is a decision, and it is worth being the decision you chose rather than the one your file happened to make, which is the pattern behind [Google fetchers that ignore the wildcard entirely](https://lantad.co/blog/google-fetchers-ignore-the-robots-txt-wildcard). Third, recount your other vendors while you are in there. This post exists because a count of published names was taken once and treated as durable, and vendor documentation is not durable. Counting is cheap and reading a changelog is cheaper than discovering the change in a chart you published. If you want the count to mean something for [AI visibility](https://lantad.co/glossary/ai-visibility) rather than for tidiness, note which of the names you find are retrieval crawlers, because those are the ones attached to citations. Fourth, stop at the file and then keep going. Permission is upstream of the question that actually decides whether you can be quoted, which is whether your sentences are in the HTML that arrives. [Seeing what a crawler receives](https://lantad.co/tools/what-gptbot-sees) answers that in one request, and if your text only appears after a script runs, no token in any group was ever your constraint. We publish [what our own crawler sends](https://lantad.co/bot) for the same reason we published this correction: a tool that asks you to check your vendors should be checkable itself. For what it is worth on the standards side, the attempt to express purpose rather than identity in this file is live and unfinished, which is the subject of [what a Content-signal line does above a Disallow rule](https://lantad.co/blog/content-signal-asks-disallow-blocks). ## Questions and answers **How many web crawlers does Meta document?** Five, read on 8 August 2026: facebookexternalhit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher. Each carries a user agent string and a stated purpose on Meta's web crawler documentation page, which is at developers.facebook.com/docs/sharing/webmasters/web-crawlers and carries no last updated date. An earlier Lantad post, published on 1 August 2026, stated that Meta publishes one token, and this post corrects that. **What is Meta-WebIndexer?** Meta describes Meta-WebIndexer as a crawler that navigates the web to improve Meta AI search result quality for users, and states that allowing it helps Meta cite and link to your content in Meta AI's responses. It sends a user agent beginning meta-webindexer/1.1. It is a retrieval crawler rather than a training crawler, which is what makes it separately addressable from Meta-ExternalAgent in a robots.txt file. Meta's page does not state what happens if you block it. **Does blocking Meta-ExternalAgent also block Meta AI search?** No. Under RFC 9309 a crawler obeys the group matching its own product token, so a group headed by meta-externalagent applies to that crawler and not to Meta-WebIndexer. A crawler whose token heads no group falls to the wildcard group, and the specification states that if there is no matching group and no wildcard group, no rules apply. Blocking the training token leaves the retrieval token governed by whatever your wildcard says, or by nothing. **Do all of Meta's crawlers respect robots.txt?** No, and Meta says so on the page. It states that the Meta-ExternalFetcher crawler may bypass robots.txt because it performs fetches that were requested by the user, and that the facebookexternalhit crawler might bypass robots.txt when performing security or integrity checks. The other three carry no bypass language. Lantad has not measured any of these crawlers, so this reports what the documentation states rather than what the clients do. --- Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup. Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot