# Can Meta crawl my website? Its own terms say yes from 30 October, and 2 of 1,011 robots.txt files named the crawler

> Meta's updated Commercial Terms, read at facebook.com/legal/commercial_terms_preview on 18 October 2026, carry an Effective Date of October 30, 2026 and authorise Meta to use scrapers, bots, crawlers and user agents against the pages a business associates with its products or services. The document in force until then contains no such clause, and the string robots.txt appears in neither. Lantad read all 1,419 hostnames in this repository's two committed corpus seed files on 18 October 2026 with no JavaScript executed: 100 of the 1,080 readable home pages carried an executing Meta business integration, and Meta-ExternalAds was allowed at the site root on 999 of the 1,011 robots.txt files that parsed.

- Canonical page: https://lantad.co/blog/can-meta-crawl-my-website-2-of-1011-named-the-crawler
- This file: https://lantad.co/blog/can-meta-crawl-my-website-2-of-1011-named-the-crawler.md
- Last substantive update: 2026-10-18

## Key facts

- **Published:** 2026-10-18
- **Category:** Findings
- **Author:** Lantad
- **Length:** 3967 words
- **Takeaway 1:** Meta's updated Commercial Terms, read at facebook.com/legal/commercial_terms_preview on 18 October 2026, carry the line "Effective Date: October 30, 2026" and authorise Meta to use automated software described as "web scrapers, bots, robots, spiders, crawlers, and user agents" against the domains, web pages and URLs associated with a business's products or services.
- **Takeaway 2:** Can Meta crawl my website becomes a contract question rather than a robots.txt question on 30 October 2026 for any business using Meta Company Products, because the string robots.txt appears in none of the four Meta terms documents Lantad read on 18 October 2026.
- **Takeaway 3:** The crawl authorisation is new to that document: the Commercial Terms in force on 18 October 2026, which carry an Effective Date of March 6, 2024, contain no instance of the words crawl, scrape, index or spider.
- **Takeaway 4:** A narrower version of the clause already bound advertisers: the Self-Serve Ad Terms dated August 22, 2025 grant the same rights on placing an Order, and that older wording names "research and training" where the new one names advertising and business products.
- **Takeaway 5:** Meta-ExternalAds, the crawler Meta's documentation ties to improving advertising and other business-related products, was allowed at the site root on 999 of the 1,011 robots.txt files Lantad parsed on 18 October 2026, and 2 of those files named it at all.

## Summary

Can Meta crawl my website has a new answer from 30 October 2026, and the answer does not come from robots.txt. It comes from a contract. Meta's updated Commercial Terms, published for preview at facebook.com/legal/commercial_terms_preview and carrying the line "Effective Date: October 30, 2026", add a paragraph that authorises Meta to point automated software at the web pages a business associates with its products or services. The version of that document in force when this post was written carries no such paragraph. Neither version contains the string robots.txt.

That makes most of this a reporting post rather than a measurement of Meta. Lantad has not observed Meta's crawlers on anybody's server, cannot see which businesses have accepted which terms, and claims nothing about either. What it can measure is the site-side half of the arrangement: how many sites carry evidence of using Meta's business products at all, and what their robots.txt currently says to the crawler Meta documents for the exact purpose the new clause names. Lantad requested the robots.txt and then the home page of all 1,419 hostnames in this repository's two committed corpus seed files on 18 October 2026, as [LantadBot](https://lantad.co/bot) with redirects followed, no JavaScript executed and no browser involved. Every count below comes from that one run, and the sampling frame is described on [the methodology page](https://lantad.co/methodology).

## What the new clause actually authorises

The paragraph sits in a section of the updated Commercial Terms headed "Licenses & Rights You Provide Meta". Read at facebook.com/legal/commercial_terms_preview on 18 October 2026, it reads: "You authorize us (or any entity acting on our behalf) to use automated software (such as web scrapers, bots, robots, spiders, crawlers, and user agents) and create test credentials to access, index, cache, analyze, collect, retrieve, or crawl the domains, web pages, or URLs (and any content available through them) that are associated with your products or services, or that are made available through your or users' use of Meta Company Products and services, and to periodically pull updates from such domains, web pages, or URLs in order to provide, develop, improve, research, and maintain advertising and business products ...".

Four details in that sentence matter more than its length. It authorises an agent as well as Meta, through the phrase "any entity acting on our behalf". It authorises the creation of test credentials, which is the only clause here that reaches past a public page and into whatever sits behind a login. It authorises repetition, through "periodically pull updates", so this describes a standing crawl rather than a one-off fetch. And it states a purpose that is commercial rather than scientific: providing, developing, improving, researching and maintaining advertising and business products.

What the clause does not say is as worth recording as what it does. The word training does not appear in it, and neither does any form of the phrase artificial intelligence. The clause is easy to read as a training licence, and the document does not say that: it states a commercial purpose, and the step from crawling a page to training a foundation model on it is not written here. The document does carry a separate section headed "AI Features", but that section is about AI features Meta makes available to the business, including an instruction not to remove content-authenticating metadata, and it is not a grant over crawled pages. Treating the purpose clause as a training clause would be the same category of error as reading a scoring weight as a finding, which is why [this site withholds a grade it did not measure](https://lantad.co/blog/why-we-withhold-a-grade).

The scope sentence is the part a site owner should read twice. The updated Commercial Terms apply to access or use of Meta Company Products for a business or commercial purpose, and the document lists what that includes: using ads or marketing messages, selling or listing products, developing apps, creating or using a product catalog, managing a Page, managing a Group for business purposes, using any business product such as Ads Manager or Meta Business Suite, or using Meta's measurement services. Managing a Facebook Page for a business is on that list. The population this clause reaches is therefore much wider than the population that buys ads.

## Can Meta crawl my website, and what decides it?

Two different instruments are in play and they answer to different standards. The contract is a permission: once it takes effect, a business using Meta Company Products has authorised the crawl in writing, and nothing on the server is needed to grant it. robots.txt is a request. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), the Standards Track specification for the Robots Exclusion Protocol, says in its own words that its rules are not a form of access authorization, and repeats in its security considerations that the protocol is not a substitute for valid content security measures. A Disallow line has never been a lock, and this is the clearest illustration yet of the gap: the permission and the refusal are written in different documents, by different parties, and only one of them is enforceable.

Meta's own documented position is that the request still works. Its crawler documentation says the company makes it easy for site managers to indicate their preferences "by using industry-standard practices like robots.txt rather than non-standard formats like NoAI tags". That page, at developers.facebook.com/docs/sharing/webmasters/web-crawlers and read on 18 October 2026, gives the syntax for blocking each of its crawlers by product token. So the practical answer to the question in this heading is that Meta says yes it may, and separately says a robots.txt rule will be honoured, while the contract that grants the permission does not mention robots.txt once.

Three further details on that documentation page decide how much a rule buys. It states that changes to robots.txt may take up to 24 hours to take effect because the crawlers may cache the file for that long, so a rule written the night before 30 October is not necessarily in force on the day. It states that Meta-ExternalFetcher may bypass robots.txt rules, because it fetches links at a user's request. And it states that FacebookExternalHit might bypass robots.txt when performing security or integrity checks. Two of Meta's five documented crawlers therefore carry published exceptions to the only control on offer, which is the same shape of problem as [a user agent being a claim rather than an identity](https://lantad.co/blog/a-user-agent-is-a-claim-not-an-identity).

None of this is unique to Meta, and it is worth saying so rather than implying a single bad actor. Six of ten assistants Lantad probed [never asked for robots.txt at all](https://lantad.co/blog/do-ai-crawlers-respect-robots-txt-six-of-ten-never-asked), fifteen sites have [reserved rights in their terms while leaving the crawlers allowed](https://lantad.co/blog/text-and-data-mining-opt-out-15-in-prose-none-in-a-file), 52 of 54 sites refusing Google-Extended [still allowed Googlebot](https://lantad.co/blog/does-blocking-ai-training-affect-search-rankings-52-of-54), and proposals such as [RSL](https://lantad.co/blog/rsl-license-directive-changes-no-access) and [ai.txt](https://lantad.co/blog/ai-txt-well-known-training-defaults-to-deny) exist precisely because robots.txt cannot express a party or a licence. What is new here is the direction of travel: a commercial agreement now supplies the permission that the protocol was never able to withhold.

## The crawler the clause describes is the ads crawler, not the training one

Meta documents five crawlers, and the one whose stated job matches the new clause is not the one most robots.txt files have heard of. Read at developers.facebook.com/docs/sharing/webmasters/web-crawlers on 18 October 2026, the page says Meta-ExternalAds "crawls the web for use cases such as improving advertising and other business-related products and services". The clause in the Commercial Terms authorises crawling "in order to provide, develop, improve, research, and maintain advertising and business products". Those two sentences describe the same activity in almost the same words.

Meta-ExternalAgent is the training crawler, documented for "use cases such as training foundation AI models or improving products by indexing content directly", and it is the Meta token this corpus names by far the most often, in 119 of the 1,011 files that parsed. Meta-WebIndexer is the citation crawler: the documentation says allowing it helps Meta cite and link to a site's content in Meta AI's responses, which makes it the one token on the list that a site chasing [AI visibility](https://lantad.co/glossary/ai-visibility) has a positive reason to permit. Meta-ExternalFetcher serves user-requested fetches and agentic tasks. FacebookExternalHit renders link previews.

The asymmetry between those five is the practical finding. A site that has carefully refused Meta-ExternalAgent has refused training and has said nothing at all about the crawl the new terms describe, because the terms describe advertising and business products and the token for that is a different string. This is the same failure mode measured in [the September count of Meta tokens in robots.txt](https://lantad.co/blog/meta-ai-crawler-6-of-1065-named-the-citation-token), where 42 of the 58 files that refused Meta's training crawler at the root still allowed its search crawler there, and it repeats an error this site made in its own reporting when it published that Meta documented one crawler and [its documentation named five](https://lantad.co/blog/meta-documents-five-crawler-tokens).

It is worth being plain about a limit on this scanner rather than only on other people's files. BOT_REGISTRY in core/src/bots.ts, the list behind [the crawler reference here](https://lantad.co/tools/ai-crawlers), carries 15 tokens and exactly one of them is Meta's, which is Meta-ExternalAgent. The robots verdicts in this post were computed by parsing each file with this repository's own parseRobotsTxt and evaluating it with evaluateRobots against the token string directly, so the Meta-ExternalAds figures below are real evaluations rather than registry lookups, but the product itself does not yet report on that token. OpenAI's equivalent ads crawler was measured the same way: [OAI-AdsBot appeared in 4 of 1,014 files](https://lantad.co/blog/oai-adsbot-4-of-1014-robots-txt-files-named-it).

## How many sites sit inside the clause's reach

The clause binds businesses that use Meta Company Products, and no outside observer can read a contract list. What can be read from the outside is the footprint those products leave on a page, so that is what Lantad counted. Of the 1,419 hostnames requested on 18 October 2026, 1,080 answered HTTP 200 with HTML to a plainly identified crawler, and those 1,080 are the denominator for everything in this section.

100 of the 1,080 carried an executing Meta business integration in the bytes a crawler received: a script from connect.facebook.net on 89 of them, the Meta Pixel specifically on 61 counted as any of fbevents.js, an fbq call or a request to the facebook.com/tr pixel endpoint, and a business.facebook.com reference on one. A further 71 declared an fb:app_id meta tag and nothing executable, which associates the page with a Meta app without sending anything, and that is a weaker signal kept in its own row rather than folded into the first. Another 624 linked a Meta surface and did neither, counting facebook.com, instagram.com, threads and the wa.me and api.whatsapp.com endpoints. 795 of the 1,080 pages carried at least one of those three things and 285 carried none.

Two comparisons keep that first number in proportion. 608 of the same 1,080 pages carried Google Tag Manager or a gtag call, so Meta's executing footprint at 100 is about a sixth of Google's on the same pages and an integration of this kind is ordinary rather than remarkable. And the distribution is uneven in a way that matches what the strata are for: 7 of 30 local media sites, 7 of 34 Shopify storefronts and 8 of 41 small-business WordPress sites carried the executing integration, against 3 of 92 finance sites, 3 of 85 government sites and none at all of the 31 documentation sites. A static docs site has no catalog and no Page, and it shows.

The honest reading of 100 is that it is a floor rather than an estimate. No JavaScript was executed in this run, so a pixel injected at runtime by a tag manager or a consent platform was invisible to it, and consent platforms are not rare: a previous run found [269 of 1,093 home pages running one](https://lantad.co/blog/do-cookie-banners-block-ai-crawlers-269-of-1093). One page per site was read, so an integration present only on a checkout or a product page was missed. The footprint also runs the wrong way round as evidence: it shows the business uses a Meta product, which is what brings the terms into play, but a site with no footprint may still have a Page or an ad account that leaves no mark on its home page at all.

## What those robots.txt files say to Meta-ExternalAds

1,011 of the 1,419 hostnames served a robots.txt that parsed into at least one user-agent group. Evaluating each of those files for the token meta-externalads at the path / with this repository's own evaluator returns allowed on 999 of them and disallowed on 12. Two files in the whole set name the token at all. Token matching is case-insensitive both in the specification and in this evaluator, a detail measured on its own when [84 mis-cased crawler tokens all bound anyway](https://lantad.co/blog/is-robots-txt-case-sensitive-84-tokens-bound-9-pages-open), so the count of two is not an artefact of how the string was spelled.

The composition of those 12 is the finding rather than the 12 itself. Eleven were decided by a group that never mentions Meta: a wildcard group carrying Disallow: /, which shuts out every crawler that does not have a group of its own, on hosts including amsterdam.nl, wa.gov, helsinki.fi, sciencedirect.com, theregister.com, sap.com and instacart.com. Those sites have not ruled on Meta's ads crawler; they have ruled on everything, and this crawler falls under it. Only eluniversal.com.mx refused Meta-ExternalAds through a group that names it, and that file names all five Meta tokens with a Disallow: / under each, which is what a deliberate decision looks like. The same wildcard arithmetic has turned up every time this corpus has been measured, most starkly when [559 of the 581 paths GPTBot lost were closed by a rule that never named it](https://lantad.co/blog/blocked-by-robots-txt-559-of-581-by-a-wildcard).

slate.com is the one file that names Meta-ExternalAds and lets it through, and it does so in a way that reads as considered rather than accidental. Its robots.txt is 11,372 bytes in four groups. One group heads 97 tokens, Meta-ExternalAgent among them, with a single rule: Disallow: /. A separate group heads 328 tokens, Meta-ExternalAds among them, with eight narrow disallows covering /search, /comments/, /_, /css, /fonts and /media, and no root block. The same publisher that shuts the training crawler out of the entire site gives the ads crawler everything except its search pages and its asset directories. Whether that is the intended split only slate.com can say, but the file distinguishes between the two crawlers, which 1,009 of the 1,011 files do not.

Narrowing to the sites with the strongest reason to care does not change the picture. Of the 100 pages carrying an executing Meta integration, 89 also served a parseable robots.txt. 15 of those 89 name any Meta token, one names Meta-ExternalAds, and the token is allowed at the site root on 87 of the 89. Meta-ExternalAgent, the training token, is refused at the root on 5 of them. So among the sites whose own pages show they use Meta's business products, which is the population the new clause is written for, the crawl the clause describes is permitted almost universally, and in all but one case by silence rather than by choice. A file can be silent and still be scored, which is why [a robots.txt that blocks every citation crawler can still grade B](https://lantad.co/blog/a-robots-txt-that-blocks-every-citation-crawler-still-grades-b) on a scale that weights four things.

## What this measurement does not show

The terms are reported here, not measured. Lantad opened four Meta documents and a documentation page on 18 October 2026 and recorded what they say, their effective dates and, in the case of the string robots.txt, what they do not contain. It has not seen Meta's crawlers arrive anywhere, has counted no request from any Meta user agent in any log, and has no way to know which of these businesses has accepted which agreement or when. Every sentence above about what Meta will do is Meta's own published wording.

The footprint count is evidence of product use and not of contract acceptance, and the two are not the same set. A Meta Pixel on a page says a business sends event data to Meta. The Commercial Terms bind the party that uses the product, which may be an agency, a parent company or a franchisor rather than the operator of the domain the pixel sits on, and the clause reaches the pages associated with that party's products or services rather than the pages that happen to carry its code. There will be sites in the 285 with no footprint that are squarely inside the clause and sites in the 100 that are not.

Three mechanical limits apply to the run itself. No JavaScript was executed, which makes 100 a floor. One page per hostname was read, from one network location, with a twenty second timeout, so a slow or geofenced site counts as unreadable rather than as anything else. And 339 of the 1,419 hostnames produced no readable home page at all, which is a large enough share that the 1,080 should be read as the pages that answered rather than as the corpus.

Finally, a robots.txt verdict is a statement about a file and not a prediction about a fetch. The evaluator used here implements RFC 9309 with the Google-documented refinements, and a crawler that merges groups differently, caches the file for 24 hours or carries a documented bypass will behave differently from the verdict. That distance between a file and a fetch is the thing this scanner exists to measure, and it is the reason [the robots.txt tester](https://lantad.co/tools/robots-txt-tester) resolves a real path against a real file rather than summarising the file, and why [what GPTBot sees](https://lantad.co/tools/what-gptbot-sees) answers for one named agent at a time. The wider corpus numbers these figures sit beside are on [the research page](https://lantad.co/research).

## Questions and answers

**Can Meta crawl my website without my permission?**

From 30 October 2026 the permission is in the contract rather than on the server, for businesses using Meta Company Products. The updated Commercial Terms at facebook.com/legal/commercial_terms_preview, read on 18 October 2026, authorise Meta and any entity acting on its behalf to use scrapers, bots, crawlers and user agents against the domains, web pages and URLs associated with a business's products or services, and to pull updates periodically. The document lists managing a Page and using a product catalog among the business purposes that bring it into effect, so it is not limited to advertisers. The version in force until then, dated March 6, 2024, contains no instance of the words crawl, scrape, index or spider.

**Does blocking Meta-ExternalAgent stop the crawl the new terms describe?**

Not on the documentation as written. Meta-ExternalAgent is documented for training foundation AI models and indexing content directly. The new clause authorises crawling in order to provide, develop, improve, research and maintain advertising and business products, and the crawler Meta documents for that purpose is Meta-ExternalAds, a different product token. Of the 1,011 robots.txt files Lantad parsed on 18 October 2026, 119 named Meta-ExternalAgent and 2 named Meta-ExternalAds, and Meta-ExternalAds was allowed at the site root on 999 of the 1,011.

**Which Meta crawler should I rule on in robots.txt?**

All five, deliberately, because their documented purposes differ and a rule for one says nothing about the others. Meta's web crawler documentation names FacebookExternalHit for link previews, Meta-WebIndexer for Meta AI search quality and citation, Meta-ExternalAds for advertising and business products, Meta-ExternalAgent for training, and Meta-ExternalFetcher for user-requested fetches. The documentation states that Meta-ExternalFetcher may bypass robots.txt and that FacebookExternalHit might bypass it during security or integrity checks, and that changes can take up to 24 hours to take effect because the file may be cached. Note that refusing Meta-WebIndexer is the one choice with a stated cost to visibility, since Meta's page says allowing it helps Meta cite and link to your content.

**Was the crawl permission always there, or is this new?**

It existed in a narrower form and is being widened. The Self-Serve Ad Terms in force on 18 October 2026, dated August 22, 2025, already granted Meta the right to access, index, cache, analyze or crawl content on a business's domains, and that wording is triggered by placing an Order and explicitly names research and training among its purposes. The change is that from 30 October 2026 the authorisation also sits in the Commercial Terms, which apply to business or commercial use of Meta Company Products generally rather than to placing an ad order, and the new wording names advertising and business products where the old one named training.

---

Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering
crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup.

Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot