Does noindex block AI crawlers: 2 of 9 operator pages name the tag

Does noindex block AI crawlers: what the nine operator pages say

The sample is not a judgement about which crawlers matter. It is the set of documentation addresses stored in core/src/bots.ts, the registry behind our public crawler directory, which holds fifteen tokens across nine operators and one documentation URL for each. We fetched all nine on 9 September 2026 with this scanner's user agent, followed redirects, allowed forty five seconds, then stripped script and style blocks and searched both the extracted prose and the raw bytes for the string.

All nine answered HTTP 200. Two of the stored addresses no longer resolve to themselves: the Perplexity entry landed on docs.perplexity.ai/docs/resources/perplexity-crawlers, and the Google entry landed on the newer /crawling/docs/crawlers-fetchers/ path, which is drift in our own file rather than a finding about either company. Sizes ranged from 5,574 bytes at Meta to 471,873 at Amazon, and the relationship between bytes and words is not what a reader would guess: Amazon's near half megabyte of HTML yielded 4,463 characters of prose, while Meta's five kilobytes yielded 5,465.

Two pages contain the word. Apple's carries it four times and Amazon's once. Seven do not contain it at all, including OpenAI's crawler documentation, which describes four agents and their robots.txt tokens in 15,016 characters without once naming a meta tag. One of the seven is a special case worth stating rather than burying: the ByteDance page returned 91,455 bytes of HTML and no extractable prose, the same client rendered shell we recorded when we asked these nine pages whether their crawlers run JavaScript on 5 September. A directive could be documented there in text we cannot read, though the string does not appear anywhere in the bytes we received.

Two limits belong here rather than in a footnote. This survey reads the page our registry cites, not everything an operator has ever published, and Google is the clearest case: its crawler overview does not mention noindex, while a different Google page documents the tag at length and is handled in its own section below. And the registry stores Applebot-Extended rather than Applebot for Apple, so the page we read describes a crawler adjacent to the token we evaluate. Our scoring method is public for the same reason we are printing the denominators here.

The two pages that name the tag, and what each says it does

Apple's page is the most detailed treatment of on-page directives anywhere in this set, and it is five days old: the About Applebot page at support.apple.com/en-us/119829 carries a published date of September 04, 2026. It states that Applebot supports robots meta tags in HTML documents, shows where to put them, and then defines each one. Its noindex means Applebot will not index the page and the page will not appear in Spotlight or Siri Suggestions. Its nofollow means links on the page are not followed. Its none combines the two.

The interesting definition is nosnippet, because it is the only one in the set that reaches an AI output surface. Apple states that Applebot will not generate a description or web answer for the page, and adds that Apple will not use data tagged nosnippet as additional context and up-to-date content when AI models are used to generate output for display in Apple products and services. The same page documents the X-Robots-Tag response header for non-HTML resources such as PDFs and images, including a crawler scoped form written as X-Robots-Tag: applebot: nosnippet, and it accepts an equivalent scoped meta tag named for applebot rather than robots.

Apple also documents a structured data route to the same outcome. A page carrying the schema.org isAccessibleForFree property set to false stays eligible for search results, while Applebot will not use that content as additional context when AI models generate output. Apple states that this signal applies at page level and that section level markup using hasPart is not supported, which is the kind of boundary most vendor pages leave a reader to discover.

Amazon says far less and says something different. Its page at developer.amazon.com documents three user agents, Amazonbot, Amzn-SearchBot and Amzn-User, where our registry holds only Amazonbot. One sentence carries the whole of its position on page level controls: those user agents respect the link level rel=nofollow directive, and the page level robots meta tags noarchive, glossed there as do not use the page for model training, noindex, glossed as do not index the page, and none. The same sentence's neighbour states that they do not support the crawl-delay directive, which Anthropic's page does support, so two operators in one registry disagree about which non-standard extension is worth honouring.

That noarchive gloss is the single most useful line in the survey. It is the only place in these nine pages where an on-page tag is tied to model training, which is the outcome most people are reaching for when they write a directive at all. It is also a promise rather than an observation: nothing here measures whether any crawler behaved that way, and a token that sends no request of its own leaves nothing in a log to check, which is the trap we wrote up in the tokens that never appear in your logs.

A robots meta tag is read after the fetch it cannot prevent

Everything above concerns what a directive is claimed to do once it has been read. Whether it can be read at all is decided earlier, and the ordering is the part that decides what any of this is worth.

RFC 9309, the Robots Exclusion Protocol, defines rules that decide whether a crawler may access a URI. Access is its entire subject. A rule in that file is evaluated from a fixed path before any page request is made, which is why one fetch of one file can answer for a whole site. A meta tag is the opposite shape: it is bytes inside a document, so the crawler must request the page, receive the response and parse it before the directive exists as far as it is concerned. The retrieval has already happened.

Google states the general form of this in one sentence on its robots meta tag documentation, which says these settings can be read and followed only if crawlers are allowed to access the pages that include them. The practical consequence catches people out regularly: a Disallow rule and a noindex on the same URL do not reinforce each other, because the rule that stops the fetch also stops the tag being seen. Whichever half you meant, you get the half that runs first.

There is a second consequence for anyone trying to audit a site. A robots.txt position is one document at a known address, readable by anyone in one request. A tag position is one statement per URL, and surveying it means fetching every page. That asymmetry is why an external scanner can tell you your robots.txt stance in a second and cannot tell you your tag stance at all, and it is the same structural point we made when Microsoft's AI opt out turned out to be a meta tag rather than a robots.txt token. Google's own exclusion setting for its generative surfaces goes one step further and does not live on your site at all, which no external tool can verify either.

What the other seven pages document instead of a tag

Silence about noindex is not silence about control. Every readable page in the set documents something, and the something is almost always a name to write into robots.txt.

OpenAI's page separates its agents by purpose and states that each setting is independent of the others, so a site can allow the search agent and disallow the training one. Anthropic's crawler article states that its bots honour industry standard directives in robots.txt, supports the non-standard Crawl-delay extension with a worked example, and adds that its crawlers respect anti-circumvention technologies and will not attempt to bypass CAPTCHAs. Perplexity's crawler documentation lists its tokens and then documents Cloudflare and AWS WAF configuration and published IP addresses, which is a control surface sitting in front of the site rather than inside the page.

Common Crawl's CCBot page is the shortest useful document in the set at 2,678 characters. It gives one robots.txt example, warns that crawlers falsely identify themselves as CCBot, and shows a reverse DNS check with real output so a site owner can verify a logged request. Its archive is also the one in this group whose contents we have looked at directly, when it archived 2.14 billion pages without running JavaScript.

Meta's page is the one that addresses the question by refusing it. It states that Meta makes it easy for site managers and content owners to indicate their preferences by using industry-standard practices like robots.txt rather than non-standard formats like NoAI tags, at developers.facebook.com/documentation/sharing/webmasters/web-crawlers. Read against Apple's page, published in the same week, that is a genuine disagreement between two large operators about where a preference belongs, and a site owner cannot satisfy both with one edit.

The published IP address lists that three of these pages point at deserve one caution, since they are the usual next step for anyone verifying this in logs. When we compared thirteen of those files on 19 August 2026 we found six of them carrying two dates that disagree, so a list fetched today is not automatically a list maintained today.

Google binds nosnippet to AI Overviews, and noindex to search engine crawlers

Google is the operator whose answer the survey above cannot give, because the tag is documented away from the crawler overview our registry stores. The robots meta tag page, carrying a last updated date of 2026-03-24 UTC, is the reference, and two sentences on it settle the question for Google's surfaces.

The first is about scope. Google states that the noindex rule applies to search engine crawlers, and that blocking a non-search crawler may need a rule targeted at that crawler by name. Read literally, noindex is an instruction to a search index about a listing, not an instruction to a model about a document, and every use of it against an AI product depends on that product being fed by the search index in the first place.

The second is about which directive actually reaches the answer surface. Google's entry for nosnippet states that it applies to all forms of search results, naming web search, Google Images, Discover, AI Overviews and AI Mode, and that it will also prevent the content from being used as a direct input for AI Overviews and AI Mode. That is the clearest published link anywhere between the old snippet controls and a generative surface, and it names nosnippet rather than noindex. Anyone editing a tag to stay out of an AI answer at Google is editing the wrong word if they write noindex, which is why the platform guide for AI Overviews leads on snippet eligibility.

Worth keeping in proportion: almost nobody sets either. When we requested 120 home pages on 6 September 2026, none of the 81 that answered blocked their own snippet and two of them carried noindex. The directives in this post are a live control surface for a small number of publishers and a source of confident wrong advice for everybody else.

What this scanner does with the tag, which is nothing

The rule on this site is that a post about a control has to say what our own product does with it, especially when the answer is unflattering.

Our extractor collects every comma separated token from a robots meta tag, and the pipeline collects the tokens from an X-Robots-Tag response header, so both directive families reach the scoring inputs intact. From there, exactly two words are read. The Access sub-score deducts a fixed penalty when noai or nosnippet is present, and the same pair raises a defect in the report. That penalty is a setting we chose, ACCESS_PENALTIES.metaDirective in core/src/config.ts, not a measured effect of anything, and it is worth naming the difference because a weight and a finding look identical on a chart.

noindex is parsed into the same set and then read by nothing. No deduction, no defect, no row in the report. On the evidence in this post that omission is defensible for the AI surfaces, since seven of nine operators never mention the word, and indefensible as a description of a page: a page carrying noindex is being kept out of the index that most of these answer surfaces still draw from.

There is a sharper gap next to it. The extractor reads meta elements whose name is robots, and only those. Apple's crawler scoped form, a meta element named applebot, is not collected, and neither is the bingbot scoped tag Microsoft documents. So the two vendors in this survey with the most specific on-page instructions are the two our parser is least able to see, and a clean Lantad report says nothing whatever about either. Our crawler conduct policy and the crawler view of a page are both public, and this is the sort of limit that belongs alongside them rather than in a footnote.