BlogFindings
Robots.txt noindex: 7 of 1,057 files wrote the rule Google retired in 2019
Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 13 October 2026. 1,057 returned a text file. Seven of them carried a Noindex line, 33 lines between them, and this scanner's own evaluator confirmed that 12 of those lines name a path GPTBot is still allowed to fetch.
So we counted. On 13 October 2026 we requested https://hostname/robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files, as LantadBot/1.0, over HTTPS, redirects followed, twenty second timeout, from one network location, with no browser involved. 1,092 answered HTTP 200. Thirty five of those served an HTML page at that path rather than a text file, which leaves 1,057 real robots.txt files. We then read every one of them with this scanner's own parser, the same parser whose behaviour our methodology sets out, and which decides the access verdicts in a report, and tallied every field name appearing before a colon. Seven files carried a Noindex line. The interesting part is not that seven is a small number: it is what the other rules in those same seven files do, and do not do, about the paths the Noindex lines name.
In short
- Robots.txt noindex is a rule Google said in July 2019 it was retiring all code for on September 1, 2019, and on 13 October 2026 Lantad found 7 of 1,057 robots.txt files still writing it, 33 lines between them.
- 21 of those 33 Noindex lines were backed by a Disallow for the same path in the same file, so the path was closed regardless. The remaining 12 lines, spread over 5 of the 7 hosts, named paths this scanner's own evaluator then confirmed GPTBot may fetch.
- 217 of the 1,057 files carried at least one field outside the three RFC 9309 defines, 13 distinct field names in all, led by crawl-delay on 151 files, Cloudflare's content-signal on 35 and Host on 27.
- Nine sites in this corpus have invented an AI-era robots.txt field: LLMs or LLM pointing at an llms.txt on four, Agentmap pointing at an ai-catalog.json on two, and License, Link, Citation-Policy and Training-Window on one each.
- Lantad's own parser records every one of those lines as an unknown field and ignores it, which is what RFC 9309 section 2.2.4 permits and what Google's robots.txt documentation, last updated 2026-08-31 UTC, states Google does.
| Stage | Count | What happened |
|---|---|---|
| Hostnames asked | 1,419 | Two committed corpus seed files, stratified by industry and by platform |
| Answered HTTP 200 | 1,092 | 177 answered 403, 51 a 404, 49 a 503, 31 never answered at all, 19 returned another status |
| Delivered a text file | 1,057 | 35 served an HTML page at /robots.txt instead |
| Used only the specified fields | 840 | user-agent, allow, disallow, and the Sitemap line Google documents |
| Carried some other field | 217 | 13 distinct field names, none of them defined by RFC 9309 |
| Carried a Noindex line | 7 | 33 lines in total |
| Noindex lines a Disallow also covered | 21 | The path was closed by a rule that works |
| Noindex lines on an open path | 12 | On 5 hosts. GPTBot is allowed to fetch every one of them |
Does robots.txt noindex do anything?
No, and the clearest statement of that comes from the company that used to handle it. Google's July 2019 note on unsupported rules in robots.txt says it focused on "rules unsupported by the internet draft, such as crawl-delay, nofollow, and noindex", and then: "In the interest of maintaining a healthy ecosystem and preparing for potential future open source releases, we're retiring all code that handles unsupported and unpublished rules (such as noindex) on September 1, 2019." The same post is worth reading for its own measurement, which is the mirror of ours: "we saw their usage was contradicted by other rules in all but 0.001% of all robots.txt files on the internet". That figure counts robots.txt files in which other rules contradicted the unsupported one. Ours counts something narrower on a far smaller and differently drawn sample: of 33 Noindex lines, 12 named a path the same file left open.
Current documentation says the same thing without the history. Google's robots.txt specification page, last updated 2026-08-31 UTC, states that "Google supports the following fields (other fields such as crawl-delay aren't supported)" and lists exactly four: user-agent, allow, disallow and sitemap. The underlying standard is narrower still. RFC 9309, the Robots Exclusion Protocol, published September 2022, defines user-agent in section 2.2.1 and allow and disallow in section 2.2.2, and that is the whole grammar. Sitemap is not in it either: section 2.2.4 offers Sitemaps as an example of records a crawler "MAY interpret", not as a defined field.
None of this is the same question as whether the noindex meta tag works, which it does, and which we have measured separately: of the nine vendor documentation pages behind the crawler tokens this scanner evaluates, two mention noindex at all, and the tag has to be parsed out of a response body that has already been fetched. Of the AI vendors, OpenAI's crawler documentation, read on 13 October 2026, contains the string noindex zero times and tells site owners to allow or disallow each bot by name instead. We have also found the tag in its header form on live sites: 5 of 391 home pages sent an X-Robots-Tag noindex, and all five were a captcha rather than a page.
| Field | Defined by RFC 9309 | Listed by Google as supported | What Lantad's parser does |
|---|---|---|---|
| user-agent | Yes, section 2.2.1 | Yes | Opens a group |
| disallow | Yes, section 2.2.2 | Yes | Becomes a rule |
| allow | Yes, section 2.2.2 | Yes | Becomes a rule |
| sitemap | No, named in 2.2.4 as an example of other records | Yes | Collected, never a rule |
| crawl-delay | No | No, named on the page as unsupported | Read and discarded |
| noindex | No | No, retired 1 September 2019 | Unknown field, ignored |
| anything else | No | No | Unknown field, ignored |
The seven files that still write Noindex
They are census.gov, whoi.edu, mskcc.org, carrefour.fr, wayfair.com, zappos.com and fidelity.com: a United States federal statistics agency, an oceanographic research institution, a cancer centre, a French grocer, two large retailers and a broker. Four of the seven are in this corpus as ecommerce or finance, which is the population where a site has thousands of filter and parameter URLs it would rather not have indexed, and that is exactly the problem a Noindex line looks like it solves.
The line counts are uneven. whoi.edu carries 11, more than any other file, and they are a coherent set: /?s=, /search/ and /mywhoi/, a /cruiseplanning/files/ directory, then seven paths ending in the .do suffix of a Java servlet application, written twice over in two capitalisations for /preview.do and /Preview.do. zappos.com carries 6, wayfair.com 6, carrefour.fr 5, mskcc.org 2, fidelity.com 2 and census.gov 1. Two of the lines, both on wayfair.com, are written as "Noindex : /roomplanner/*" with a space before the colon, which means the field name as written is "Noindex " and not "Noindex" at all. That detail changes nothing, since neither spelling is read, but it is a fair summary of how much attention these lines get after the day they are added.
The shape of what they name is consistent and sensible. Search result pages, faceted filter paths, a room planner, a store selector, two configuration files on mskcc.org that nobody should be reading, a PDF of cash offer terms on fidelity.com. These are the URLs a crawlability study flags as low value, and the intention behind every one of the 33 lines is legible. The question this post can answer is narrower than the intention: not whether the site was right to want these paths out of an index, but whether the file it wrote keeps anything out of them. For 21 of the 33 lines the answer is yes, by accident of a second rule. For 12 it is no.
| Host | Noindex lines | Also disallowed | Left open |
|---|---|---|---|
| whoi.edu | 11 | 3 | 8 |
| wayfair.com | 6 | 5 | 1 |
| zappos.com | 6 | 5 | 1 |
| carrefour.fr | 5 | 5 | 0 |
| mskcc.org | 2 | 2 | 0 |
| fidelity.com | 2 | 1 | 1 |
| census.gov | 1 | 0 | 1 |
Twelve lines name a path a crawler may still fetch
This is the part a count of Noindex lines cannot tell you, and it is why the measurement was worth doing with the product's own code rather than a grep. For each Noindex line we took the path it names, substituted a literal character for any wildcard to get a concrete URL, and asked evaluateRobots what the rest of that same file says about fetching it as GPTBot. The answer came back allowed 12 times.
The clearest case is census.gov, whose file carries exactly one Noindex line, "Noindex: /quickfacts/", and no Disallow that covers it. QuickFacts is a United States Census Bureau data product, and the single instruction the site wrote about it is one no crawler reads. The widest case is whoi.edu: of its 11 lines, the three naming /?s=, /search/ and /mywhoi/ are each duplicated by a real Disallow for the same path, while /cruiseplanning/files/ and the seven servlet paths are not, so the three paths that were going to be protected anyway got the belt and braces, and the eight that depended on the Noindex line got nothing. On fidelity.com the protected path is a document directory with a working Disallow, and the unprotected one is a single PDF. On wayfair.com and zappos.com one line each slipped through a near miss: wayfair writes "Noindex: /*quick_view" where its Disallow rules cover /roomplanner/ and */filters/ but no quick_view pattern, and zappos writes "Noindex: /*?oosRedirected=" while disallowing the neighbouring /*?plsr= and /*&plsr=.
Two of the seven files are fine. carrefour.fr and mskcc.org wrote a Disallow beside every Noindex they wrote, which is the arrangement Google's 2019 figure found in all but 0.001 percent of the files it looked at. On this sample a Disallow sat beside every Noindex on 2 of 7 hosts, and beside 21 of 33 lines. That gap is not a correction of Google's figure, which was measured on a different and vastly larger population seven years earlier by a method it does not publish, and the two numbers are not comparable. It is a reason to check your own file rather than assume the pattern holds, which is what our robots.txt tester and what GPTBot sees exist to do. A path left open by a rule nobody reads is the same failure mode as a path closed by a wildcard rule that never named the crawler: in both cases the file says one thing and does another.
evaluateRobots(parsed, "gptbot", path)
- census.gov Noindex: /quickfacts/ allowed, no rule matched
- whoi.edu Noindex: /cruiseplanning/files/ allowed, no rule matched
- whoi.edu Noindex: /preview.do* allowed, no rule matched
- whoi.edu Noindex: /Preview.do* allowed, no rule matched
- whoi.edu Noindex: /live.do* allowed, no rule matched
- whoi.edu Noindex: /Site.do* allowed, no rule matched
- whoi.edu Noindex: /viewArticle.do* allowed, no rule matched
- whoi.edu Noindex: /pdfserver.do* allowed, no rule matched
- whoi.edu Noindex: /pr.do* allowed, no rule matched
- wayfair.com Noindex: /*quick_view allowed, no rule matched
- zappos.com Noindex: /*?oosRedirected= allowed, no rule matched
- fidelity.com Noindex: /misc/ekits/pdf/cash_offer_terms.pdf allowed, no rule matched
Nine sites have invented a robots.txt field for AI
Noindex is a rule that used to be read and is not any more. The more interesting group in this corpus is the opposite: fields nothing has ever read, written in the last year or two to say something about AI. Nine of the 1,057 files carry one.
Four point at an llms.txt. livechat.com, visitdubai.com and td.com each write a single "LLMs:" line naming their file, and australia.com writes 18 "LLM:" lines, of which 15 name a locale path such as /de-de/llms.txt or /ja-jp/llms.txt, one names a file on australia.cn, and two name /en/llms.txt and the root /llms.txt. The intent is discovery, and the llms.txt specification does not provide for it that way: the file is found at "the root path /llms.txt of a website or at any subpath", by convention rather than by announcement, and where the spec does recommend a pointer it is a link relation rather than a robots.txt field, stating that rel="describedby" points to the llms.txt file that covers it, and that such links can be provided as HTML link elements or as an HTTP Link header. Two sites, zapier.com and blindpay.com, write "Agentmap:" lines naming a /.well-known/ai-catalog.json, and the same specification explicitly considered and rejected /.well-known/ for this purpose, on the grounds that "well-known URIs exist only at the origin root". wizzair.com goes furthest and writes three Link lines inside robots.txt, with rel values of "ai-policy" and "machine-readable", which is the right header in the wrong file. thefurrow.tv writes "Citation-Policy: allow-with-attribution" and "Training-Window: 30d", and unesco.org writes "License:" pointing at its terms of use page.
Each of these is a reasonable thing to want and none of them is a thing robots.txt can do. The honest versions already exist and we have measured them: an llms.txt at the conventional path, which our llms.txt checker reads, and which 19 of the 23 sites publishing one pair with a robots.txt that says nothing about AI; ai.txt at /.well-known/, whose 22 policy fields default Training to deny; and the RSL licence vocabulary, which writes a licence into robots.txt and says in its own documentation that it changes no access. Lantad's parser treats every line in this paragraph the same way: an unknown field, recorded in parseNotes, ignored for every verdict. That is not a gap in the parser. RFC 9309 section 2.2.4 says a crawler "MAY interpret other records that are not part of the robots.txt protocol", which makes reading them optional for everyone, and adds that parsing them "MUST NOT interfere with the parsing of explicitly defined records".
| Host | Field | Lines | What the value names |
|---|---|---|---|
| australia.com | LLM | 18 | 15 locale paths, one file on australia.cn, plus /en/ and the root |
| livechat.com | LLMs | 1 | Its own llms.txt |
| visitdubai.com | LLMS | 1 | Its own llms.txt |
| td.com | LLMS | 1 | Its own LLMS.txt, capitalised in the path too |
| zapier.com | Agentmap | 1 | /.well-known/ai-catalog.json |
| blindpay.com | Agentmap | 1 | /.well-known/ai-catalog.json |
| wizzair.com | Link | 3 | An ai.txt and two markdown files, with rel values |
| thefurrow.tv | Citation-Policy, Training-Window | 2 | allow-with-attribution, and 30d |
| unesco.org | License | 1 | Its terms of use page |
217 of 1,057 files carry a field no crawler owes them
The full census puts Noindex in perspective. Of the 1,057 files, 840 use nothing but user-agent, allow, disallow and Sitemap. The other 217 carry at least one of 13 further field names between them, and the distribution is steeply skewed: crawl-delay accounts for 151 of those files and 304 lines, which we have measured on its own and found that 152 of 1,056 sites set one and 11 gave it to an AI crawler. Cloudflare's content-signal appears on 35 files, and we have shown that the Content-signal line asks while the Disallow lines under it block.
Third by frequency, and almost unremarked, is Host on 27 files. Twenty six of the 27 values name the file's own hostname, so the line asserts nothing and could be deleted with no effect on anything. One does not: getalembic.com writes "Host: https://alembic.com", naming a different hostname from the one serving the file, which is the only Host line in this corpus that would change something if anything read it. There is a second oddity worth stating plainly, because it is the kind of detail that reveals how a line was added rather than written: 26 of the 27 values carry an https:// scheme, which makes them URLs rather than hostnames, and only buildkite.com writes a bare "Host: www.buildkite.com". All 27 sit in the SaaS, finance, travel, Framer, Webflow, single page app and documentation strata of this corpus, and not one appears on a WordPress, Drupal, Shopify, Wix or Squarespace site, which is consistent with a template shared between hand-built marketing sites rather than a setting any content management system offers.
A fourth group is not a field at all. Forty five of the 1,057 files parse into no user-agent group, meaning they contain no rules of any kind: nine are zero bytes, seven hold only comments, and 29 hold only a Sitemap line. A file like that permits everything, which may well be the intention, and it is the same outcome as a 404 on robots.txt, which allows every crawler. Two lines on amsterdam.nl are bare URLs with no field name, which our parser reads as a field called "https" and discards. The Sitemap line itself remains the one non-RFC field with near universal support, declared by 839 of 1,016 files when we counted it, and the reason it works is that Google, Bing and the rest each chose to read it, not that a standard told them to.
What to check on your own robots.txt
The practical lesson from 33 lines on seven sites is not about Noindex specifically. It is that robots.txt gives no feedback. A misspelled Disallow, a directive from another search engine, a field invented last year and a field retired in 2019 all look identical in the file and all produce the same silence. The only way to know what your file does is to evaluate it the way a crawler would, for the token you care about, against the path you care about.
Three checks follow from this measurement. First, for every path you believe robots.txt is protecting, confirm a Disallow covers it, in the group that matches the crawler you mean, and not merely a line that mentions the path. Five of the seven sites here failed that check on at least one path. Second, if the goal is to keep a page out of an index rather than out of a crawl, robots.txt is the wrong file entirely, and the 2019 Google note says so directly: the noindex robots meta tag is "the most effective way to remove URLs from the index when crawling is allowed", with 404 and 410 given as the alternatives. A disallowed URL cannot be read, so it also cannot be read to discover that you wanted it dropped. Third, if what you want to say is about AI specifically, say it where something reads it. Our AI crawler list gives the current tokens, LantadBot's own page is the model for publishing what a crawler does, and the per platform guides such as how to get cited by ChatGPT and how to get cited by Claude set out what each vendor actually honours. A token-by-token reading matters because 1,004 files in this corpus named 2,209 distinct user-agent tokens, and a rule aimed at the wrong one is as quiet as a Noindex line.
What this post does not establish is worth stating. No AI crawler was observed doing anything: every verdict here is this scanner's own evaluation of a text file, using the GPTBot token, and a crawler that ignores robots.txt altogether would make all 33 lines equally irrelevant. We asked only for /robots.txt and evaluated only the paths those files name, so nothing here says whether the pages behind them are reachable in practice, and robots.txt permission predicts service poorly in any case: 79 of 115 hostnames we measured earlier refused a crawler their own file permits. The 1,057 files are 1,057 front doors from two committed sampling frames, not a sample of the web, so no proportion above generalises beyond them.
Flow: A line in robots.txt to Field name recognised?; Field name recognised? (no) to Unknown field, ignored; Field name recognised? (yes) to In a group matching the crawler's token?; In a group matching the crawler's token? (no) to No group, no effect; In a group matching the crawler's token? (yes) to Becomes an allow or disallow rule; Becomes an allow or disallow rule to Pattern matches the path?; Pattern matches the path? (no) to Path stays open; Pattern matches the path? (yes) to Crawler is refused.
Lantad
Published .
Robots.txt noindex is the shortest way to describe a thing that does not exist. A site owner who wants a path kept out of an index, and who knows robots.txt is the file where crawler instructions live, writes the word there. The file accepts it, because robots.txt is a text file and text files accept anything. Nothing downstream is obliged to read it, and the three parsers that publish a field list do not. The question matters more now than it did in 2019, because the population of clients fetching that file has grown: an AI crawler is the newest reader of robots.txt and the least likely to implement anything beyond the specified minimum.
Common questions
Does noindex work in robots.txt?
No. Google's July 2019 note on unsupported rules in robots.txt states that it was "retiring all code that handles unsupported and unpublished rules (such as noindex) on September 1, 2019", and its current robots.txt documentation, last updated 2026-08-31 UTC, lists only user-agent, allow, disallow and sitemap as supported fields. RFC 9309 defines only the first three. Lantad's parser records a Noindex line as an unknown field and ignores it, so it changes no access verdict in a report.
How many sites still use noindex in robots.txt?
In this corpus, 7 of the 1,057 robots.txt files that Lantad read on 13 October 2026, carrying 33 Noindex lines between them. They are census.gov, whoi.edu, mskcc.org, carrefour.fr, wayfair.com, zappos.com and fidelity.com. That is a count of 1,057 front doors drawn from two committed sampling frames, not an estimate for the web.
What should I use instead of robots.txt noindex?
Google's 2019 note names the alternatives directly: the noindex robots meta tag, which it calls "the most effective way to remove URLs from the index when crawling is allowed" and which works in both an HTTP header and the HTML, and 404 or 410 status codes. The distinction that matters is crawl against index. A Disallow in robots.txt stops the fetch, and a page that is never fetched cannot be read to discover that you wanted it dropped from an index.
Can I use robots.txt to set an AI policy or point at my llms.txt?
Nothing reads it if you do. Nine sites in this corpus try, with fields named LLMs, LLM, Agentmap, Link, License, Citation-Policy and Training-Window. The llms.txt specification locates the file by convention at /llms.txt or a subpath and recommends a rel="describedby" link relation in HTML or an HTTP Link header, not a robots.txt line. RFC 9309 section 2.2.4 makes reading any other record optional for every crawler.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
llms.txt
llms.txt is a proposed plain-text markdown file placed at a site's root that gives AI systems a curated map of its most important content, so models can find and read what matters without parsing every page.
Does noindex block AI crawlers: 2 of 9 operator pages name the tag
Lantad read the nine vendor documentation pages behind the fifteen AI crawler tokens it evaluates on 9 September 2026. The word noindex appears in the bytes of two of them, Apple's and Amazon's, and on both it governs indexing rather than the fetch.
Crawler user agent: 1,004 robots.txt files named 2,209 tokens, and 120 name one Anthropic no longer documents
A user-agent line in robots.txt is an exact product token, and a token that matches nothing is a rule that does nothing. Lantad requested /robots.txt once from each of 1,419 hostnames on 17 September 2026. 1,004 files parsed into at least one group and named 2,209 distinct tokens between them, 1,098 of which appear in a single file, 813 of which contain a character RFC 9309 does not let a crawler put in its own name, and 120 of which name anthropic-ai or Claude-Web, neither of which appears on Anthropic's crawler page.
Cloudflare's Content-signal line asks, and the Disallow lines under it block
Cloudflare's managed robots.txt writes one Content-signal line and eight Disallow rules into a site's file. We parsed the published example on 30 July 2026: the signal was recorded as an unknown field and ignored, and the Disallow rules blocked 8 of the 15 crawler tokens in our registry.
A 404 on robots.txt allows every crawler, a 503 blocks them all
RFC 9309 gives a missing robots.txt and a broken robots.txt opposite meanings: a 4xx lets a crawler access any resource on the server, a 5xx obliges it to assume complete disallow. Google's documentation carves 429 out of that rule. Lantad's code does not.
