BlogFindings
Robots meta tag: 629 of 1,023 home pages carried no page level directive at all
Lantad requested robots.txt and then the home page of all 1,419 hostnames in this repository's two committed corpus seed files on 2 November 2026, as LantadBot/1.0 with redirects followed and no JavaScript executed. 1,023 answered HTTP 200 with HTML. 629 of those carried no robots meta tag, no crawler named meta tag and no X-Robots-Tag header, so there was no page level carrier on the page for any directive at all. 380 carried a generic robots meta tag, 33 carried one addressed to a named crawler, and 12 sent the header.
So the question this post answers is narrow and, as far as this blog can tell, unmeasured: how many sites serve any of those three carriers in the first place. Not which directives they chose, which Lantad has measured for noarchive and for an X-Robots-Tag noindex before, but whether there is anywhere on the page for a directive to sit. On 2 November 2026 the answer across 1,023 readable corpus home pages was that 629 of them had nowhere. That is not a finding about intent. A site with no robots meta tag has usually decided nothing rather than decided to allow, and the distinction matters because the defaults are not the same everywhere and the carrier is becoming the thing vendors ask for. This matters more than it did a week ago, because Microsoft published a page level only grounding preference on 30 October 2026, which is reported separately and is not re-reported here. The measurement below is the site side of that, and it is the only part of it Lantad can measure.
In short
- Lantad measured 629 of 1,023 readable corpus home pages carrying no page level robots directive carrier of any kind on 2 November 2026: no robots meta tag, no meta tag addressed to a named crawler, and no X-Robots-Tag response header.
- The robots meta tag is the carrier that does exist where any does: 380 of the 1,023 pages carried a generic one, against 33 carrying a meta tag addressed to a named crawler and 12 sending an X-Robots-Tag header.
- Almost everything those carriers said was permissive. Across the 1,023 pages the directives counted were index 315 times, follow 304, max-image-preview 224, max-snippet 129 and max-video-preview 127, while only 27 of the 1,023 pages carried any restrictive directive at all.
- Google's robots meta tag documentation, read at source on 2 November 2026, lists noarchive and nocache under Reference of historical and other unused rules and states that the noarchive rule is no longer used by Google Search to control whether a cached link is shown in search results.
- 7 of the 1,023 pages carried noarchive or nocache on 2 November 2026: six sent noarchive in a generic robots meta tag, and the one page sending nocache sent it in a meta tag addressed to bingbot rather than to robots.
What was measured, and what the population actually is
The frame is the two committed corpus seed files in this repository: worker/seeds/corpus-seeds-platform.json, which holds 392 hostnames in ten platform strata and carries a generated date of 2026-07-30, and worker/seeds/corpus-seeds-industry.json, which holds 1,027 hostnames in eight industry strata and carries 2026-08-03. Together they are 1,419 distinct hostnames. On 2 November 2026 each one was asked for https://host/robots.txt and then, where that file left the site root open to this scanner, for the home page, every request sent as LantadBot/1.0 (+https://lantad.co/bot) from one network location with redirects followed, no JavaScript executed and no browser involved. The identity the requests carried is published at the scanner's bot page.
The attrition between 1,419 and the 1,023 pages the figures rest on is worth stating, because a percentage quoted against the wrong denominator is the most common way a corpus study misleads. 11 hosts closed the site root to LantadBot in robots.txt and their home pages were not fetched. Of the 1,408 attempted, 105 did not answer at all, failing on DNS, on TLS or by timing out at fifteen seconds, and 280 answered with a status other than 200. That leaves 1,023 responses that were HTTP 200 and carried an HTML content type, and every figure in this post is a count over those 1,023. The share of a corpus that simply does not answer a plainly identified crawler is itself a recurring measurement here and is written up in the crawlability study.
Each of those 1,023 responses was then read twice: the response headers for an X-Robots-Tag, and the served HTML for every meta element carrying both a name and a content attribute. A meta tag was counted as a robots directive carrier only when its name was robots or the name of a crawler, which is the part of the method that needed care. Classifying by the content string instead produces false positives that look convincing in a table: a Parse.ly meta tag named parsely-type with the content index, a Windows tile configuration named msapplication-config with the content none, and page descriptions whose prose happens to contain the word index were all caught and discarded by hand before any count below was taken. Every number here is a count of carriers identified by name, not of pages whose markup contained a directive word.
| Stage | Hosts | What it means |
|---|---|---|
| Corpus hostnames | 1,419 | Two committed seed files, stratified by platform and sector |
| Closed to LantadBot in robots.txt | 11 | Home page not requested |
| Did not answer at all | 105 | DNS, TLS, or a fifteen second timeout |
| Answered a status other than 200 | 280 | Nothing to read for a directive |
| Analysed: HTTP 200 with HTML | 1,023 | Every figure below counts over these |
How many home pages carry a robots meta tag, and how many carry nothing?
380 of the 1,023 pages carried a generic robots meta tag. 33 carried at least one meta tag addressed to a named crawler rather than to robots in general, 43 such tags between them, and the names used were googlebot on 20 pages, google on 14, bingbot on 7, adsbot-google on 1 and googlebot-news on 1. 12 pages sent an X-Robots-Tag response header. Counting any page with at least one of the three gives 394, and the remaining 629 carried none of them, which is 61.5 percent of the readable set.
Two things follow that are worth separating. The first is that the robots meta tag is not a fringe mechanism: where a carrier exists at all it is overwhelmingly the generic meta tag, by roughly thirty to one over the header. The second is that the header is almost unused, 12 pages in 1,023, and that is the carrier with the lowest cost to deploy for anyone who cannot edit a template, because it is set in server or CDN configuration and needs no change to the HTML. An earlier run of this scanner found 7 of 392 home pages sending an X-Robots-Tag of any kind on 9 September 2026, five of them a captcha, on a smaller sample and with the same shape of answer.
The 33 pages using a crawler addressed meta tag are the interesting minority, because that is the exact form the newest page level controls take. Google's robots meta tag documentation, read at source on 2 November 2026, states that Google supports two user agent tokens in the robots meta tag, googlebot for all text results and googlebot-news for news results, and gives the example tag addressed to googlebot with the content nosnippet. The measurement matches that documentation closely: googlebot is the most used crawler name on the page, and googlebot-news appears once. What the corpus shows is a mechanism that a few hundred sites understand and the large majority have never touched, which is the practical context for any vendor that now asks for a preference to be expressed this way rather than in a file. Whether a crawler can even reach the markup is a separate question, and the one this product usually measures: see what a crawler is and the method behind every score.
| Meta tag name | Tags | Documented by |
|---|---|---|
| googlebot | 20 | Google, for all text results |
| 14 | Google, carrying notranslate on these pages | |
| bingbot | 7 | Microsoft |
| adsbot-google | 1 | |
| googlebot-news | 1 | Google, for news results |
What the carriers that do exist actually said
Counting every directive inside those robots and crawler named carriers across the 1,023 pages, the tally was index 315 times, follow 304, max-image-preview 224, max-snippet 129, max-video-preview 127, notranslate 16, noindex 13, nofollow 9, noarchive 6, noai 2, noimageai 2, nosnippet 1, noimageindex 1 and nocache 1. The shape of that list is the finding. The five most common directives are all permissive or expansive: index and follow restate the default, and the three preview directives ask search engines to show more rather than less. Only 27 of the 1,023 pages carried any restrictive directive at all, counting noindex, nofollow, noarchive, nocache, nosnippet, noimageindex, none, noai and noimageai together.
That is a different statement from saying the web has opted in. A page carrying index, follow is making no request that changes any engine's behaviour, because both are the default, and a page carrying max-image-preview:large is asking for a larger thumbnail rather than expressing anything about AI use. So the honest summary of the page level layer on this corpus is that most pages have no carrier, most carriers that exist restate defaults, and 27 pages in 1,023 use the layer to restrict anything. Lantad has measured the same pattern in the file layer: 1,004 robots.txt files named 2,209 user agent tokens between them, and the overwhelming majority of corpus files make no per vendor choice.
The two pages carrying noai and noimageai are worth naming because those directives are in neither Google's documentation nor any RFC. They are a convention that image and text hosting communities adopted, and the two pages serving them in this corpus were variety.com and hollywoodreporter.com. A directive no engine has documented support for is a request rather than a control, and that is the category it belongs in: a site that has written it has expressed a preference clearly and has no published commitment from any vendor that it will be read. Lantad measures whether the bytes carry it, which is a different claim from whether anything honours it, and that boundary is the one this blog tries never to blur.
The seven pages carrying noarchive or nocache
Six pages sent noarchive and one sent nocache, seven distinct sites in all, and none of the 1,023 pages carried any token containing ai-training or ai-grounding. The six sending noarchive were aftenposten.no, nzz.ch, handelsblatt.com, nzherald.co.nz, japantimes.co.jp and bbva.com, which is five news publishers and one bank, and each was re-fetched by hand on 2 November 2026 to read the tag directly rather than trusting a parse. The single page sending nocache was nos.nl, and the detail that makes it worth reporting is the carrier rather than the directive: it sent nocache in a meta tag addressed to bingbot, not in a generic robots tag.
These two directives carry more weight than their counts suggest, for a reason that is documented rather than inferred. Google's robots meta tag documentation, read at source on 2 November 2026, lists both under a section headed Reference of historical and other unused rules, states that the noarchive rule is no longer used by Google Search to control whether a cached link is shown in search results because the cached link feature no longer exists, and states that Google Search does not use nocache. So on Google's side these are dead letters by its own account, while Microsoft's published guidance has historically pointed at exactly these two tags, which this blog wrote up when it noted that Microsoft's AI opt out was a meta tag rather than a robots.txt token on 30 July 2026.
An earlier run of this scanner measured 7 of 1,077 readable home pages sending noarchive on 5 October 2026, and none sending nocache. Today's run gives 6 and 1 over 1,023 readable pages. Those two results are close, and the temptation is to read a change into the difference, so it is worth saying plainly that there is no measured change here: the responding set is not the same 1,077 hosts, the two runs used different parsing code, and a difference of one page between two single shot runs a month apart is inside the noise either run can produce. What can be said is that both runs found the same order of magnitude, a handful of pages in a thousand, and that the handful is concentrated in news publishing.
| Host | Carrier | Directive |
|---|---|---|
| japantimes.co.jp | meta name=robots | noarchive |
| aftenposten.no | meta name=robots | noarchive |
| nzz.ch | meta name=robots | noarchive |
| handelsblatt.com | meta name=robots | noarchive |
| nzherald.co.nz | meta name=robots | noarchive |
| bbva.com | meta name=robots | noarchive |
| nos.nl | meta name=bingbot | nocache |
Why the carrier matters more than the directive
A file at the site root and a tag on a page are not interchangeable, and the difference is operational rather than theoretical. A robots.txt rule is one edit in one place that covers every path it matches, which is why it is the layer most sites have actually used: Lantad measured 62 of 1,019 corpus robots.txt files naming bingbot on 22 October 2026, and 1,004 files naming at least one token on 17 September 2026. A page level directive has to be emitted by whatever renders the page, on every page where it is wanted, and then has to survive in the bytes a crawler receives rather than appearing only after a browser runs. That last condition is the one this scanner exists to check, and the one a site owner is least likely to have verified: a preference injected by client side script is a preference a JS blind fetch never sees.
The 629 pages with no carrier are therefore not 629 sites that need to add a directive. Most of them have no reason to want one. They are 629 sites for which the page level layer is currently an empty mechanism, so any instruction that arrives in the form add a tag to your pages starts from nothing rather than from an edit. For a site on a hosted platform the practical question is whether the platform lets the head be edited at all, which is the subject of the per stack guides at the fix pages, and for a site that can set response headers the X-Robots-Tag route avoids the template question entirely at the cost of being the carrier only 12 pages in this corpus used.
There is a second asymmetry that the counts above make concrete. The file layer has one reader per vendor token and a specification, RFC 9309, that says what happens when no group matches: crawlers obey the wildcard group if one is present. The page level layer has no equivalent. There is no specification that defines what an unrecognised meta tag name means, no fallback rule, and no registry of which names a given engine reads, so the behaviour is whatever each engine's own documentation says and nothing more. That is why a count of 7 bingbot addressed tags in 1,023 pages is a fact about deployment and not a fact about compliance, and why Lantad reports the former and declines to report the latter. The same boundary runs through how robots.txt compliance is being graded by third parties and through the way one vendor's token can cover two systems at once.
Page level, on 1,023 readable home pages
- 629 pages carried no carrier at all
- 380 carried a generic robots meta tag
- 33 carried a crawler addressed meta tag
- 12 sent an X-Robots-Tag header
- 27 used the layer to restrict anything
- No specification defines a fallback rule
File level, measured on earlier dates
- 1,004 robots.txt files named 2,209 tokens, 17 September 2026
- 62 of 1,019 files named bingbot, 22 October 2026
- One edit covers every matching path
- RFC 9309 defines the wildcard fallback
- Needs no change to any page template
What to check on your own site, and what this did not measure
The check that follows from this measurement is short and does not need a tool. Fetch your own page without a browser, read the bytes that come back, and see whether any of the three carriers is present and whether it says what you think it says. The free tools here do the fetch for you: the robots.txt tester reads the file layer per crawler, what GPTBot sees shows the served bytes for one page, and the AI crawler list names the tokens in the registry this product ships. The failure this corpus run points at is not a wrong directive, it is a directive that is not in the bytes, and for a page whose head is assembled in the browser that is the normal outcome rather than an unusual one.
What this run did not measure needs stating as plainly as the counts. It did not ask any engine anything, so nothing here reports whether any directive on any of these pages was honoured, read, or even fetched by the vendor it names. It did not test whether a page would serve different markup to a vendor's own crawler: every request went out as LantadBot, so a host that refused this scanner may well admit Bingbot or Googlebot, and 105 non answers and 280 non-200 responses are consistent with that. It read one apex home page per host, usually fetched once, so a site carrying a directive on its article template and not on its home page counts here as carrying none. It executed no JavaScript, which is the point of the method and also a limit on it: a carrier present only after hydration was not counted, and this run cannot say how many of the 629 would gain one in a browser, a gap Lantad has measured in a different context where 27 of 404 pages gained structured data once a browser ran.
And the sampling frame is two committed seed files stratified by platform and sector, not a random draw from the web, with the smaller strata holding around thirty sites each. That makes these counts a description of this corpus on one date. They are not an estimate of the web, and the 61.5 percent figure in particular should be read as what 1,023 specific readable home pages did on 2 November 2026 rather than as a web wide rate. The vocabulary behind all of this is defined at the generative engine optimization entry and the AI visibility entry, and what the product does with a page it cannot read is set out in the methodology and in the AEO glossary entry.
Flow: GET / as LantadBot to Read response headers; Read response headers to X-Robots-Tag present?; X-Robots-Tag present? (yes, 12) to Carrier found; X-Robots-Tag present? (no) to Read served HTML; Read served HTML to Meta name robots or a crawler?; Meta name robots or a crawler? (yes, 394 total) to Carrier found; Meta name robots or a crawler? (no) to No carrier: 629 pages.
Lantad
Published .
A directive that keeps a page out of something has to reach the crawler somehow, and there are only three places it can ride: a robots meta tag in the HTML, a meta tag addressed to one named crawler, or an X-Robots-Tag header on the response. RFC 9309, the Robots Exclusion Protocol, defines none of them. It is a path matching protocol over a single file at the site root, and the specification contains no per page markup and no header at all, which is why every page level control an engine offers has to be documented by the engine rather than by the standard.
Common questions
What is a robots meta tag?
It is a meta element in a page's HTML that carries crawling and indexing directives for that page, written as a name and a content attribute. The name is either robots, addressing every crawler that reads the tag, or the name of one crawler such as googlebot or bingbot, addressing only that one. Google's robots meta tag documentation, read at source on 2 November 2026, states that Google supports two such user agent tokens, googlebot for all text results and googlebot-news for news results. Lantad measured 380 of 1,023 readable corpus home pages carrying a generic robots meta tag on 2 November 2026 and 33 carrying one addressed to a named crawler.
Is a robots meta tag the same as robots.txt?
No, and the difference is where the instruction lives. Robots.txt is a single file at the site root that matches URI paths, defined by RFC 9309, and one edit in it covers every path it matches. A robots meta tag sits in the HTML of one page and has to be emitted on every page where it is wanted. RFC 9309 defines no page level markup and no HTTP header at all, so every page level directive is documented by the engine that reads it rather than by the standard, and there is no specification defining what happens when an engine meets a meta tag name it does not recognise.
Does a page without a robots meta tag mean the site has allowed everything?
It usually means the site has decided nothing rather than decided to allow. Lantad measured 629 of 1,023 readable corpus home pages carrying no robots meta tag, no crawler addressed meta tag and no X-Robots-Tag header on 2 November 2026. Absence of a carrier is not consent, and it is not refusal either: what it means in practice depends on each engine's published default for the directive in question, which is stated in that engine's own documentation and not in any standard.
Should a directive go in a meta tag or in an X-Robots-Tag header?
Both reach a crawler that reads the response, and the choice is usually decided by what you can edit. A meta tag needs a change to the HTML head, which on a hosted platform may not be possible on every page, and will not reach a crawler at all if it is injected by client side script after the page loads. An X-Robots-Tag is set in server or CDN configuration and needs no template change, which makes it the lower cost route for anyone who cannot edit the head, and it was also the least used carrier in this corpus: 12 of the 1,023 pages sent one on 2 November 2026 against 380 carrying a generic meta tag.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
Generative Engine Optimization (GEO)
Generative Engine Optimization (GEO) is the practice of structuring and publishing content so that generative AI engines can fetch it, read it in full, and cite it when they answer a question.
Answer Engine Optimization (AEO)
Answer Engine Optimization (AEO) is the practice of making a page readable, allowed, and structured enough that AI answer engines can use it as a source when they compose an answer.
noarchive: 7 of 1,077 home pages sent the tag two AI vendors read as a training opt out
Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 5 October 2026 and read every permitted home page with no JavaScript executed. 1,077 answered HTTP 200 with HTML. 7 of them sent noarchive, the page level directive Amazon's own crawler page reads as do not train and Microsoft's reads as exclusion from Bing Chat answers, and Google's current documentation says it no longer uses at all. None sent nocache.
X-Robots-Tag noindex: 5 of 391 home pages sent one, and all five were a captcha
Lantad requested the home page of 392 hostnames on 9 September 2026 and recorded the response headers. Seven responses carried an X-Robots-Tag. The five that said noindex all arrived as an HTTP 202 of 168 or 169 bytes whose body was a redirect to a captcha, and the same response comes back from the hosting company's own domain.
Microsoft's AI opt out is a meta tag, not a robots.txt token
Every other major vendor gave site owners a robots.txt name to disallow. Microsoft's published guidance for keeping a page out of its AI answers is NOCACHE or NOARCHIVE in a meta robots tag, which is why none of the 15 tokens in our crawler registry belongs to Microsoft.
