X-Robots-Tag noindex: 5 of 391 home pages sent one, and all five were a captcha

What does an X-Robots-Tag noindex header do to a page?

Google's robots meta tag documentation, last updated 24 March 2026, puts the equivalence in one sentence: any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag. The same page defines the noindex rule as do not show this page, media, or resource in search results, and gives the header its own reason to exist, which is that you can use it for non-HTML files like image files where the usage of robots meta tags in HTML is not possible. A PDF has no head element to put a meta tag in. A header works on anything the server sends.

Two consequences follow, and they are the reason this is worth a measurement rather than a paragraph. The first is that the header is written by whatever sits closest to the wire. A theme, a plugin, a reverse proxy, a WAF or a hosting layer can add it without anything in the site's own content changing, which means the person who owns the words on the page is not necessarily the person who set the rule about them. The second is that the rule only works if the fetch happens at all. The same Google page states that if a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving rules will not be found and will therefore be ignored, so a Disallow and a noindex on the same URL cancel each other rather than reinforcing each other.

Where the header lands for anyone thinking about answer engines is narrower than it looks, and we published the narrower version when we read the nine crawler documentation pages behind the fifteen tokens this scanner evaluates. Two of the nine name noindex at all. At Google the directive that reaches AI Overviews and AI Mode is nosnippet rather than noindex. What noindex reliably removes is the search result, and for an AI crawler that grounds an answer on the search index, removing the search result is not a small side effect. It is the entire supply.

What we requested, and what 391 hostnames answered

The list is the platform seed frame this scanner already keeps, 392 hostnames grouped into ten categories by the site technology they were collected for: Framer, Webflow, Wix and Squarespace, Bubble and other no-code, single page application startups, Shopify direct to consumer, WordPress small business, static documentation, software marketing sites and local media. It is a hand-built list rather than a random sample of the web, so it describes those ten groups and nothing wider. We requested each home page once over HTTPS, identifying honestly as LantadBot with the address of the page that explains what the crawler is for, followed redirects, and set a 15 second timeout.

371 answered HTTP 200 with an HTML document. Twelve answered 429, five answered 202, two answered 403, one answered 404, and one did not answer inside the timeout. That leaves 20 of 391 responses, or one in twenty, where an honest crawler asking once for a public home page did not receive the home page. The pattern was not spread evenly. All twelve of the 429s came from the 34 hostnames in the Shopify group. Four of the five 202s came from the 41 in the WordPress small business group and the fifth from the 39 software marketing sites. The two 403s were both local media.

On the 371 pages that did arrive, we also read the robots family of meta tags out of the HTML, which is the comparison that gives the header numbers their scale. 122 of the 371 carried a robots meta tag and 249 carried none at all. Exactly one page carried noindex in a meta tag. 64 of the 371 named max-snippet and every one of those 64 set it to -1, which asks for no limit rather than imposing one. None carried nosnippet. That distribution is close to what we found when we read the meta robots tags on 81 home pages three days earlier, and it is the ordinary state of a small site: the tag is either absent or permissive.

Two of the 371 sent an X-Robots-Tag alongside a real page, and both were permissive, one reading index, follow and the other index, follow with the same three max-preview values a WordPress SEO plugin emits by default. Those two are the boring case and they are the case a site owner would expect. Everything interesting in this run is in the other five, and it is the same shape as the finding when we asked 120 hostnames for a robots.txt and a home page on 8 September: what a small site sends is usually a default that arrived with the hosting, not a decision.

Every noindex header in the run came from a captcha, not a site

The five noindex headers were identical to each other. Each arrived on an HTTP 202 response from nginx with a content length of 168 or 169 bytes, a header reading sg-captcha: challenge, a Cache-Control of no-store, and a body holding nothing but a meta refresh to /.well-known/sgcaptcha/ with the requesting IP address written into the query string. There is no page in that response. There is a redirect to a challenge, and a header instructing whoever received it not to index the URL it was asked for.

We then asked the obvious question, which is whose behaviour this is. On the same day we requested www.siteground.com with the same scanner, and got HTTP 202, sg-captcha: challenge and X-Robots-Tag: noindex, with a 185 byte body refreshing to the same /.well-known/sgcaptcha/ path. We asked for that company's own knowledge base article about seeing a captcha on your website and received the same 202 with the same header and a 197 byte body. So the mechanism reproduces on the vendor's own domain, which is as close to a first-party confirmation as an outside observer gets, and it is why we are naming the hosting layer rather than the five sites. Those five did not write this header. Their host did, and it does the same thing to itself. Anyone can check it at the plain text address https://www.siteground.com/kb/seeing-captcha-website/.

The status code is the part that decides how much this matters. RFC 9110 section 15.3.3 defines 202 as indicating that the request has been accepted for processing, but the processing has not been completed, which is a fair description of a challenge. Google's documentation on HTTP status codes and network errors, last updated 4 February 2026, says that for 2xx responses Google considers the content for processing, for example in the case of Google Search, for indexing, and for 202 specifically that Google waits for the content for a limited time, then passes on whatever it received to the next processing step. A 202 is therefore not an error that gets retried and forgotten. It is a success carrying a noindex.

We did not observe what Google or any AI crawler actually did with these five URLs, and we are not claiming it. Every major crawler publishes verified IP ranges and hosting bot defences commonly allowlist them, so a request from Googlebot may well receive the page while a request from us receives the challenge. That is exactly why the difference is worth knowing about: a user agent string is a claim rather than an identity, and an honest crawler that has no allowlist entry anywhere is the cheapest available probe for what an unverified one sees. It is the same view our own what GPTBot sees tool reports on a single URL.

Twelve Shopify stores answered 429, and none of them said noindex

The other twelve blocked responses were a different mechanism with the opposite header behaviour. Each carried server: cloudflare and cf-mitigated: challenge, each was about 9.2 KB, and each was a managed challenge page titled with a line saying your connection needs to be verified before you can proceed. All twelve came from the Shopify group, which is 12 of that group's 34 hostnames and none of the other 358. Not one of the twelve carried an X-Robots-Tag.

The first pass ran twelve requests at a time from one address, so a rate limiter reacting to our own concurrency was a live explanation and we tested it. We re-requested all twelve one at a time on the same day, 2.5 seconds apart, first as LantadBot and then as a desktop Chrome user agent. Four then returned the page normally to both, which is consistent with the concurrency having caused those four. Eight still returned 429 to both. On one of the eight, the Chrome user agent received the page and LantadBot received the challenge, which is the only user agent dependent difference in the whole run. The five 202s repeated identically for both user agents, so those are not about the name in the header.

Google's status code documentation says its crawlers treat 429 as a signal that the server is overloaded and consider it a server error, which is the polite reading and the reason a 429 is a softer failure than a noindex on a 202. It does mean these twelve stores were, for a crawler arriving from our network on that day, a shop with the lights on and the door held shut. Cloudflare's own defaults in this area are moving, and we wrote up the change that from 15 September 2026 blocks Training and Agent classified bots by default on ad-carrying pages for newly onboarded domains, which is a policy about a page rather than a policy about a token.

The wider point is one we have measured before from a different angle. When we probed 60 AI crawler tokens against sites whose robots.txt allowed all of them, 11 got a different answer from the network than the file promised. A robots.txt is a statement of intent published by the site. A 429 or a 202 is what the infrastructure in front of the site actually did, and only the second one is the site's real AI visibility. If you sell through a storefront, the stack guide for Shopify is the place we keep the per-platform version of this.

How a crawler resolves the status, the header and the tag together

Three things arrive in one response and they are read in a fixed order. The status decides whether there is anything to read. The response headers are parsed before the body, so an X-Robots-Tag applies whether or not the body is HTML, and it applies to the URL that was requested rather than to whatever the body happens to contain. Only then does an HTML parser reach a robots meta tag. When several rules address the same crawler, Google's robots meta tag page states that where multiple crawlers are specified with different rules, the search engine uses the sum of the negative rules, so the restrictive one wins and a permissive tag does not undo a restrictive header.

Reading it in that order is what makes the five 202s a problem rather than a curiosity. Nothing in the site's own HTML says noindex. The one page in the whole run whose HTML said noindex was a different host entirely. The instruction is added after the content management system has finished, at a layer the site owner does not edit, and it is attached to the address of the home page rather than to the address of the challenge. A crawler that stores what it was told about that URL has been told the URL should not appear in search results.

For anyone whose interest is the answer engines rather than the ten blue links, the chain runs one step further. Google publishes exactly one hard eligibility rule for a supporting link in AI Overviews, which is that the page must be indexed and eligible to be shown in Search with a snippet, and we set out what that means and what it does not on the page about getting cited in Google AI Overviews. A noindex attached to your home page does not need to be about AI to remove you from AI. It removes the index entry, and the citation surface is built on the index. That is also why the crawler token questions and the header questions are the same question in the end, whether you are looking at which of the fifteen crawler tokens publish a user agent string to match or at what your own server hands back to one.

What to check on your own site, and what this scanner missed

The check costs one request. Ask your own home page for its headers with something that prints them, from a network that is not your office, and read the status line as well as the X-Robots-Tag. A 200 with no X-Robots-Tag is the ordinary answer. A 202, a 429 or a 403 means the thing that answered was not your site, and if a noindex is attached to it then the instruction has your home page's address on it. Repeat it for the pages that earn you money rather than only the home page, because a challenge is usually triggered by traffic patterns rather than by a URL. Our robots.txt tester answers the neighbouring question of which crawlers your file admits, and the methodology page sets out what a full scan does and does not measure.

Now the part that is inconvenient for us. This scanner already reads the header: core/src/pipeline.ts lifts x-robots-tag off the response and core/src/verdict.ts merges those tokens with the robots meta tokens into one set. It then acts on exactly two of them, noai and nosnippet, which deduct a configured access penalty and raise a defect. noindex is parsed and read by nothing, so a page carrying it scores as though it did not. That is a gap we are recording rather than one we are closing in this post.

The second gap is worse, because it is the one this run walked straight into. core/src/challenge.ts flags a bot defence challenge when the response carries a cf-mitigated header, or when a body under 20 KB matches a list of interstitial fingerprints, and core/src/pipeline.ts only applies that test to responses below status 400. The five SiteGround 202s have no cf-mitigated header and their 169 byte bodies match none of the fingerprints, so our own detector passes them through as a real page. The twelve Cloudflare pages match the fingerprints and carry the header, but arrive at 429, above the threshold at which the test runs. Seventeen blocked responses, and our challenge detector would have called none of them a challenge.

That matters more than a missing score, because a challenge shell extracted as though it were the page is exactly how a tool emits a confidently wrong grade, which is the failure the detector exists to prevent. It is the same class of problem as the nine crawler documentation pages where two operators state whether their crawler runs JavaScript and the rest are silent: the measurement is only as good as the thing it can tell apart. If you want the wider picture of what crawlers meet on real sites, the crawlability study collects it, the glossary entry for generative engine optimization defines the vocabulary, and the count of how many robots.txt files name OAI-SearchBot is the nearest neighbour to this one.