Soft 404: 125 of 1,094 sites answered a URL that does not exist with HTTP 200

How we asked 1,419 sites for a page that does not exist

The corpus is the union of the two committed sampling frames in this repository, worker/seeds/corpus-seeds-industry.json and worker/seeds/corpus-seeds-platform.json, 1,419 distinct hostnames between them, stratified by sector in the first file and by the platform a site is built on in the second. It is an editorial frame rather than a random draw from the web, so every figure below describes this corpus and no wider population. The same frame is behind the other sweeps published here, which is what makes them comparable to each other.

Each host was asked twice over plain HTTPS with no browser and no JavaScript, identifying as LantadBot/1.0, the token whose conduct is set out on our crawler page. The first request was the home page at the site root, which establishes that the host is reachable and serving HTML. The second was the path /lantad-crawl-check-7b19e4c2-does-not-exist, chosen to contain a random segment so that no site in the frame could plausibly have it. Redirects were followed, with the final URL recorded, and the response body was read only when the content type was textual. A fifteen second timeout applied to both.

The losses matter and they are not random. 45 hosts failed the home request outright, 24 of them by timeout and 21 by connection failure. A further 275 answered the home page with something other than a 2xx HTML response, and the largest group inside that number is 215 responses of 403, which is a refusal of this scanner rather than an absence of content and is the same wall measured directly in the crawler refusal sweep. Another 40 answered 503. That leaves 1,099 home pages answering 2xx with an HTML content type, of which 1,094 also returned an answer to the probe. 1,094 is the denominator for every number in this post. Because a site that refuses this scanner at the door never reaches the test, the corpus that survives is biased toward sites that let crawlers in, and how this scanner reasons about that bias in production is set out in the methodology.

What a soft 404 does to an AI crawler

The status code is the only machine-readable statement a server makes about whether something exists. Everything else on the page is prose, and prose is what the reader has to interpret. Google's documentation on HTTP status codes and network errors, which carried a last updated date of 4 February 2026 when we read it on 20 September 2026, states that "all 4xx errors, except 429, are treated the same: Google crawlers inform the next processing system that the content doesn't exist". A 200 sends the opposite instruction to the same pipeline.

Two things follow, and only the first is widely discussed. The first is indexing: an error page that answers 200 is a candidate for the index like any other page, which is why Search Console reports soft 404s as a defect at all. The second matters more for a site being read by an answer engine. A URL that always answers 200 cannot be falsified. If a model produces a link to a page on your domain that was never written, and something fetches that link to check it, a soft 404 confirms it. The check passes, the citation survives, and the reader arrives at a page that is not the page they were promised. How a page becomes a cited source in the first place is a separate subject, set out in the ChatGPT citation guide. That is a different failure from being unreadable, which is the failure this scanner usually measures and which shows up as zero words reaching a crawler or as prose the server never sent.

We should be precise about what is and is not established here. Google documents its own handling. What GPTBot, ClaudeBot or PerplexityBot do with a 200 carrying an error page is not something we measured today and not something we found stated in their documentation when writing earlier posts on crawler behaviour by vendor. The finding below is about what the sites sent, not about what any named bot concluded. That distinction is the whole reason this post exists separately from a claim about citations.

The 125, sorted by what they actually served

The 125 sites that answered 200 did not all do the same thing, and the differences change how bad each case is. Sorting them by what came back gives five groups. 39 followed a redirect that ended at the home page, so the probe URL resolved away entirely and the final URL recorded was the site root. 32 returned the probe URL unchanged but served a document whose title element was identical to the home page's title, which is the signature of a catch-all route rendering the front page for anything unmatched. 25 returned a page whose text did say not found, in one of the several languages the detection covered, while still carrying a 200. 17 returned fewer than 20 readable words and no message of any kind, which is the shape a single page application produces when the router handles the miss in the browser. The remaining 12 returned a different real-looking page with substantial text and no indication that anything was wrong.

Read across those groups, 93 of the 125 carried no not-found wording anywhere in the first 4,000 bytes of the response. 55 of those 93 carried 100 or more readable words, and the median across all 125 was 199 words with the largest at 14,701. So in the majority of these cases neither the status line nor the text says anything is missing, and a reader that only has the bytes has no signal at all. Two of the 125 carried an X-Robots-Tag of noindex on the probe response, which at least keeps the page out of an index even though it does not correct the assertion of existence.

The 17 near-empty shells are the group most likely to be misread, and they connect to a measurement published here earlier today. A page that ships almost no text and fills itself in the browser is the client side rendering pattern measured across the platform corpus, and Google's guidance on fixing search-related JavaScript problems, last updated 18 December 2025 when read on 20 September 2026, names this case directly: it tells developers either to "redirect to a URL where the server responds with a 404 status code" or to "add or change the robots meta tag to noindex". Per-framework versions of the same advice live in the React guide and the Next.js guide.

111 of the 125 did it again on a different path

A single probe can be misread. A path at the site root might hit a rule that a deeper path would not, and one odd response could be a cache artefact rather than a routing decision. So the same 125 hosts were asked again the same day for a second path of a deliberately different shape, /docs/2f8a1d/nonexistent-check-page, which sits two directories down and looks like documentation rather than like a scanner probe.

All 125 answered. 111 returned 200 for the second path as well. 11 returned 404, which is the interesting minority: those sites do distinguish, and the first probe found a rule specific to root-level paths rather than a blanket catch-all. One returned 406, one 444 and one 500. Of the 111 that answered 200 twice, 91 again carried no not-found wording and 67 again served the home page's own title. The behaviour holds across two unrelated paths on the large majority of these sites, which is what makes it a routing default rather than a coincidence.

This is worth stating in the other direction as well, because the headline number is a minority finding and the post would be dishonest if it read otherwise. 936 of 1,094 sites, 85.6 percent, returned a clean 404, and one returned 410 Gone. The correct behaviour is the overwhelming norm in this corpus. Those 936 error pages were not empty either: the median carried 180 readable words and only 29 returned none at all, so most sites are serving a helpful page and the honest status code together, which is exactly the combination the specification allows and the documentation asks for.

The soft 404 also interacts with a site's own declarations about its URL space. A sitemap is an assertion that the listed URLs exist, and this blog has measured how many sites declare one in robots.txt and separately how a crawl of real links compares with what a sitemap claims. A catch-all 200 removes the only check on those assertions, because every URL a sitemap could list, correct or stale, answers as valid.

Which sectors returned 200 most often

The industry frame files each host under one of eight sectors, and the platform frame is counted here as a ninth stratum, so the rate can be compared across them. The denominators are small enough in places that these are differences worth noticing rather than differences worth trusting, and none of them is a statement about any named organisation.

Ecommerce was highest at 17 of 68, or 25.0 percent, with travel almost level at 19 of 77, or 24.7 percent. Both are sectors where a large generated URL space is normal, product and listing pages appearing and disappearing under filter and search parameters, and a catch-all route is an easy way to keep those pages from breaking. Education followed at 17 of 106, finance at 14 of 95, government at 12 of 88 and healthcare at 12 of 94, a band running from 16.0 percent down to 12.8 percent. News was much lower at 4 of 59, or 6.8 percent, which fits a sector whose publishing systems have been tuned against search behaviour for two decades.

The two lowest are the most interesting. The platform stratum returned 24 of 388, or 6.2 percent, and SaaS returned 6 of 119, or 5.0 percent. The platform frame is dominated by sites built on hosted builders, and a hosted builder ships one error route for every site it serves, so the behaviour is decided once by the vendor rather than 388 times by site owners. That is the same mechanism that makes platform-level defaults visible in the robots.txt sweeps published here, where a generated file appears verbatim across dozens of unrelated sites.

None of this is a ranking of sectors by quality. The rate measures one specific routing decision on one specific day, and a sector with a high rate may be doing better on every other signal this scanner reads, including structured data and prose parity. What it does say is that the defect concentrates where URL spaces are generated rather than authored.

What this measurement does not tell you

It does not tell you that any AI crawler was misled. No AI crawler was used here and no answer engine was queried. The measurement is of what 1,094 servers sent to one plainly identified scanner on one day, and every inference beyond that is reasoning about mechanism rather than an observation. Whether a soft 404 has ever caused a wrong citation is a question this test cannot reach.

It does not measure how many real pages exist behind these URLs, so it says nothing about the size or quality of any site. It does not measure rendered behaviour: the 17 near-empty shells may well display a correct error message to a person, and that possibility is why they are described here as shells rather than as failures of user experience. It is also a single observation per path per host. The confirmation run raises confidence that the routing is stable, but a site can change this in one configuration line, and a figure from 20 September 2026 is a figure about 20 September 2026 and nothing later.

Two things are worth checking on your own site, and neither needs this scanner. Ask for a path you know does not exist and read the status line rather than the page, because the page is the part designed to reassure you and the status line is the part machines act on. Then ask for a second one of a different shape, deeper in the tree, because the 11 sites that answered 404 the second time show that one probe can miss a rule. If a soft 404 turns up, the fix is the one the platform documentation already describes: return 404 or 410 for the miss, and keep whatever helpful page you want to show at that status.

The same posture applies to the other machine-readable statements a site makes. A robots.txt that does not parse is a rule that does nothing, which is the finding behind the robots.txt defect sweep, and a response that carries no validator is a site that cannot say whether anything changed, which is the change signal sweep. Each of these is the same class of problem as a soft 404: not content that is missing, but a machine-readable claim that is untrue, which is a different thing from the readability that AI visibility usually refers to. If you want to see what a crawler actually receives from a given URL, the crawler view and the robots.txt tester both answer that live, and the wider corpus work sits in the crawlability study.