Do AI crawlers read HTML comments? 85,359 sat on 1,060 home pages, and 27,494 were empty

Do AI crawlers read HTML comments?

Start with what the specification settles. Mozilla's reference on using HTML comments, last modified on 7 November 2025, states that a comment "is used to add explanatory notes to the markup or to prevent the browser from interpreting specific parts of the document", and that "the browser ignores comments as it renders the code. In other words, they are not visible on the page". The same page makes a second point that matters more than it looks: "while comments do start with a < and end in a >, a comment is not an HTML element". It is not markup with content inside it. It is one node whose whole value is a string.

That node is real, and this is where the question stops being rhetorical. Mozilla's reference on the Comment interface, last modified on 24 September 2024, says it "represents textual notations within markup; although it is generally not visually shown, such comments are available to be read in the source view", and its inheritance chain runs EventTarget, then Node, then CharacterData, then Comment. A comment is a Node. Anything walking the document tree can reach it and read its data. So whether a comment's words count is not a fact about HTML at all. It is a decision taken inside whatever reads the page, and different readers may decide differently.

Lantad's reader decides to drop them, and that is checkable rather than asserted. The extraction ruleset in core/src/extract.ts drives a streaming parser and registers handlers for text, for tags and for the JSON payloads it harvests. It registers nothing for comments, so comment text is never offered to the sink that builds the extracted document, and the ruleset's own drop list covers script, style, noscript, template, SVG and iframe subtrees on top of that. Running the real extractor over four of the captured pages confirmed it end to end on 9 November 2026: the Yoast string on growforagecookferment.com, the Blazor state on peru.travel, the placeholder note on amtrak.com and the OneTrust rationale on acc.org are each present in the raw HTML and absent from both the visible text and the main text the extractor returns. Nothing in a comment can move prose parity, which is the largest component of the score, and the methodology page describes that same pipeline.

One limit belongs here rather than at the end. This measures what is in the comments, and what Lantad's extractor does with them. It does not measure what OpenAI, Anthropic, Google or Perplexity do inside their own pipelines, because none of them publishes that and none of it is observable from the outside. A reader who wants the stronger claim will not get it from this post. What follows is the part that is knowable: the comments are there, something downloads them, and here is what they say.

What 85,359 comments on 1,060 home pages contained

900 of the 1,060 pages carried at least one comment and 160 carried none. The interesting split is narrower than that: 671 pages carried at least one comment holding natural language, meaning letter runs that look like words rather than encoded data. Across all 1,060 pages those comments held 95,483 words, against 1,304,861 words of visible prose on the same pages. Put the two together and comment text amounts to roughly seven percent as much language as the pages actually show, which is a surprising amount of writing for a surface with no readers.

It is also, overwhelmingly, not writing anybody meant as content. 33,553 comments were short markers, the open-and-close labels a template emits around a region. 27,494 were empty. 12,554 held natural language that ran to fewer than twelve words, which in practice means a phrase like a section name. Only 1,448 comments on 1,060 home pages, 1.7 percent of the total, carried more than twelve words of language, and those are the ones worth reading, because a comment long enough to be a sentence was written by a person with something to say.

This is a different shape from the hidden content this blog has measured before, and the difference is worth being precise about. When Lantad measured words sitting behind a hidden marker, the text was in the DOM, renderable, and withheld by styling: a crawler that ignores CSS reads it as page content, which is exactly why it matters. A comment is the opposite case. It is in the bytes and never in the rendered page under any styling, so the question is not whether a crawler is fooled into reading it. The question is what you have published without noticing. Lantad's own method notes have quietly said so for a while: the prose counts in the meta refresh measurement exclude script, style, noscript, template and SVG content along with HTML comments, and that exclusion has never been examined until now.

Why 27,494 of the comments were empty

An empty comment is the single most common thing a page says to a crawler and nobody. 27,494 of the 85,359, just under a third, held nothing at all between the delimiters, and the exact spelling is consistent enough to be diagnostic: 23,825 were the four character sequence with no space, written out as an opening delimiter immediately followed by a closing one, and 3,578 carried a single space. The remaining 91 were a newline followed by indentation, or a short run of spaces.

These are not notes anybody wrote. They are placeholders, the marker a JavaScript rendering framework leaves where a conditional branch did not render, and the evidence for that reading is in the distribution rather than in any vendor's documentation. 119 of the 1,060 pages carried ten or more empty comments, and 84 of those 119 also carried a signature of a JavaScript framework in the same HTML: a hydration payload, a build asset path, a scoped style attribute or a framework version attribute. The concentration is extreme at the top. oliverbonas.com carried 6,136 comments, of which 6,127 were empty, in 43,631 bytes. handelsblatt.com carried 4,488 empty ones. The per-comment cost is trivial and the per-page total is not.

Nothing here is a defect, and it is worth saying so plainly rather than inflating the finding. A framework that marks an empty slot with an empty comment is doing something reasonable, and 43,631 bytes on a 489,404 byte page is not what decides whether that page gets cited. It is a clean illustration of a point this blog keeps arriving at from different directions: a large share of what a modern page sends an AI crawler is machinery rather than meaning. The same corpus has shown a Next.js payload running fifteen times the prose, custom elements arriving with no text and text in shadow DOM that reaches the browser and not the extractor. If you are chasing this, the stack guides for React and Next.js are the places where it is actually fixed.

The comments that are dead code

810 comments on 93 of the 1,060 pages carried an Internet Explorer conditional test, the construction that wraps markup in a comment so that only certain versions of that one browser act on it. The verbatim text names the versions it is testing for. landscapevermont.com carries branches for IE 7 and IE 8, and a third testing for less than IE 9 whose body is a script element loading html5shim.googlecode.com over plain HTTP. sundhed.dk carries a Danish instruction to update your browser inside a test for less than IE 9, and a second branch whose test also covers clients that are not Internet Explorer at all. Whatever any browser does with these today, every crawler that fetches these pages downloads the whole branch, and the extractor discards all of it.

Commented out markup is the broader version of the same thing, and some of it is more candid than its authors can have intended. becu.org wraps a set of stylesheet links in a comment whose own text explains that the resources were moved to the head section and that the values of the hrefs are needed here for reference. aiims.edu has gone further: inside a comment sit a style element and a script element whose tag names have been deliberately corrupted, the words style and script each broken by an inserted hyphen, which is how somebody disabled a block of code without deleting it. A popup div with its inline positioning intact sits in another comment on the same page.

None of this is large enough to move a byte limit on its own, and the honest framing matters because the opposite framing is easy and wrong. Googlebot's documented ceiling is the first two megabytes, which this blog has written about in what Googlebot reads before it stops, and the corpus measurement of how many home pages pass two megabytes found very few that do. Dead branches are not a budget problem for most sites. They are a maintenance signal, and the reason to mention them in a post about crawlers is narrower: they are the part of a page that is guaranteed to be read by machines and guaranteed never to be read by a person, which is precisely the condition under which nobody notices what is in them for years.

What a comment discloses about your stack

756 comments carried a URL inside natural language, and a further five were a bare URL with nothing around them, spread across 236 of the 1,060 pages and amounting to 1,263 URLs in total. Almost none of them are links a reader could follow. They are vendor banners, and they name the software and very often its exact version. amref.org states that the site "is optimized with the Yoast SEO Premium plugin v14.6.1". auditoria.ai states the same thing at v28.8. berkeley.edu names Yoast SEO at v28.8 and, in a second comment, a WordPress user activity plugin at v4.4.0. albertahealthservices.ca marks the start of a data collector tag at v10.4.23. Three separate sites carry a HubSpot WordPress plugin banner at v11.3.75 whose text opens with the instruction "DO NOT COPY THIS SNIPPET!", which is addressed to a developer and delivered to everything that fetches the page.

Version strings are the sharp end of this, and the comparison inside the corpus makes the point without any need to speculate about consequences: the same measurement found the same plugin declared at v28.8 on several sites and at v14.6.1 on another, so a comment of this shape tells any reader which release a site is running and how far behind it is. That is a fact about the page rather than a vulnerability claim, and it is a fact the site is publishing on purpose by shipping the plugin's default banner.

Two other kinds of disclosure turned up often enough to name. The first is content management internals: cnn.com carries comments holding component instance identifiers in the form of a CMS host, a component path and a published state marker, which describes the shape of the system that built the page. The second is engineering rationale written for colleagues. acc.org carries an 85 word comment explaining that a subresource integrity attribute "is intentionally omitted here" because the consent platform regenerates the script whenever its configuration changes in the vendor's admin console. That is a clear, reasonable decision, documented in the one place where it is readable by everyone and visible to nobody. The same page carries a note about caching the harder parts of the footer.

Then there are the notes that were simply never meant to leave a branch. 133 comments matched a developer-note vocabulary. albertahealthservices.ca asks "maybe switch colors h2 h3? mockup looks reversed but typography says to this way", an unresolved design question shipped to production. bloomandwild.com carries a TODO about moving a block to after Angular has loaded. australia.com carries "Remove this meta tag or handle caching in server response". If you want to see what a crawler receives from your own page before you go looking by hand, what GPTBot sees returns the raw fetch, and the AI crawler reference lists the tokens that do the fetching.

What this means for a page you own

The first conclusion is the dull one, and it is the one that saves work: nothing you write in a comment will help you get cited. There is no version of this measurement in which comment text contributes to what a crawler extracts from the page, at least not in any pipeline that can be inspected, and there is no instruction, no keyword and no structured hint that becomes effective by being placed there. If you have been told otherwise, the two Mozilla references above are the shortest route to settling it. Comments are also not where structured data goes, and a note to a crawler inside one is addressed to nobody.

The second is that comments are a disclosure surface with no review process. Everything else on a page gets looked at, because somebody can see it. A comment is read by the author once and then by every automated client that fetches the page, forever, and by no human again. That asymmetry is the whole finding, and it is why the three worth auditing are plugin banners that state a version, notes that document a deliberate security tradeoff, and anything a developer wrote to another developer. On 46 of the 1,060 pages the language inside comments outweighed the visible prose outright, led by banque-france.fr at 5,414 words across 736 comments against 1,931 words of prose.

The third is a caution against over correcting, which this blog owes its readers more than another checklist. Stripping comments in a build is a reasonable hygiene step and it will not change a score. The components that decide whether an AI crawler can use your page are the ones Lantad weights and publishes: whether the prose is in the HTML at all, whether robots.txt lets the crawler in, and whether the page states what it is. The corpus measurement of what the graded sites actually lost points for put parity first by a wide margin, and how many pages declare no main landmark is a better use of an afternoon than deleting markers. If the comment on your page is a prompt aimed at a model, the measured answer is already in: an HTML comment was one of the five hiding places in the hidden text attack that averaged 0.3 percent success across thirteen backends, the weakest of the five attacks tested.

What this run did not do is worth restating in one line, because the title is a question and the answer has an edge. The comments were downloaded, classified and counted on 9 November 2026. Lantad's extractor demonstrably drops them. Whether any given vendor's pipeline keeps them is not something this corpus can see, and no figure in this post should be read as if it could. The platform notes for ChatGPT collect what OpenAI does document, the published study is where the corpus figures live, and the crawler policy describes how these 1,419 hostnames were asked and which of them said no.