# Documentation for AI search: 43 of 220 docs pages gave a crawler under 100 words

> Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 16 October 2026, read every permitted home page with no JavaScript executed, then followed each page's own link to a documentation, developer, help or knowledge base surface and read that too. 239 of the 1,069 readable home pages linked one, 187 of those sat on a different hostname, and of the 220 pages that answered HTTP 200 with HTML, 43 delivered fewer than 100 words and 15 delivered none.

- Canonical page: https://lantad.co/blog/documentation-for-ai-search-43-of-220-under-100-words
- This file: https://lantad.co/blog/documentation-for-ai-search-43-of-220-under-100-words.md
- Last substantive update: 2026-10-16

## Key facts

- **Published:** 2026-10-16
- **Category:** Findings
- **Author:** Lantad
- **Length:** 3996 words
- **Takeaway 1:** Documentation for AI search was measured on 239 corpus sites on 16 October 2026, and 43 of the 220 documentation or help pages that answered a plainly identified crawler delivered fewer than 100 words of extractable text, 15 of them none at all.
- **Takeaway 2:** 187 of those 239 documentation surfaces sit on a hostname other than the site that links them, so a second robots.txt governs them, and on 23 of the 187 at least one of fifteen AI crawler tokens received a different verdict at the documentation path than at the site's own root.
- **Takeaway 3:** 112 of the 239 documentation hosts served a plain text llms.txt on 16 October 2026, against 302 of the same 1,419 hostnames measured at their apex on 19 September 2026, and 15 of the 43 pages that delivered under 100 words sat on a host serving one.
- **Takeaway 4:** All 20 pages read in the corpus stratum of open source project documentation carried 100 words or more, while 8 of the 13 ecommerce help centres read did not and 4 of those 8 delivered nothing.
- **Takeaway 5:** docs.stripe.com returned 44 words of extractable text to two requests on 16 October 2026 and serves a 92,764 byte llms.txt on the same hostname.

## Summary

Ask an answer engine how to authenticate against an API, how to return a parcel, or what a configuration flag does, and the page it ought to cite is documentation. That makes documentation for AI search a narrower problem than most on-page work. There is no ranking to win on a help centre and no snippet to tune. There is one question: when a crawler that executes no JavaScript asks for the page, do the words arrive.

This run measured that across the whole corpus. Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt, evaluated the site root for [its own crawler token](https://lantad.co/bot) before requesting anything else, read every home page it was permitted to read with no JavaScript executed, then took each site's own link to a documentation, developer, help or knowledge base surface and read that page too. 1,069 home pages answered HTTP 200 with an HTML content type. 239 of them linked such a surface. 220 of those surfaces answered with HTML, and 43 of the 220 handed back fewer than 100 words. The rest of this post is where those pages are, what they delivered, which robots.txt actually governs them, and what a 112 count of llms.txt files on the same hosts does and does not make up for.

## What does documentation for AI search actually have to deliver?

Only one vendor in this field publishes enough detail to test against, and it is Google. [Google's guide to JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics), carrying Last updated 2026-03-04 UTC when it was read on 16 October 2026, states that Google processes JavaScript web apps in three main phases, being crawling, rendering and indexing, and that "Googlebot queues pages for both crawling and rendering". The same page says that "it is not immediately obvious when a page is waiting for crawling and when it is waiting for rendering", that a page "may stay on this queue for a few seconds, but it can take longer than that", and that a headless Chromium renders the page "once Google's resources allow". So for Googlebot, a page whose text arrives only after a script runs is not invisible. It is deferred, by an interval Google declines to bound.

The other side of the market publishes nothing comparable. [OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots), read the same day, names four product tokens and says what each is for: OAI-SearchBot surfaces sites in ChatGPT's search features, GPTBot crawls content that may be used to train foundation models, OAI-AdsBot checks the safety of landing pages submitted as ads, and ChatGPT-User handles certain user-initiated actions. The page carries no last updated date and says nothing at all about JavaScript, rendering or script execution. This blog has already surveyed that gap across the field and found that [two of nine operators say either way](https://lantad.co/blog/do-ai-crawlers-render-javascript-two-of-nine), which is the state of the published record rather than a claim about what any crawler does in practice.

That asymmetry is why this measurement is worth taking at the byte level. What a non-rendering reader receives is the floor of what every reader receives, and it is the only part a publisher can verify from outside. It is also the quantity this scanner grades as [prose parity](https://lantad.co/glossary/prose-parity), the gap between the words in the delivered HTML and the words a browser ends up showing. On the corpus as a whole, parity is where [the graded sites lost most of their points](https://lantad.co/blog/ai-visibility-score-972-of-1419-were-graded). The question here is narrower than a site score: not how a whole site fares, but whether the one page type an answer engine most wants to quote is in the delivered bytes.

## How many sites publish documentation, and where does it live?

The selection rule was deliberately narrow, because a loose one picks up contact pages. A link counted only when the first segment of its path, or the first label of its hostname, itself named the surface: docs, doc, documentation, developer, developers, api-docs, api-reference and the like for the documentation tier, and help, support, kb, knowledge-base and their hyphenated variants for the help tier. Anchor text alone never qualified, the link had to stay inside the same registrable domain, and paths deeper than three segments were dropped. An earlier pass that trusted anchor text pulled in a page called Support that was a contact form, which is the reason the rule reads as it does.

Under that rule 239 of the 1,069 readable home pages linked at least one documentation or help surface, and those 239 sites linked 825 distinct surfaces between them. 142 of the 239 best candidates were a documentation or developer surface and 97 were a help, support or knowledge base surface. 101 were identified by the hostname label and 138 by the path segment. The split by stratum is sharp and worth stating plainly: 97 of the 239 are in the SaaS stratum, 21 in the stratum of open source project documentation sites, 21 in finance and 13 in ecommerce, while the news, government, travel and education strata contributed 14, 8, 3 and 8. A help centre is close to universal among software companies in this frame and close to absent elsewhere, which is a fact about the frame as much as about the web.

The structural finding is the second number. 187 of the 239 surfaces, 78.2 percent, sit on a hostname other than the one that links them: docs.example.com rather than example.com/docs. That is not a styling detail. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) requires the rules to be accessible in a file named /robots.txt, all lowercase, in the top-level path of the service, so a documentation subdomain is a separate service with a separate file, and the file a site owner edits when they decide how to treat an [AI crawler](https://lantad.co/glossary/ai-crawler) is usually not the file that governs their documentation. This blog measured the general version of that problem and found [73 of 277 subdomains giving an AI crawler a different answer from the apex](https://lantad.co/blog/robots-txt-subdomains-73-of-277-differ). The documentation case is the one where the stakes are highest, because the content is the content an answer engine wants.

Seven of the 239 picked paths were refused to LantadBot by the documentation host's own file, five of them by a wildcard Disallow of everything and two by a 5xx response, which this scanner treats as a site-wide block. 232 pages were therefore requested and 220 answered HTTP 200 with an HTML content type. Of the 12 that did not, eight answered 403, one 401, one 400, one never completed a connection and one exceeded the redirect limit. Those twelve are not counted as thin pages anywhere below, because nothing was read from them.

## What the documentation pages delivered

Every page was read twice on the same day, and all 220 word counts were identical on the second read, so none of what follows is a transient. The median page delivered 278 words. The lower quartile delivered 123. The largest, the API reference at api.aiven.io, delivered 68,142. 43 pages delivered fewer than 100 words and 15 of those delivered zero. 22 of the 220 carried no heading element of any kind, and 89 carried at least one block of [structured data](https://lantad.co/glossary/structured-data).

The tier split runs the way the names suggest, and not by much. Documentation and developer surfaces had a median of 309 words across 136 pages, help and support surfaces a median of 173 across 84. But 18 of the 136 documentation pages fell under 100 words against 25 of the 84 help pages, so proportionally the help centre is roughly twice as likely to arrive empty. The stratum table above holds the sharper contrast: not one of the 20 readable pages in the open source documentation stratum fell under 100 words, and 16 of those 20 carried the signature of the same static documentation generator, while 8 of the 13 ecommerce help centres did fall under and 4 of those delivered nothing at all.

The named cases are the useful part, because they are checkable. developer.atlassian.com, www.jetbrains.com/help, help.nationalgeographic.com, docs.ramp.com, docs.robinhood.com/crypto/trading, developer.starlingbank.com, developer.workday.com, www.bestbuy.com/support, www.dollargeneral.com/help-center, support.everlane.com, hotro.tiki.vn/knowledge-base, helpcenter.frontiersin.org, help.cbsnews.com, help.irishtimes.com and developer.misa.vn each returned zero extractable words to both requests. www.nike.com/help returned two words, www.airbnb.com/help ten, www.walmart.com/help nineteen, clickhouse.com/docs twenty three, and docs.stripe.com forty four. None of those is a small company that never got round to it.

A thin entry page is not automatically a dead end, and the distinction matters enough to measure. Each of the 43 was requested a third time and its anchor elements counted in the delivered HTML. 42 returned the same word count again and one, help.gitee.com, answered 503. Seven delivered no anchor element at all, so a crawler reading those bytes received neither words nor any onward path: help.nationalgeographic.com, www.jetbrains.com/help, developer.misa.vn, hotro.tiki.vn/knowledge-base, docs.ramp.com, docs.robinhood.com/crypto/trading and support.everlane.com. Eight offered no link to their own host, and 18 offered fewer than ten. At the other end, www.nike.com/help delivered two words and 410 links, 375 of them same-host, which is a navigation shell rather than a blank: the crawl can continue even though the entry page says nothing. The general version of that problem, [84 of 1,091 home pages offering a crawler no path at all](https://lantad.co/blog/internal-links-ai-crawlers-84-of-1091-no-path), is rarer on home pages than it is on these seven documentation surfaces.

## The robots.txt you did not edit

Of the 187 surfaces on a separate hostname, 159 documentation hosts served a file that parsed into at least one group and 16 answered HTTP 404, which under this scanner's evaluator and under the standard leaves everything allowed. 62 of the 187 pairs served byte identical files at the apex and the documentation host, which is what a shared edge configuration produces and is the case where the two documents cannot disagree.

The rest can, and the AI tokens are where it shows. Across the 187 pairs, 34 apex files named at least one of the fifteen AI crawler tokens in this scanner's registry, while only 20 of the documentation hosts did. On 19 pairs the apex named a token and the documentation host named none, and on 148 of the 187 neither file named a single one. Taking each of the fifteen tokens in turn and comparing the verdict at the site root with the verdict at the documentation path, 23 of the 187 pairs produced at least one disagreement. For GPTBot alone, ten pairs blocked it at the apex root while allowing it at the documentation path, and seven did the reverse.

Counting across all 239 surfaces rather than just the off-host ones, 13 of the picked documentation paths were closed to GPTBot: eight by a rule inside a group that names gptbot, three by a wildcard group that never mentions it, and two by a 5xx on the robots.txt itself. The eight named ones are the interesting set, because somebody wrote them. uct.ac.za/support/ombud, www.motherjones.com/support, www.nikkei.com/help, www.figma.com/developers, us.ovhcloud.com/support, sendbird.com/docs, www.airbnb.com/help and www.ally.com/help each sit behind a group naming gptbot with a Disallow that covers the path. Whether that is the intended policy is not knowable from the file. What is knowable is that the decision was expressed at the site level and lands on the documentation, and that the same count for [the whole corpus of robots.txt tokens](https://lantad.co/blog/crawler-user-agent-2209-tokens-in-1004-files) shows how often these rules are inherited rather than authored. Anyone who wants to see which of their own two files answers for a given path can run [the robots.txt tester](https://lantad.co/tools/robots-txt-tester) against the documentation hostname rather than the apex, which is the whole point: they are different files and only one of them is usually checked.

## Is llms.txt doing the work instead?

There is a second file that turns up on exactly these hosts. Every one of the 239 documentation hostnames was asked for /llms.txt, and 112 served a plain text file with content in it. The median was 23,954 bytes, the largest 2,354,595 bytes on www.twilio.com, and the 112 files carried 62,163 markdown links between them. Split by tier, 91 of the 142 documentation and developer hosts served one against 21 of the 97 help and support hosts, so this is a developer documentation convention rather than a help centre one.

That rate is far above the site level. This blog measured the same corpus at the apex on 19 September 2026 and found [302 of the 1,419 hostnames serving an llms.txt](https://lantad.co/blog/agentic-seo-one-agent-card-in-1419-sites), which is 21.3 percent against 46.9 percent here. The comparison is suggestive rather than controlled, and the stratum table says why: 60 of the 112 are in the SaaS stratum, software companies were already the heaviest publishers of the file, and this set of 239 is dominated by them by construction. The honest statement is that the file concentrates where developer documentation lives, not that putting documentation on a subdomain causes anybody to write one.

The overlap with the thin pages is the part worth sitting with. 15 of the 43 pages that delivered under 100 words sit on a host that serves an llms.txt, and three of the 15 zero word pages do. docs.stripe.com delivered 44 words of HTML and serves 92,764 bytes of llms.txt. clickhouse.com/docs delivered 23 words and serves 102,984. developer.workday.com delivered nothing and serves 146,237. docs.ramp.com delivered nothing, offered no link onward, and serves 14,961. Read generously, those teams have supplied a machine readable corpus precisely because they know the rendered page is not one. Read strictly, they have moved their documentation into a file whose own [proposal page](https://llmstxt.org/) describes itself as "a proposal to standardise on using an /llms.txt file" and says the specification "is open for community input", with a modified date of 10 August 2026, and which no crawler vendor's documentation commits to reading.

Both readings can be true, and this measurement does not settle which applies to any one site, because it observed no crawler fetching either file. What it does settle is the dependency. For those 15 pages, a reader that takes only the HTML gets under 100 words, and everything else rests on a convention rather than on anything a vendor has promised. The evidence this site has published on [what llms.txt actually achieves](https://lantad.co/blog/what-the-evidence-says-about-llms-txt) and on how [the file compares with a sitemap](https://lantad.co/blog/llms-txt-vs-sitemap-xml-23-urls-against-969) is the context for deciding whether that is a hedge or a single point of failure, and [the llms.txt tool](https://lantad.co/tools/llms-txt) will show what any given host is currently serving.

## What this measured, and what it did not

One page per site, and that page was the shallowest documentation or help surface the home page itself linked. These counts describe entry points, not documentation corpora. A hub page is frequently thinner than the articles beneath it, so a site whose entry page delivered 40 words may well serve 4,000 on every article under it, and nothing here contradicts that. What the measurement does establish is what a crawler receives at the door the site chose to advertise, which for seven of these hosts was neither words nor a link to anywhere else.

No JavaScript was executed and no browser was involved, which is the point rather than a limitation, but it bounds the claim precisely. A rendering reader would find more on many of these 43 pages. Google documents that it renders, on a queue it does not bound; no other vendor documents either way. So "fewer than 100 words" means fewer than 100 words in the delivered bytes, and it is a statement about the floor every reader shares, not a prediction about what any particular engine ends up holding. The same distinction runs through this blog's earlier measurements of [client side rendering](https://lantad.co/blog/client-side-rendering-11-of-271-returned-no-words) and of [text that lives only in hydration JSON](https://lantad.co/blog/hydration-json-scored-the-same-as-no-text), and the per stack fixes for the frameworks involved are written up for [Next.js](https://lantad.co/fix/nextjs) and [React](https://lantad.co/fix/react).

LantadBot is not GPTBot. Every AI crawler verdict above was computed from a parsed file with this scanner's own evaluator, never observed from crawler traffic, so a rule that blocks GPTBot on paper may or may not describe what reaches that site. The 403 responses are the same kind of unknown: an edge refusing an unfamiliar user agent is not evidence of a policy anybody set, and this blog has measured that gap separately. Word counts come from the shipped extractor, which drops script, style, noscript, template, svg and iframe content, so text that exists only inside those elements is counted as absent, which is a rule of this extractor rather than a claim about every reader. The corpus itself is an editorial frame of 1,419 hostnames rather than a random sample of the web, described in full on [the methodology page](https://lantad.co/methodology), and the strata in the table above are its strata, not the web's.

Finally, this says nothing about whether any engine cited any of these pages. Measuring citation is a different instrument, and what this site can say about the relationship between readability and being named in an answer sits on [the research page](https://lantad.co/research) and in the [AI visibility](https://lantad.co/glossary/ai-visibility) definition rather than in a corpus crawl. The practical check is narrower than any of that, and it takes a minute: request your own documentation hostname with no JavaScript and count the words, which is what [this tool](https://lantad.co/tools/what-gptbot-sees) does, then ask whether the file governing that hostname is the one you last edited. If you want the same read on the platform tokens rather than on your own pages, [the crawler list](https://lantad.co/tools/ai-crawlers) is the reference, and the platform specific routes into [ChatGPT citations](https://lantad.co/how-to-get-cited/chatgpt) are written up separately. Documentation is the content an answer engine is most likely to want. On 43 of 220 surfaces measured here, it was also the content least likely to arrive.

## Questions and answers

**Does documentation for AI search have to be server rendered?**

Nothing a vendor publishes requires it, but the delivered bytes are the floor every reader shares. Google's JavaScript SEO basics page, read on 16 October 2026, says Googlebot queues pages for rendering and that the wait can take longer than a few seconds, and no other AI crawler vendor documents rendering either way. In this run 43 of 220 documentation and help entry pages delivered under 100 words without JavaScript, and 15 delivered none.

**Why does it matter that documentation sits on a subdomain?**

Because robots.txt is per service. RFC 9309 requires the file at the top-level path of the service, so docs.example.com has its own file, separate from example.com. 187 of the 239 surfaces measured on 16 October 2026 were on a separate hostname, and on 23 of those 187 at least one of fifteen AI crawler tokens got a different verdict at the documentation path than at the site root.

**Is an llms.txt enough if the documentation page itself is empty?**

It is a convention, not a commitment. llmstxt.org describes itself as a proposal open for community input, and no crawler vendor's documentation promises to read the file. 112 of the 239 documentation hosts served one on 16 October 2026, and 15 of the 43 pages delivering under 100 words of HTML sat on such a host, which means those 15 depend on the convention holding.

**Which documentation in the corpus was readable?**

All 20 readable pages in the open source project documentation stratum carried 100 words or more, 16 of them carrying a Docusaurus signature, and the largest page in the whole run was the API reference at api.aiven.io at 68,142 words. The pattern is a static generator rather than a vendor: the ecommerce help centres, 8 of 13 under 100 words, were the weakest stratum in this frame.

---

Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering
crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup.

Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot