Do AI crawlers render JavaScript: 2 of 9 operators say either way

Do AI crawlers render JavaScript, according to their own documentation?

The set here is not a sample of the web's crawler documentation. It is exactly the set this scanner already depends on. Every entry in the registry carries a docsUrl, and those fifteen entries resolve to nine distinct pages because OpenAI documents three tokens on one page, Anthropic four on one page and Perplexity two on one page. If a claim about crawler behaviour is going to be made anywhere in this product, that is where it has to come from, so those nine pages are the ones worth auditing.

The method was deliberately dull. One GET per page, redirects followed, sent as LantadBot/1.0 with an Accept header naming HTML, which is the same identity our scanner publishes. No browser, no JavaScript, no retries. Each response was parsed with the extractor a scan uses, the visible prose was split into sentences, and every sentence containing JavaScript, render or headless was kept. Searching the prose rather than the raw bytes matters: the string render appears three times in the markup OpenAI's page ships and zero times in anything a reader would call a sentence, which is the sort of thing that turns a grep into a false finding.

Two pages returned a sentence. Seven did not. That ratio is the finding, and it is worth stating plainly what it does and does not mean. It does not mean seven crawlers run no JavaScript, and it does not mean they all do. It means that a site owner deciding how to serve seven of the nine largest AI crawling operations has nothing from those operations to decide against, which is a poor position for anyone trying to do generative engine optimization on evidence rather than on folklore. The absence is the result. We are reporting a documented silence, not a measured behaviour, and the distinction is the whole reason this post separates its two halves.

The two pages that answer, and the exact words they use

Common Crawl's FAQ answers under the question of how CCBot fetches a web page, and the answer ends: currently, JavaScript is not executed and Cookies are not used. That page returned 163 words of prose to us, the shortest of the nine and the only one that settles the question in the negative. It is also the operator behind the single largest published corpus of crawled pages, which is why an earlier post here went through what a non-JavaScript crawler actually archived in one month.

Apple's About Applebot page answers in the affirmative, and it is the only page in the set with a section heading dedicated to the subject. Under Rendering and robot rules it states that Applebot may render the content of your website within a browser, and then that if javascript, CSS, and other resources are blocked via robots.txt, it may not be able to render the content properly. The page carried a last updated date of 4 September 2026, the day before we read it, and it lives at support.apple.com/en-us/119829, which we name in plain text because our outbound policy has no entry for that host.

The Apple result carries a nuance we would rather state than bury. The token in our registry is Applebot-Extended, which is the preference token a site uses to refuse Apple's use of crawled content for training, and the rendering sentence is about Applebot, the fetching crawler. So the only affirmative statement in the set is about a crawler this scanner does not evaluate. That is a gap in our registry rather than a gap in Apple's documentation, and it belongs in the same family as the crawler tokens whose published user agent strings do not match what a log actually holds.

Google is the interesting silence, because Google does state a position, just not here. The crawler overview page our registry points at, last updated 2026-06-12, contains no sentence about rendering. The statement lives on Google's JavaScript SEO basics guide, last updated 2026-03-04, which says that once Google's resources allow, a headless Chromium renders the page and executes the JavaScript. The word doing the work there is allow, and a queue is a thing you wait in rather than a thing you have arrived at, which is why we treat rendering as a delay rather than a guarantee when reasoning about citations in AI Overviews.

Two crawler documentation pages our own extractor read nothing from

The two zero rows in the table above have completely different causes, and only one of them is anybody's failure.

ByteDance is the case the post is about. The page our registry points at, at zhanzhang.toutiao.com/docs/intro/26899, returned HTTP 200 and 91,455 bytes on 5 September 2026. Strip the tags and the scripts and 33 characters of text remain, which is the page title and nothing else. The word Bytespider appears 37 times in the response, every one of them inside a script payload rather than in a paragraph. So the documentation that tells a site owner how Bytespider behaves is itself delivered by exactly the mechanism that makes pages invisible to a crawler which does not render. We are not claiming that is ironic on purpose. We are reporting that a fetch with no JavaScript, of the kind Common Crawl documents itself performing, receives nothing usable from it.

Meta is the case that corrected us. Its page at developers.facebook.com/documentation/sharing/webmasters/web-crawlers returned 5,574 characters and zero words to our extractor, which looked at first like a second client rendered shell. It is not. The response carried Content-Type: text/markdown, and its body is complete, readable prose beginning with a markdown heading. Our extractor found no visible text because it parses HTML and there was no HTML to parse. The reader can see everything; our tooling saw nothing. That is a limitation in our pipeline rather than a defect in Meta's page, and it is a live one for anybody thinking about markdown as a delivery format for agents, which we went through when ten AI crawlers fetched markdown by URL and none by content negotiation.

Both rows argue for the same discipline. A zero is a measurement of a pipeline meeting a page, not a verdict on the page, and the only way to tell the two apart is to open the response and look. This is the same reason the scanner withholds a grade rather than guessing when one half of a measurement fails, and the same reason our methodology names what each component can and cannot see.

What a fetch with no JavaScript received from 40 real pages

Documentation says what an operator intends. It says nothing about your pages. So on the same day we requested 40 well known pages, one GET each, same user agent, no JavaScript, and ran the responses through the extractor and the parity corpus rule the scanner uses to decide what counts as main content. 31 returned HTTP 200. Nine returned HTTP 403 to us, among them four national news sites, and those nine are excluded rather than counted as empty, because a refused fetch is evidence about the refusal.

Across the 31 that answered, the shape is not uniform failure and it is not uniform health. Two returned zero extractable words: a Jira pricing page that shipped 1,394,000 bytes of HTML to do it, and a marketing pricing page that did the same in 74,485 bytes. Four more fell below 100 tokens in the corpus the scanner would measure against, including a database vendor's pricing page that returned 1,882,103 bytes and 50 words. At the other end, one pricing page returned 35,020 words and one encyclopedia article returned 4,133. The median across the 31 was one token of main content for every 310 bytes of HTML received.

Four of the 31 had no main content containers at all, which is the case our corpus rule exists for: where a page carries no main, article, heading, paragraph or list elements holding its text, prose parity would otherwise read as a confident zero on a page that is merely old fashioned. The fallback recovered 648 tokens from one, and 10, 41 and 27 tokens from three others, which tells you which of the four was genuinely container-less and which three were genuinely empty.

One more number is worth having, because it is where the missing text usually is. The extractor also harvests string values out of inline JSON payloads, the shape a hydration framework leaves behind, and 18 of the 31 pages carried at least one. The largest held 177,344 characters of harvested JSON strings; the next two held 69,389 and 41,594. That content is in the response, so a determined consumer could reach it, and it is not in the prose, so an ordinary extractor will not. We treat it as a separate reading called Retrievability for exactly that reason. Only 14 of the 31 pages carried a JSON-LD block at all, so for the other 17 there was no structured data in the response either. If your pages look like these, the fix is stack shaped rather than general, which is why the guidance splits into the Next.js guide, the React guide and the Shopify guide rather than one page of advice that fits nobody.

What this does not measure, and what to check on your own site

This post contains no measurement of any AI crawler's behaviour, and we want that said in its own paragraph rather than implied by careful phrasing elsewhere. We did not observe GPTBot, ClaudeBot or PerplexityBot fetching anything. We hold no log sample and no packet capture. Every statement about what an operator does comes from that operator's published page, quoted with the date we read it, and the two halves of this post are joined by a question rather than by a causal claim.

Nor did we render the 40 pages. The environment this measurement ran in has no outbound browser, so the raw fetch is all we have, and the honest consequence is that we can report what a non-rendering client received and not how much more a browser would have shown. A page returning 50 words to us might return 5,000 in a browser or might be a genuinely thin page. Telling those apart takes both halves, which is what a full scan does and what this run could not. Where a number here would have been better with a rendered comparison, it is absent rather than estimated.

What the evidence does support is narrower and still useful. Seven of the nine operators publish nothing about JavaScript execution, so any advice that begins with the phrase AI crawlers do not run JavaScript is citing something other than those operators. One operator states it does not. One states it may. And a site owner who wants a defensible answer for their own pages can get it without waiting for anybody's documentation, by comparing what a plain fetch returns against what a browser shows on the same URL, which is what what GPTBot sees does on a URL you choose and what the crawler access tester does for the permission question underneath it.

The practical order is unchanged by any of this. Permission first, then payload: a page that a crawler is allowed to fetch and that returns its text in the server response is legible to every client in the set, the ones that render and the ones that do not, which removes the delivery question entirely and leaves you arguing about content. That is the same conclusion an earlier audit of five real captures reached from the link graph rather than the prose, and it is the reason prose parity carries more weight in the composite than any other single component. If your text is in the HTML, you do not need to know the answer to this post's title.