BlogFindings
Content negotiation for markdown: 34 of 718 home pages answered, and six of those sent HTML
Lantad requested the home page of 1,027 hostnames twice on 18 September 2026, once normally and once with an Accept header naming text/markdown. 718 answered the first request with HTML. 34 answered the second with a markdown content type, six of those returned HTML bytes under that label, and four refused with 406 Not Acceptable. None of the nine AI crawler vendors behind the 15 tokens this scanner evaluates documents its crawlers asking.
We measured the second half on 18 September 2026 and read the first half at source the same day. The method is narrow on purpose. Two GET requests per hostname for every entry in this repository's industry corpus, 2,054 requests in total, both sent as LantadBot/1.0 with redirects followed, a 20 second timeout and no browser involved. The first request carried the default Accept header that curl sends. The second was identical except for one line, Accept colon text slash markdown, which is the header Cloudflare's Markdown for Agents documentation tells a client to send. 718 hostnames answered the first request with HTTP 200 and an HTML content type, and those 718 are the base for every figure below. Nothing here reports what any AI crawler actually did, because no access log was read.
In short
- Content negotiation for markdown reached a markdown content type on 34 of 718 home pages: Lantad asked each one twice on 18 September 2026, once normally and once with an Accept header naming text/markdown.
- Six of those 34 replies were mislabelled, with a markdown content type on a body that begins with an HTML doctype. All six are Shopify storefronts, so 28 of the 718 returned markdown a reader could use.
- None of the nine AI crawler vendors behind the 15 tokens this scanner evaluates documents its crawlers sending an Accept header for markdown, read at source on 18 September 2026. Google's crawler overview, last updated 2026-06-12, names Accept-Encoding and nothing else.
- Across the 11 sites that published both of Cloudflare's token headers, 73,396 markdown tokens replaced 1,696,789 HTML tokens on 18 September 2026, which is 95.7 percent fewer.
- Four of the 718 refused the request outright with 406 Not Acceptable rather than fall back to HTML: ca.gov, nature.com, contpaqi.com and otpbank.hu.
| What was asked | Hostnames | What came back |
|---|---|---|
| In the industry corpus | 1,027 | The sampling frame, unchanged since 3 August 2026 |
| Answered GET / with 200 and HTML | 718 | The base for every figure below |
| Asked again with Accept: text/markdown | 718 | One header changed, nothing else |
| Replied with a markdown content type | 34 | 4.7 percent of the 718 |
| Of those 34, bytes that were really markdown | 28 | Six sent an HTML doctype under a markdown label |
| Refused with 406 Not Acceptable | 4 | ca.gov, nature.com, contpaqi.com, otpbank.hu |
Do AI crawlers like GPTBot support content negotiation for markdown?
Not according to any of them. This scanner evaluates 15 crawler tokens belonging to nine vendors, and we opened all nine of the pages those vendors publish about their own crawlers on 18 September 2026 and searched each one for the string Accept and the string text slash markdown. Not one page says its crawlers send an Accept header naming markdown. Most of them say nothing about request headers at all.
The exception proves how narrow the subject is rather than widening it. Google's crawler overview, which carries Last updated 2026-06-12 UTC, is the only one of the nine that names an Accept header of any kind, and the one it names is Accept-Encoding: the page states that the content encodings supported by each Google user agent is advertised in the Accept-Encoding header of each request they make, and gives gzip, deflate, br as the example. That is compression, not format. OpenAI's crawler documentation carries no date and names GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot without describing a single request header. Anthropic's crawler article, updated 7 April 2026, and Perplexity's bots page, which carries no date, both list tokens and user agent strings and stop there. Apple's Applebot page, published 4 September 2026, Meta's web crawlers page, Amazon's Amazonbot page, Common Crawl's CCBot page and ByteDance's Bytespider page complete the nine, and none of those five mentions markdown or names an Accept request header. The only occurrences of the string accept across them are ordinary English and bundled JavaScript. Meta's page is worth one aside: it was served to us as text slash markdown, 5,580 bytes of it, without our asking and whatever Accept header we sent, which appears to be a decision about our user agent rather than about our preferences.
There is one first-party statement pointing the other way and it is worth quoting precisely, because it is about a different class of client. Cloudflare's launch post for the feature, Introducing Markdown for Agents, published 12 February 2026, says that we already see some of the most popular coding agents today, like Claude Code and OpenCode, send these accept headers with their requests for content. Coding agents are not indexing crawlers. They arrive because a person asked for a specific URL, which is the same distinction OpenAI draws between ChatGPT-User and GPTBot. So the honest answer to the question in this heading, as of today, is that the header is documented as something agents send and not as something the crawlers behind the 15 tokens in our crawler registry send.
That matters for how a site owner should read the rest of this post. Serving markdown is a real optimisation for agent traffic and for anything a person points at a URL. It is not, on the published evidence, a way to change what a training or search crawler reads, because the crawlers do not ask. The same gap runs through the rest of this subject: a name in a file is a claim about intent, and a user agent string is a claim rather than an identity.
| Vendor | Date on the page | Names an Accept header |
|---|---|---|
| 2026-06-12 | Accept-Encoding only | |
| OpenAI | No date | No |
| Anthropic | 7 April 2026 | No |
| Perplexity | No date | No |
| Apple | 4 September 2026 | No |
| Meta | No date | No |
| Amazon | No date | No |
| Common Crawl | No date | No |
| ByteDance | No date | No |
What 718 home pages did when we asked for text/markdown
34 of the 718 replied with a content type naming markdown. 673 replied 200 with the same HTML content type they had served a moment earlier, four replied 406 Not Acceptable, and seven answered something else: four connections failed at the transport layer on the second request, two returned 403 and one returned 401. So the plain rate at which a large organisation's home page will answer this header today is 4.7 percent, and the rate at which it will ignore it entirely is 93.7 percent.
Where those 34 sit is more useful than the headline. 22 of the 116 saas hostnames answered, which is 19 percent of that sector and by a wide margin the highest. Ecommerce contributed eight of 59, government one of 86, finance two of 87 and travel one of 73. Education, healthcare and news contributed none at all between them, across 105, 96 and 96 hostnames respectively. This is the same shape we keep finding in this corpus: developer-facing companies adopt a machine-readable convention early and everybody else does not, which is exactly how 31 llms.txt files carrying 75 dead links and 32 of 44 documentation sites serving llms-full.txt distributed themselves.
The response headers add a detail that is easy to miss and worth checking on your own site. 27 of the 34 named Accept in a Vary header, which is what tells a cache that this URL has more than one representation. Seven did not: typeform.com, zendesk.com, netlify.com, snyk.io, zapier.com, supabase.com and sentry.io all varied their content on a request header without declaring it. Going the other way, nine of the remaining 684 named Accept in Vary and then served HTML anyway, among them mpg.de, uni-heidelberg.de, grist.org, spectator.co.uk, airtable.com, elastic.co and klarna.com. A Vary header is a promise about behaviour and on both sides of this measurement it was inaccurate more often than a cache would like.
This is the same mechanism as language negotiation, one field along. We asked the same corpus a comparable question with a German Accept-Language header and found that 27 of 719 home pages changed language on a header alone. Format negotiation is rarer than language negotiation on the same sites, which is what you would expect from a convention that is months rather than decades old.
Six of the 34 sent HTML under a markdown label
Six of the 34 markdown content types were wrong. allbirds.com, brooklinen.com, everlane.com, glossier.com, gymshark.com and rothys.com each returned a response headed content type text slash markdown whose body opens with an HTML doctype and continues as an ordinary page. The byte counts say the same thing without needing the bodies read: brooklinen.com returned 1,654,990 bytes against 1,655,310 for the HTML response, a difference of 320 bytes, and gymshark.com returned 50,750 against 51,010, a difference of 260. The other four sit in the same range, with everlane.com the widest at 5,497 bytes out of 859,640. Nothing was converted. A label was changed.
All six carry Shopify markup signatures, which we checked by looking for cdn.shopify.com and theme references in the bytes they returned, and all six answer from Cloudflare. None of the six carried an x-markdown-tokens header, which 16 of the other 28 did, and that absence is consistent with the byte counts rather than proof on its own, since 12 responses that really were markdown did not carry it either. We are not in a position to say which layer produced the label, and we are not going to guess: the observation is that the content type and the content disagree, on six storefronts, on one day.
The cost of this is not theoretical for anything that trusts the header. A client that asks for markdown, reads text slash markdown in the reply and hands the body to a markdown parser gets a document full of script tags and class attributes. That is a worse outcome than the 673 sites that ignored the header, because those at least told the truth about what they were sending. It belongs in the same family as the parity failures we keep measuring on storefronts, where 38 Shopify stores served 1,112 products as open JSON that nobody had decided to publish, and it is worth adding to the list on our Shopify fix page for anyone running one.
Counting the six out leaves 28 of 718, or 3.9 percent, as the rate at which asking for markdown on this corpus gets you markdown. That is the number we would use. It is also a reminder that a content type is a claim like any other header, and the only way to know is to read the first bytes.
GET / with Accept: text/markdown, six Shopify storefronts, 18 September 2026
- GET https://brooklinen.com/ with Accept: text/markdown 200
- content-type: text/markdown; charset=utf-8 header says markdown
- body begins: <!doctype html> <html lang='en'> bytes say HTML
- 1,654,990 bytes against 1,655,310 for the HTML reply 320 bytes apart
- x-markdown-tokens absent on all six
- Same result on allbirds, everlane, glossier, gymshark, rothys 6 of 34
What markdown saves, measured from the servers' own token headers
Sixteen of the 34 responses carried an x-markdown-tokens header, and eleven carried x-original-tokens alongside it. Those two fields are defined in Cloudflare's Markdown for Agents documentation, which was last updated on 13 July 2026 and describes them as the estimated number of tokens in the markdown document and in the original HTML document before conversion. They are the origin's own estimate rather than ours, and we are reporting them as such, but they have the useful property of being produced by the machine doing the conversion rather than by a vendor making a case.
Across the eleven that published both, 73,396 markdown tokens replaced 1,696,789 HTML tokens, which is 95.7 percent fewer. The range is narrow. argentina.gob.ar was the smallest saving at 90.7 percent, 706 tokens against 7,567, and shein.com the largest at 99.2 percent, 2,226 against 276,426. In between, gitlab.com reported 5,204 against 56,512 and visitbritain.org 19,961 against 350,656. Cloudflare's launch post reports a comparable figure for its own blog post, 3,150 tokens in markdown against 16,180 in HTML, which it describes as an 80 percent reduction.
Measured in bytes rather than tokens, across the 28 responses that really were markdown, 392,693 bytes replaced 19,895,172, a 98.0 percent reduction with a median of 98.5 percent per site. The extremes are instructive. sentry.io returned 976 bytes against 630,646, and its markdown is a hand written stub that opens by telling the reader they have hit the web UI and that it is HTML meant for humans. argentina.gob.ar, the smallest saving in the set, returned 2,824 against 30,653. A big reduction usually means the HTML was mostly framework rather than that the markdown was rich, which is the same accounting we ran on Next.js, where 28 home pages sent 15 times more payload than prose.
Two cautions belong with these numbers. A token count in a response header is a server's estimate and no tokeniser is named, so treat the percentages as the shape of the saving rather than as a figure to budget against. And a smaller document is not automatically a better one: some of these markdown responses are a title, a description and a handful of links, which is cheap to read and carries less than the page did. Prose parity is the measure that catches that, and it is not what a byte count measures.
| Hostname | Markdown tokens | HTML tokens | Fewer |
|---|---|---|---|
| shein.com | 2,226 | 276,426 | 99.2% |
| backmarket.fr | 8,533 | 262,983 | 96.8% |
| zendesk.com | 7,710 | 214,372 | 96.4% |
| tiendanube.com | 8,732 | 235,125 | 96.3% |
| visma.com | 951 | 23,241 | 95.9% |
| visitbritain.org | 19,961 | 350,656 | 94.3% |
| clickhouse.com | 5,317 | 87,228 | 93.9% |
| pipedrive.com | 6,969 | 104,400 | 93.3% |
| typeform.com | 7,087 | 78,279 | 90.9% |
| gitlab.com | 5,204 | 56,512 | 90.8% |
| argentina.gob.ar | 706 | 7,567 | 90.7% |
Four sites answered 406 Not Acceptable rather than serve the page
ca.gov, nature.com, contpaqi.com and otpbank.hu each replied 406 Not Acceptable to the request carrying the markdown Accept header, having served a normal HTML home page to the identical request a moment earlier. nature.com's refusal was 532 bytes and ca.gov's was 98. This is the failure mode that makes the header worth thinking about before a crawler ever sends one, and it is not a bug in those four servers. It is one of the two behaviours the specification explicitly allows.
RFC 9110, the IETF's HTTP Semantics document published in June 2022, defines the Accept field in section 12.5.1 and the 406 status in section 15.5.7, where it states that the status code indicates that the target resource does not have a current representation that would be acceptable to the user agent, according to the proactive negotiation header fields received in the request, and the server is unwilling to supply a default representation. Section 12.1 spells out the choice a server has: if a content negotiation header field is present in a request and none of the available representations for the response can be considered acceptable according to it, the origin server can either honor the header field by sending a 406 response or disregard the header field. Both are conforming. The same section warns a client directly that a user agent cannot rely on proactive negotiation preferences being consistently honored.
That is the strongest argument in this whole subject for why a crawler operator would not adopt the header quickly. A crawler that starts sending Accept colon text slash markdown across the open web gets markdown from four percent of sites, HTML from most of the rest, and nothing at all from a small tail that refuses. It would have to send the header, handle a 406, and retry without it, which is two requests where one used to do. The alternative conventions cost the crawler nothing. One is a path suffix: requesting OpenAI's own crawler documentation page with .md appended to its URL returned 4,872 bytes of text slash markdown when we tried it on 18 September 2026, though the page itself never says so, and the string markdown does not appear in its HTML. The other is a declared alternate, which is how docs.perplexity.ai advertises a markdown twin of its own crawlers page, through a link element with rel alternate and type text slash markdown pointing at a .md URL that returned 6,834 bytes the same day. Neither risks a 406.
The file based convention sits in the same space and has the same advantage of never breaking a normal request. The llms.txt proposal puts a markdown index at a fixed path rather than negotiating for it, and we have written about what llms.txt and robots.txt each actually decide. Content negotiation is the most elegant of the three and the only one that can fail closed.
-
Ignored the header, served HTML673 sites Conforming under RFC 9110 section 12.1, which permits disregarding the field -
Served real markdown28 sites Bytes that begin as markdown rather than as an HTML doctype -
Served HTML under a markdown label6 sites All six Shopify storefronts, none carrying a token count header -
Refused with 4064 sites ca.gov, nature.com, contpaqi.com, otpbank.hu -
Answered 401, 403 or failed7 sites Four transport failures, two 403s and one 401 on the second request only
What this measurement does not show, including on our own site
This is two requests per hostname from one network location on one day, sent by a client that says it is LantadBot. It is a reading of responses, not of behaviour. No access log was read, no AI crawler was observed asking any site for markdown, and nothing here is evidence that any model received any of these documents. A site that answers our header would presumably answer anyone's, but we did not test that, and a server free to vary on Accept is equally free to vary on user agent.
Three further limits are worth naming. The 309 hostnames that did not answer the baseline request with 200 and HTML are absent from every figure, and sites most defensive about bot traffic are over-represented in that group, so the 4.7 percent is a rate among sites that talk to us. Only the home page was asked, and a documentation subdirectory is far likelier to negotiate than a marketing root, which means the SaaS figure of 22 in 116 is a floor for those companies rather than a description of their whole estate. And we sent one media type; a client sending a longer Accept string with quality values could get different answers from the same servers.
The inconvenient part applies to this site. lantad.co does not implement content negotiation either. Asked for text slash markdown on 18 September 2026, it returned the same 146,722 bytes of HTML it returns to anyone. What it does instead is publish a markdown twin at a path suffix: every page has a .md URL and they are all listed, and index.md came back as 9,413 bytes of text slash markdown, 93.6 percent smaller than the HTML. We chose the suffix over the header for the reason this post keeps arriving at, which is that no crawler we measure is documented as sending the header, and we would rather ship the convention that works today than the one that is tidier.
If you are deciding where to spend effort, the ordering we would defend from this run is unglamorous. The things that decide whether a crawler gets your content at all come first: whether the server answers it, which we found 79 of 115 sites failing on pages their own robots.txt allows, and whether the page carries words in the raw bytes, where we found 17 of 380 home pages sending a crawler none. Markdown negotiation is a real improvement for agent traffic and it is measurably cheap where it is already available on your platform. It is not, on today's evidence, a lever on what a training or search crawler reads. Our own fetch rules and conduct policy and the scoring behind every figure we publish are written up if you want to check how any of this was produced.
-
Responses to an Accept headerMeasured 1,436 requests across 718 hostnames answering the baseline -
What vendor documentation saysMeasured Nine crawler pages opened and searched at source on 18 September 2026 -
What any AI crawler actually sendsNot measured No access log was read and no crawler was observed -
Pages other than the home pageNot measured One URL per hostname, so documentation subtrees are excluded -
Quality of the markdown servedNot measured Byte and token counts say nothing about what the document retained -
Effect on any citationNot measured No model was asked anything in this run
Lantad
Published .
Somebody typing this question into a search box is usually one step into a decision. They have read that markdown is cheaper for a model to read than HTML, they have found that their stack can serve it, and before they spend a sprint on it they want to know whether anything at the other end is going to ask. That is a question with two halves, and both are answerable without guessing: what the crawler vendors say their crawlers send, and what real servers do when something does ask.
Common questions
Do AI crawlers like GPTBot support content negotiation for markdown?
No vendor documents it. Lantad opened the crawler documentation of all nine vendors behind the 15 tokens this scanner evaluates on 18 September 2026 and none states that its crawlers send an Accept header naming text/markdown. Google's crawler overview, last updated 2026-06-12, is the only one naming an Accept header at all and the one it names is Accept-Encoding, which is compression rather than format.
How many sites actually serve markdown if you ask for it?
34 of 718 on this corpus, measured 18 September 2026, which is 4.7 percent. Six of those 34 returned HTML bytes under a markdown content type, so the rate at which asking produced usable markdown was 28 of 718, or 3.9 percent. SaaS hostnames accounted for 22 of the 34, while education, healthcare and news contributed none across 297 hostnames.
Can asking for markdown break a request?
Yes, on a small minority. Four of the 718 hostnames answered 406 Not Acceptable rather than fall back to HTML: ca.gov, nature.com, contpaqi.com and otpbank.hu. RFC 9110 section 12.1 permits both behaviours, letting an origin either honour the header with a 406 or disregard it, and it warns that a user agent cannot rely on proactive negotiation preferences being consistently honored.
How much smaller is the markdown version of a page?
Across the 11 hostnames that published both of Cloudflare's token count headers on 18 September 2026, 73,396 markdown tokens replaced 1,696,789 HTML tokens, which is 95.7 percent fewer, with a per-site range from 90.7 to 99.2 percent. Measured in bytes across all 28 real markdown responses, 392,693 bytes replaced 19,895,172, a 98.0 percent reduction with a median of 98.5 percent.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
Prose Parity
Prose Parity is Lantad's metric for the share of a page's rendered, human-visible text that is present in the raw HTML an AI crawler receives, where 100% means a crawler sees everything a human visitor sees.
llms.txt
llms.txt is a proposed plain-text markdown file placed at a site's root that gives AI systems a curated map of its most important content, so models can find and read what matters without parsing every page.
Framer AI crawlers: 35 of 36 sites served markdown, and one robots.txt named a crawler
Lantad requested the home page of 392 hostnames on 11 September 2026 and found 36 served by Framer. Asked with an Accept header naming text/markdown, 35 of the 36 returned a markdown document. Asked for robots.txt, 32 of the 33 that answered HTTP 200 returned the same four line file, and one of the 33 named an AI crawler token at all.
llms-full.txt: 32 of 44 sites served one, and the specification never names it
Lantad requested /llms.txt and /llms-full.txt once each from 54 documentation and developer tool hostnames on 12 September 2026. 44 returned an llms.txt as plain text or markdown, 32 of those also returned an llms-full.txt, and not one returned the second file without the first. The llms.txt proposal, modified on 10 August 2026, does not contain the string llms-full.txt anywhere.
Hreflang and AI crawlers: 27 of 719 home pages changed language on a header alone
Lantad requested the home page of 1,027 hostnames on 15 September 2026 and read the bytes with no JavaScript executed. 730 answered with a 2xx and an HTML content type, 255 of those declared a hreflang alternate, and asking the same 730 again with a German Accept-Language header changed the declared language on 27 of them. Four of the 27 publish no alternate at all.
