Ten AI crawlers fetched markdown by URL, and none of them by content negotiation

What content negotiation actually asks a server to do

Content negotiation is section 12 of RFC 9110, the HTTP semantics specification published in June 2022. The variety relevant here is what the specification calls proactive negotiation, which it describes as a mechanism for selecting a response representation based on active request header fields. In practice that means the client sends a header listing what it can handle and the server chooses from that list.

The header is Accept, defined in section 12.5.1 with a sentence short enough to quote in full: the Accept header field can be used by user agents to specify what response content types are acceptable. Mozilla's reference page for the Accept header puts the same idea in terms of the round trip, saying the header indicates which content types the sender is able to understand, and that in requests the server uses content negotiation to select one of the proposals and informs the client of the choice with the Content-Type response header.

Two properties of that design matter for anyone planning around it. The first is that the preference is expressed by the client and nowhere else. A URL carries no opinion about media types, so a server that wants to vary its answer has to read something the requester sent. The second is that the specification names a cost. RFC 9110 states as the disadvantage of proactive negotiation that it is difficult for the server to determine what representation would be best for the user. A server guessing on the strength of a header is a server that can guess wrong, which is why the answer has to be marked as varying by Accept before any cache stores it.

This is the same class of mechanism as the language case, and it fails in the same direction. When Google documents that Googlebot sends no Accept-Language header, the consequence is not that the crawler receives an error, it is that the server quietly answers for it. A crawler that expresses no preference about media types gets HTML, because HTML is what the address returns when nothing asks for anything else. Nothing fails, nothing is logged as a problem, and the site owner who configured the alternative has no signal that it went unused.

Cloudflare built a markdown feature that keys entirely on the header

On 12 February 2026 Cloudflare announced Markdown for Agents in its changelog, describing it as support for real time content conversion at the source, for enabled zones using content negotiation headers. The product documentation is specific about what triggers it: to fetch the markdown version of a page from a zone with the feature enabled, the client needs to add the Accept negotiation header with text/markdown as one of the options. Cloudflare then fetches the HTML from the origin and converts it before serving.

The engineering around that trigger is careful. The converted response carries Content-Type set to text/markdown with a charset, and two headers reporting an estimated token count for each version, x-markdown-tokens for the converted document and x-original-tokens for the HTML it came from. The response includes Accept in its Vary header so that caches store the two variants separately, which is the correct handling and the part most hand rolled implementations get wrong. Origin headers are preserved through the conversion so that security and cache relevant headers survive. The documented ceiling is an origin response of 2 MB, a limit in the same territory as the first two megabytes Googlebot reads and worth knowing for the same reason.

The output shape is the interesting part for anyone thinking about extraction. Cloudflare states that the response follows a consistent, predictable structure so that AI systems can rely on it without per site parsing logic: YAML frontmatter built from the page's meta tags, then the body converted to markdown, then any JSON-LD preserved as a fenced code block at the end. That last detail means structured data survives a conversion that discards almost everything else, which is a deliberate and sensible choice.

None of this is criticism of the feature. It is a well specified answer to a real problem, and it is opt in rather than a default, so no site gets it by accident. The point is narrower and it is about sequencing. The entire mechanism is gated on a request header, and the only published measurement of whether crawlers send that header landed three weeks later.

The column that was zero in every row

Buytaert's post is titled around markdown, llms.txt and AI crawlers, and it is available at dri.es/markdown-llms-txt-and-ai-crawlers. It reports a month of Cloudflare logs from his own site and breaks the traffic down by crawler into total requests, HTML files, markdown files and a column counting requests that used content negotiation. His summary sentence states that AI crawlers rarely request markdown and never use content negotiation, and the table backs it: the content negotiation column reads zero for Amazonbot, ChatGPT-User, Meta AI, ClaudeBot, OAI-SearchBot, GPTBot, Bytespider, CCBot, PerplexityBot and Claude-User alike. He puts it plainly in the prose as well, writing that no AI crawler uses content negotiation, not one.

The ten rows sum to 58,279 requests, of which 53,312 were HTML and 4,967 were markdown, which is arithmetic on his published figures rather than a number he states. Every internal percentage in the table checks out against its own row, which is worth saying because a table that fails its own arithmetic is the first sign a figure has been retyped rather than measured.

Two caveats belong here and neither is hidden in his post. This is one site, so it is a sample of one origin's traffic rather than a survey, and a crawler's behaviour on a technical blog run by a well known open source developer is not automatically its behaviour everywhere. It is also a month, in a field where crawler tokens get renamed and fleets get split, so it is a reading rather than a constant. What makes it worth reporting anyway is that it is a direct observation of the thing the advice depends on, taken at the layer where the answer is decided, and nobody else appears to have published one.

He measured llms.txt in the same logs and the result is of a piece with the rest. He reports 52 requests for the file in the month, states that every one came from an SEO audit tool, and says that not a single request came from an AI answer engine or crawler. That does not settle the value of llms.txt as a specification, and the v2 revision of the spec changes what the file can express, and Chrome's agentic browsing audit checks for the file whatever crawlers do with it, but it is one more instance of the same pattern: a file or a header published for crawlers, and crawlers not reaching for it.

They fetched markdown anyway, by address rather than by header

The finding is not that crawlers ignore markdown. Nearly five thousand of the requests in that table were for markdown files, and two crawlers reached for it heavily: GPTBot took markdown on 34.8 percent of its requests and OAI-SearchBot on 22.7 percent. Something in those fleets clearly prefers the format when it can get it.

What they did not do is negotiate. Buytaert states that the crawlers only discover the markdown pages through the dedicated URLs, and only via the auto discovery link. The distinction is the whole practical content of the finding. A markdown representation reachable by content negotiation is invisible to a client that never asks, because there is no address to crawl and nothing in the HTML to follow. A markdown representation published at its own address is an ordinary URL: it appears in links, it can sit in a sitemap, which tells a crawler where pages are, it gets crawled by anything that crawls, and it can be requested and inspected by a site owner with no special tooling.

That is the pattern this site was already built on, which is worth stating as a fact about the code rather than as a recommendation. Every public page here has a markdown twin at its own address, generated by a route that appends .md to the path, so this post is served at /blog/no-ai-crawler-used-content-negotiation.md and the site root at /index.md. There is a human readable directory of all of them at the markdown index and a flat machine readable list at /llms.txt. No content negotiation is involved anywhere in that, and on the evidence above that is the arrangement crawlers actually use.

The header approach and the address approach are not in conflict and a site can do both. But they carry different risk. If you publish markdown at addresses and no crawler wants it, you have spent some build time and added some URLs. If you configure negotiation and no crawler asks, you have a feature that produces no observable difference and no error, which is the harder failure to notice. This is the same distinction as the two layers that decide whether AI can read your site: what you configured and what actually came back are separate questions, and only the second one is evidence.

The clients that do send the header are not the crawlers

A header nobody sends would be a strange thing to build on, and the fuller picture is that somebody does send it, just not the population being optimised for. On 19 February 2026 the monitoring company Checkly published the request headers of seven AI coding agents, captured by pointing each one's own fetch tool at an endpoint that echoes back what it received. Its post is at checklyhq.com/blog/state-of-ai-agent-content-negotation, and it names versions for each agent tested.

Three of the seven asked for markdown. Checkly reports Claude Code sending an Accept header of text/markdown, text/html and a wildcard; Cursor sending text/markdown first with a quality ranked list behind it; and OpenCode sending text/markdown at the top quality value with three fallbacks below. The other four did not: OpenAI Codex and GitHub Copilot sent a standard browser style Accept header, while Gemini CLI and Windsurf sent a bare wildcard. Checkly's own summary of the result is that only three out of seven agents request markdown.

Those are developer tools fetching a page because someone asked them to, which is a different population from the indexing crawlers in Buytaert's logs, and the two measurements are not in tension. Read together they say the header is used by interactive tooling and not by the fleets that crawl at volume. That distinction has shown up in this corpus before from the other direction, when coding agents arrived identifying themselves as curl and axios rather than under product names.

It also means the honest version of the advice depends on who you are serving. Documentation aimed at developers whose agents fetch pages on demand has a real case for content negotiation, because a measurable share of those clients ask. A marketing site hoping to be read by the crawlers that feed answer engines has, on the only published evidence, no client to negotiate with. Both of those are defensible positions and they are not the same position, which is the part the generic version of the advice loses.

What a site owner can check today without waiting for anyone

None of the above tells you what happens on your origin, and your logs are the only source that does. The useful thing about this particular question is that it is cheap to answer, because content negotiation leaves a trace on the request rather than in some vendor's dashboard.

Start with the request headers you already store. If your log format records Accept, count how many requests from named AI crawler tokens carry a media type other than a wildcard. If it does not record Accept, that is the change to make first, because you cannot reason about negotiation from a log that discards the field it turns on. Then separate the crawlers from everything else, using the tokens the vendors publish rather than a guess, remembering that most vendors publish more than one token and that a rule written against a single name will miss the rest of the fleet.

If you already serve markdown at addresses, count the fetches. That is a plain URL comparison and it needs no header parsing at all: requests for /page against requests for /page.md, grouped by user agent. That number is the one Buytaert's table is made of, and it is the one that told him crawlers wanted the format even though they never negotiated for it.

The wider check is unchanged by any of this and it is still where most of the loss is. Whether a crawler receives markdown or HTML matters far less than whether the HTML it receives contains the text a reader sees, which is the thing prose parity measures and the thing that fails when content lives somewhere the fetcher cannot reach, as it does with text inside a shadow root. A markdown file generated from a page whose text arrives via client side rendering carries the same hole the HTML did. Serving the format correctly is worth doing after the content is there, not instead. How Lantad grades a page treats it that way, and the format of the response is not one of the weighted signals.

The last check is the one that costs nothing and gets skipped anyway: request your own alternative representation and read what comes back. Send an Accept header naming text/markdown to a page you have configured, and separately request the .md address if you publish one. A configuration that quietly does nothing returns a perfectly ordinary 200, which is why nobody notices, and why the only way to know is to look. Our own crawler's behaviour is documented for the same reason.