Framer AI crawlers: 35 of 36 sites served markdown, and one robots.txt named a crawler

What do Framer AI crawlers find in robots.txt?

We requested /robots.txt from all 36 hostnames twice, once with redirect following switched off and once with it on. Fifteen of the 36 answered the first request with a 3xx to the www host, which is ordinary and resolves to a file on the same registrable domain. Thirty three returned HTTP 200, one returned 403 carrying a Cloudflare page titled Edge IP Restricted, and two returned 404.

Thirty two of the 33 files that answered 200 were the same document. User-agent followed by a star, Allow followed by a slash, a blank line, and a single Sitemap line pointing at the site's own sitemap. Between 60 and 86 bytes, and the only difference between any two of them was the hostname inside the sitemap URL. Not one of those 32 names GPTBot, ClaudeBot, PerplexityBot, Google-Extended or any of the other tokens our registry evaluates. There is no group for an AI crawler to match other than the wildcard, and RFC 9309, the robots exclusion protocol published in September 2022, is explicit that a crawler matching a group carrying Allow with a slash may fetch the whole server. You can check the same thing on your own domain with the robots.txt tester, which evaluates each token separately rather than reading the file as prose.

That is a deliberate default rather than an omission. Framer's own help article on accessing the file, read on 11 September 2026 and carrying an updated date of 7 August 2026, states that Framer automatically generates a robots.txt for every published site, and that a custom file can be uploaded through Static Files on Pro and Enterprise plans, replacing the generated one. The article is at www.framer.com/help/articles/how-can-i-access-the-robots-txt-file/ and is worth reading before anyone tells you to edit a file you cannot edit on the plan you are on.

It is also a different shape from every other hosting product we have measured in the last week. Webflow left 20 of 40 files with no rule at all, seven of them zero bytes. Wix named no AI crawler on any of 69 files. Squarespace generated a file naming 26 tokens and put them in the same group as the wildcard, which allows them. WordPress sites named none on 16 of 24. Framer is the tidiest of the five and lands in the same place: a file that says yes to everything, written by the platform, on a site whose owner has probably never opened it.

One file in the 33 was different. It carried the Content-Signal preamble and the begin and end markers of Cloudflare managed content, with a wildcard group setting search equal to yes and ai-train equal to no, then nine Disallow groups naming Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, a Cloudflare browser rendering crawler, Google-Extended, GPTBot and meta-externalagent. Eight of those match tokens in our registry. The seven registry tokens it does not name, among them OAI-SearchBot, PerplexityBot and Claude-SearchBot, fall through to the wildcard and are allowed, which is the intended shape of that template: block the training fleet, keep the search fleet. We have written separately about how Cloudflare generates that file and moves its token list without you, and about how the Content-Signal line asks while the Disallow lines under it block. The point for this sample is narrower. That file carries Cloudflare's own begin and end markers, it was served with a Cloudflare Server header rather than the Framer one, and its text is the template rather than anything hand written, so the one site of 33 with an AI crawler policy in its robots.txt appears to have acquired it at the CDN rather than in the site builder. We cannot see a site's settings from outside and do not claim to.

Does Framer serve markdown to AI crawlers?

Framer publishes a claim that can be tested from outside, which is rarer than it sounds. Its help article on making a site readable by AI agents, also carrying an updated date of 7 August 2026 and living at www.framer.com/help/articles/make-site-readable-by-ai-agents/, states that Framer serves a markdown version of every page on the same URL, and that when an agent requests a page with an Accept header naming text/markdown it receives markdown instead of HTML. The article calls that the primary method used by most AI tools. It documents a second route as well, a query parameter of md appended to any page URL.

We tested both on all 36 hostnames. Thirty five of the 36 returned a response with Content-Type set to text/markdown and a charset of utf-8 when the request carried the Accept header. The one that did not was the site returning 404 for everything. The query parameter route worked on 33 of the 36. The two sites that honoured the header but returned HTML for the parameter both reach the Framer edge through something else first: one answers from nginx behind CloudFront, and the other begins with a 301 from an Amazon EC2 host and arrives at Framer as HTML two redirects later. A query string that does not survive a redirect chain is the ordinary explanation, and it is a reminder that the feature belongs to the edge rather than to the site.

The responses are correct HTTP rather than a trick. Every one of the 35 carried a Vary header listing Accept alongside Accept-Encoding, which is what RFC 9110 requires of a server that varies its representation by request header, and it is what stops a shared cache from serving the markdown to a browser or the HTML to an agent. Content negotiation has been in the specification since the early web, and this is a textbook implementation of it.

The documents themselves are clean. All 35 opened with YAML front matter carrying the page title and meta description, and in many cases a published timestamp. Twenty two of the 35 carried the Navigation section Framer's article says it appends at the bottom of each page, a short list of internal links; the other 13 did not, which on a small single page site is the expected outcome rather than a fault. Links inside the body arrive as relative markdown links. Across those same 35 URLs the median markdown document was 4,204 bytes and the median HTML document was 561,074 bytes, and the median ratio of one to the other, taken site by site, was 0.75 percent. On one site in the sample it was 7,907 bytes of markdown against 1,449,770 bytes of HTML.

Does any AI crawler actually ask for markdown?

This is the part that decides whether the feature is worth anything to a site owner today, and it is the part a platform cannot answer from its own documentation. Serving markdown on content negotiation is only useful if something sends the header.

The one logged dataset this site has reported points the other way. Dries Buytaert published a month of Cloudflare logs from his own site on 5 March 2026, broken down by crawler, and we covered it in ten AI crawlers fetched markdown by URL and none of them by content negotiation. Across ten AI crawlers and 58,279 requests, 4,967 fetches were for markdown, and the content negotiation column read zero for every single crawler. Markdown was being fetched roughly 8.5 percent of the time, and it was being fetched by requesting a dedicated address that ends in md rather than by asking for a media type. That is one site's logs for one month, published by one person, and it is not a measurement of Framer or of anything in this sample. It is also the only published evidence on the question that we have found.

Read together with today's scan, the two produce an uncomfortable pairing. The mechanism Framer's article calls the primary method used by most AI tools is the mechanism the one available log study recorded nobody using. The query parameter route is closer to the pattern the logs did record, since it is a distinct URL a crawler can discover and follow, but it is not linked from the HTML, it does not appear in the sitemap, and nothing in the generated robots.txt points at it. A crawler that never guesses the parameter never sees it.

None of that makes the feature wrong. Header based negotiation is the correct engineering answer, it costs the site owner nothing, it was implemented properly on 35 of 36 sites we asked, and the population that does send the header today is developer tooling rather than indexing crawlers, which may change. It does mean the honest description of a Framer site's current AI posture is the HTML and the robots.txt, not the markdown. If you want a route that crawlers demonstrably do follow, a discoverable file at a fixed path is the pattern with evidence behind it, which is the argument for llms.txt, with the large caveat that the evidence for llms.txt being read at all is itself thin.

Can a crawler read a Framer page without running JavaScript?

Framer's article states that although its sites are built with React, every page is pre-rendered to HTML on its servers before being served, so agents that do not execute JavaScript still receive the full text. That is also testable, and it held.

We took the raw HTML of each home page exactly as it arrived, stripped script, style, noscript and comment content without executing anything, and counted the words left. Across the 34 hostnames that answered HTTP 200, the lowest count was 153 words and the highest was 6,473, with a median of 1,119. Not one of the 34 came back under 100 words. All 34 carried a meta description in the head. On a client rendered site that figure collapses: we have measured a page where the text lived in hydration JSON and scored the same as no text at all, and the general failure is the subject of our guide to fixing prose parity on React sites. Framer sites do not have that problem, and a reader evaluating the platform on prose parity alone should know that it passes.

There is a cost attached that the article does not mention. The HTML those pages ship is very large. Across the 34 the smallest was 136,381 bytes and the largest 1,807,067, with a median of 564,231, and seven of the 34 exceeded one megabyte. None exceeded two. That matters because the only crawler vendor that publishes a byte limit sets it at roughly that order: Googlebot reads the first 2MB of a file, and three AI crawler vendors name no limit at all, which means nobody outside those companies knows where their own cutoff sits. A one megabyte page of mostly inlined assets carrying 1,100 words of prose is a thin ratio to be sending to a client whose budget is undocumented.

That is the strongest practical argument for the markdown route on this platform, and it is worth stating plainly even though the previous section found no evidence anything is asking for it. The same page as markdown is under one percent of the bytes and is almost entirely prose. If a crawler ever does send the header, it gets a far better deal than the HTML offers. Today, the HTML is what arrives.

What the platform does not do for you

Three things in this scan are the site owner's job, and the sample shows most owners have not done them.

Structured data is the largest gap. Twenty seven of the 34 home pages carried no JSON-LD block of any kind, and seven carried at least one. Framer's own article is accurate about this: it says agents receive any JSON-LD structured data you have added, which is a precise way of saying the platform adds none. The vocabulary is published at schema.org and what it is for in this context is set out in our note on structured data. Whether it moves anything is a separate question we have tried to answer honestly elsewhere, having found that three of five captured pages carried JSON-LD and one held almost all of it.

Heading structure is the second. Across the 34 pages, five carried no h1 element at all, ten carried exactly one, and 19 carried more than one. The highest count on a single home page was 36. A visual builder makes it easy to promote any large piece of text to a heading, and the markdown conversion inherits the result: the markdown documents showed the same pattern, with 18 of 35 carrying a single top level heading and 11 carrying several. We have written before about how an h1 inside a header element counted as no h1 at all for an extractor that is reading structure rather than styling.

The third is the robots.txt itself, and it is the one with a plan restriction attached. Uploading a custom file requires Static Files, which Framer's article places on Pro and Enterprise plans. A site owner on a lower plan who reads the standard advice about blocking or allowing named crawlers cannot act on it in that file at all. Whether they should want to is a different question: on this evidence the generated file already allows everything, which is what most sites chasing AI visibility want, and the sites that deliberately block are a distinct and smaller population.

If you are on Framer and want to see what an AI crawler receives rather than what the panel says it should, the practical sequence is short. Fetch your own page with an Accept header naming text/markdown and read what comes back. Fetch it again without, and count the prose. Open your robots.txt and see whether it is the four line default. Our Framer fix guide covers the platform specifics, what GPTBot sees renders the crawler side of a single URL, and the methodology page sets out what a Lantad grade counts and what it refuses to guess.

What this post did not measure is worth naming precisely. We read files and response headers on one day, 11 September 2026, from one network location, using the user agent documented on our bot page. We did not observe any AI crawler requesting any of these sites, we read no access log, and no figure here reports what a crawler did. We did not render any page in a browser, so the word counts describe what arrives in the HTML and not what a human eventually sees. The sample is 36 hostnames found by signal detection inside a 392 host frame, not a random sample of Framer sites, and it is small enough that a single unusual site moves a percentage by three points.