Webflow AI crawlers: 20 of 40 robots.txt files held no rule at all

Webflow AI crawlers: what is in the file, and what is not

The sample started as 82 candidate hostnames taken from six public roundups of sites built with Webflow. Three did not respond to a home page request at all. The rest were filtered by response header rather than by anybody's claim: we kept only hostnames whose home page came back carrying x-wf-page-id or x-wf-region, which is the Webflow edge signature, and 52 of the 82 did. Then one GET of /robots.txt per surviving hostname, as LantadBot/1.0 with redirects followed, one attempt each, from one network location on one morning. Forty eight answered HTTP 200 with a plain text body. Four answered HTTP 404, and a 404 on robots.txt allows every crawler while a 503 blocks them all, so those four are open by the same route as the empty ones, for a different reason.

Eight of the 48 were webflow.io subdomains, which is the address Webflow serves a project on before a custom domain is attached. Those are Webflow's hostname rather than a customer's, so they are counted apart from the 40 and described in their own paragraph below. The 40 that remain are what a reader on Webflow is actually asking about when they ask what their site tells an AI crawler.

Across those 40 we searched each file for 39 crawler product tokens, among them GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, CCBot, PerplexityBot, Google-Extended, Applebot-Extended, Amazonbot, Bytespider and Meta-ExternalAgent. Six files named at least one. All six of those named GPTBot, so the same six sites are both the GPTBot population and the AI crawler population: there was no file that addressed some other vendor's bot while ignoring OpenAI's. Thirty four named none. The smallest non-empty file on a custom domain was 38 bytes and held a single Sitemap line; the largest was 1,985 bytes.

The eight webflow.io subdomains split evenly and the split is worth recording. Four served exactly 25 bytes, which is User-agent: * followed by Disallow: /, a full block of the staging address. The other four served zero bytes, the same empty file as the custom domains. So a Webflow project on its pre-launch address is either closed to everything or open to everything depending on which of those two states it is in, and nothing on the site tells the owner which one they have. That matters less than it sounds for AI visibility, because a staging address is not where anybody wants to be cited, and more than it sounds for anybody who has pointed a real audience at a webflow.io link.

What an empty robots.txt actually grants

A file that exists and says nothing feels different from a file that is missing, and to a crawler it is not. RFC 9309, the Robots Exclusion Protocol, published in September 2022, builds a robots.txt out of groups, and a group has to start with a user-agent line. No user-agent line means no group. A crawler looking for rules that apply to it finds no matching group, finds no wildcard group either, and has nothing to obey. The specification is explicit about the missing-file case as well: where the server says the file is unavailable, which covers the 4xx range and therefore a 404, the crawler may access any resource on the server.

So the seven zero-byte files, the 13 sitemap-only files and the four 404s all arrive at the same place from three different directions, which is 24 of the 44 custom-domain hostnames that answered anything at all. We re-requested every zero-byte file a second time to be sure it was not a truncated transfer, and every one returned HTTP 200 with Content-Length zero again, eleven of them counting the staging subdomains.

It is worth being precise about what this does and does not mean, because the temptation on a finding like this is to call it a problem and move on. An open robots.txt is the correct setting for most marketing sites. If you want to appear in ChatGPT or in an AI Overview, the crawler has to be able to fetch the page, and getting cited by ChatGPT starts with not blocking the bot that does the fetching. The finding is not that these sites are misconfigured. The finding is that 20 of 40 have made no decision, and a reader who believes their platform ships a sensible default has believed something about a file with nothing in it.

The second thing it means is narrower and more useful. A site with no groups cannot express a preference later without writing the whole file, and the people most likely to want one are the ones who would also want the distinction between a training crawler and an answering crawler. That distinction only exists if you name tokens, and naming tokens is exactly what nobody in this sample did.

The Sitemap line is the only thing the platform writes

Thirty three of the 40 custom domains carried a Sitemap line, and that number is the one place where the platform's own behaviour shows up in the measurement. Webflow's help centre article on setting robots.txt rules says so in its own words: Webflow adds a link to your sitemap in your robots.txt by default, and there is a toggle to remove it. The page lives at help.webflow.com/hc/en-us/articles/41954080897683-Set-robots-txt-rules, and it is written as plain text here rather than linked because that host is not on this site's registered outbound list. It returned HTTP 403 to our fetch, so we read it through the help centre's own article API, which reports an updated timestamp of 8 September 2026.

The article is a good one and it is not the problem. It explains user-agent and disallow, gives eight worked examples, warns that the file is publicly visible and warns that not all bots obey it. What it does not do is name a single AI crawler. Across 13 User-agent lines in its examples it names exactly three product tokens: Googlebot, UptimeRobot and BadBot. A reader following it will learn how to allow Googlebot and block everything else, which is a 2015 shaped instruction, and will not learn that OpenAI publishes three tokens or that Anthropic publishes three more. That is the same gap our GPTBot detection work found on the vendor side, arriving from the other direction: the names exist, and the documentation a site owner actually reads does not carry them.

One part of the article is newer than the rest and is the reason to read it. Webflow now documents a Content-Signal line, with the example Content-Signal: ai-train=no, search=yes, ai-input=no, and states plainly that this is based on a proposed extension to RFC 9309 and is not yet an accepted standard. That is an honest note and more caution than most vendors print. Four of the 40 files carried such a line, including webflow.com itself, whose robots.txt declares ai-train=yes, search=yes, ai-input=yes. Webflow has opted its own marketing site in to AI training. The mechanism itself is covered in what happened when we probed the Content Signals endpoints.

The article also describes serving Content-Signal as an HTTP response header configured from the same settings screen. We recorded the response headers of all 52 home pages. None of them carried a Content-Signal header. That is a statement about 52 hostnames on one morning and not about the feature, which may well work exactly as documented for anybody who switches it on.

A sitemap reference is a genuinely useful thing for a platform to add, and it is worth saying why it is not a substitute for the rest. A sitemap tells a crawler where your pages are. It does not tell it whether it may have them, and lastmod in a sitemap is an assertion rather than a measurement even for the part it does cover.

The six files that named GPTBot, and who wrote two of them

Six files out of 40 addressed an AI crawler by name, and they do six different things, which is itself the useful observation: there is no house style on a platform that writes nothing. Two of the six are not really the site owner's work at all.

blott.studio and joinpogo.com carry the same file, 1,878 bytes on the first and 1,985 on the second: the Content Signals Policy preamble in full, as a block of comments setting out what search, ai-input, ai-train and use mean, a Content-Signal line reading search=yes,ai-train=no,use=reference, then nine groups each disallowing everything, for Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot, meta-externalagent and CloudflareBrowserRenderingCrawler. Everything joinpogo.com has beyond the other file is a second wildcard group, a Twitterbot group and a repeated Sitemap line. That CloudflareBrowserRenderingCrawler name gives it away. Two unrelated sites did not independently write the same file; a generator did, and we have written before about how Cloudflare will generate your robots.txt and move the token list underneath you. The two most complete AI policies in this sample came from the CDN, not from Webflow and not from the person who owns the domain.

swan.io wrote its own and it is the most deliberate file in the set. Four commented sections allow Googlebot and Google-Extended, then ClaudeBot, Claude-User and Claude-SearchBot, then GPTBot, ChatGPT-User and OAI-SearchBot, and then User-agent: * with Disallow: /, followed by full blocks on six SEO industry crawlers. Read as a decision it says: Google, Anthropic and OpenAI may have the site, and everybody else may not. That includes PerplexityBot, Applebot, Amazonbot and CCBot, which are blocked not by a rule naming them but by being everything else. Whether that is what the author meant, we cannot know. It is what the file says, and it is the sort of thing a robots.txt tester settles in a few seconds per token.

jasper.ai runs the opposite policy at similar effort: 21 User-agent lines grouped into allow blocks for search engines, for Perplexity and Apple and Google's extended token, for Anthropic, for OpenAI's three, and for a long tail including CCBot, cohere-ai, AI2Bot, Diffbot, Bytespider, meta-externalagent and FacebookBot. One name is disallowed, Omgilibot, and the wildcard group blocks a single pricing path. glean.com does something we have not seen in any of the four platform samples this week: it disallows the path /llm-info to Googlebot, Bingbot, Slurp, DuckDuckBot, Baiduspider and YandexBot, and explicitly allows that same path to GPTBot, ChatGPT-User, ClaudeBot, Claude-Web, PerplexityBot, Amazonbot and Bytespider. A page written for models and hidden from search engines, stated in the file rather than inferred. Agree with it or not, somebody thought about it.

A named group replaces the wildcard, it does not add to it

Two of the six careful files contain the same trap, and it is the single most expensive misunderstanding in this whole subject, so it is worth spending a section on. RFC 9309 says a crawler finds the group whose product token matches its own, case insensitively, and obeys the wildcard group only if no explicit match exists. Only if. A named group is not layered on top of the wildcard group; it takes its place entirely. Anything the wildcard blocked and the named group does not repeat becomes allowed for that crawler, and nothing anywhere reports it.

kajabi.com shows it cleanly. Its wildcard group carries Allow: / and eight disallows: /admin/, /api/, /checkout/, /account/, /login, /logout, /password and /cart. Its GPTBot group carries four disallows: /admin/, /api/, /checkout/, /account/. Four paths that every other crawler is asked to leave alone, /login, /logout, /password and /cart, are not mentioned in GPTBot's group, so GPTBot is not asked to leave them alone. The same is true of the anthropic-ai group and the Applebot-Extended group, which carry the identical four. Naming those crawlers made their rules looser than the rules for an anonymous bot that was never named at all. glean.com has the same shape: its wildcard group blocks /internal/ and /new-pages/, and the seven AI groups carry only an Allow line, so none of the seven is bound by either disallow.

This is not a defect in either site and it is certainly not a Webflow problem. It is the standard working as written, and it catches people because the intuitive reading of a file is top to bottom and cumulative, while the actual reading is select one group and apply it. We have watched the same mechanism produce the opposite error elsewhere: a trailing star changed nothing in 168 comparisons because authors expect pattern syntax to do more than it does, and a renamed token leaves a group matching nothing because authors expect a name to keep meaning what it meant. The common thread is that robots.txt gives you no feedback. You can read a file for a minute and be confidently wrong about what it does to one named crawler, which is why the check has to be per token and mechanical. Ours is the robots.txt tester, and what GPTBot sees on your page answers the next question after that one. How a capture is scored here explains what the grade does with the answer.

A dollar sign, a Citation-Policy and a Training-Window

The most instructive file in the sample is 129 bytes long and belongs to thefurrow.tv. In full, it reads: User-agent: $llm, then Allow: /, then Citation-Policy: allow-with-attribution, then Training-Window: 30d, then a blank line and a Sitemap line. Somebody sat down and wrote an AI usage policy into robots.txt. Every part of it is inert.

Start with the token. The specification's grammar for a product token allows uppercase letters, lowercase letters, underscores and hyphens, and nothing else. A dollar sign is not in that set, so $llm is not a valid product token, and a line that is not a valid start-group line does not open a group. Even a lenient parser that accepted the string would then look for a crawler calling itself $llm, and there is no such crawler: it is not a name any vendor publishes, it reads like a variable that was never substituted. This is the same failure we described when ChatGPT's own robots.txt disallowed a token no crawler can match, except that here the intent was to permit rather than to forbid, which makes the consequence milder and the lesson identical.

Then the two invented fields. Citation-Policy and Training-Window are not defined in RFC 9309. The specification does anticipate fields it does not define, and says a crawler may interpret other records and that parsing them must not interfere with the records the standard does define, which is the polite way of saying a conformant parser skips them. There is no registry they could be looked up in and no vendor documentation that gives them meaning. A crawler reading this file learns nothing about attribution and nothing about a thirty day window.

Now the part that actually matters, which is what the file does rather than what it fails to do. Because the only user-agent line in it is invalid, the file contains no group, including no wildcard group. So thefurrow.tv is in exactly the same position as the seven sites that serve zero bytes: every crawler that reads it finds nothing that applies and may take the whole site. The author wanted attribution as a condition of use and ended up publishing an unconditional open door, and nothing in the file, in Webflow, or in any tool that reports on robots.txt syntax would have told them. That is the honest reason a scan is worth running on a file you wrote yourself, and it is why our crawler reference lists the tokens vendors actually publish rather than the ones that would be convenient. If you want the machine-readable summary format instead, llms.txt is a different file with a different and much weaker evidence base, and it is not a substitute for getting the group line right.

The practical version of all of this for a Webflow site is short. Open your own robots.txt in a browser and look at how many bytes come back. If the answer is zero or one Sitemap line, you have not configured anything, whatever the settings screen implied. If you have named an AI crawler token, check each one against the wildcard group and confirm you repeated every disallow you meant to keep. If you have invented a field, delete it. And if you want to know what a request from our scanner looks like when it arrives, the bot page documents the user agent, and the research page holds the aggregate picture we are willing to publish. None of that requires a paid tool, and for the structural half of the problem the per stack fix guides cover the rendering side that robots.txt never touches.