Squarespace AI crawlers: 61 of 66 sites named 26 and blocked none

What is actually inside the robots.txt Squarespace writes for you

The platform file announces itself. Line 1 is a comment reading Squarespace Robots Txt, line 2 is blank, and lines 3 to 32 are 30 consecutive user-agent lines with nothing between them. 26 of those 30 are AI crawler tokens: AI2Bot, Ai2Bot-Dolma, aiHitBot, Amazonbot, anthropic-ai, Applebot-Extended, Bytespider, CCBot, ClaudeBot, cohere-ai, cohere-training-data-crawler, DuckAssistBot, FacebookBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, GPTBot, img2dataset, Meta-ExternalAgent, MyCentralAIScraperBot, omgili, omgilibot, Quora-Bot, TikTokSpider and YouBot. Three more are Google advertising tokens, AdsBot-Google and its two mobile variants. The thirtieth is the wildcard.

The first rule appears at line 33 and the rules run to line 60: 27 Disallow lines and one Allow. They are housekeeping paths, being /config, /search, the account and digital download paths, /api/ with an exception for /api/ui-extensions/, /static/, and a run of query string patterns covering author, tag, month, view, several format values and reversePaginate. 64 of the 66 files that answered carried that exact rule set, character for character, which is what you expect from a file a platform composes rather than a person edits.

Nothing in that rule block mentions AI, training, or any of the 26 tokens above it. There is no Disallow of a path an AI crawler would want and a search crawler would not. The file is the same file a Squarespace site would have had in 2019, with 29 names stacked on the front of it. Compare that with the platform default that WordPress composes in do_robots(), which names no crawler at all, or with the Shopify storefronts we measured the day before, where no two files were identical.

Are Squarespace AI crawlers blocked by the file the platform writes?

Not on the default setting, and the reason is a grammar rule rather than a judgement call. RFC 9309 is the standards track document that defines robots.txt, and it prints the syntax as ABNF. A group is one start group line, then zero or more further start group lines or empty lines, then zero or more rules. The comment the authors wrote beside that production says the rules are relevant for the preceding user-agent lines. Plural. Consecutive user-agent lines with no rule between them do not open 30 separate groups. They open one group with 30 names on it, and every rule underneath applies to all 30 equally.

So when a crawler that matches the token GPTBot reads this file, it finds a group matching its product token, obeys that group, and receives 27 Disallow lines about /config, /search and query string formats. When a crawler matching no token at all reads the same file, it falls through to the wildcard, which is in the same group, and receives the identical 27 lines. Naming GPTBot changed the outcome for GPTBot by nothing. The document is explicit that a crawler goes to the wildcard group only when no group matches its token, which is the mechanism a robots.txt that names the wildcard and not the crawler relies on, and here both routes arrive at the same place.

This is a different failure from the ones we have written about before. A renamed token leaves a stale group matching nothing, and a file can disallow a token no crawler can match. Here the tokens are current, correctly spelled and matched by real crawlers. What is missing is the rule. A name in robots.txt is an address, not an instruction, and an address with no instruction under it delivers nothing. You can check the group structure of any file with the robots.txt tester on this site.

What 66 live Squarespace sites served on 10 September 2026

We started from 72 hostnames whose home page returned a Server header of Squarespace. 66 answered a request for /robots.txt with HTTP 200. Five answered HTTP 404, and under RFC 9309 an unavailable status in the 400 range means a crawler may access any resource on the server, which is why a 404 and a 503 on robots.txt are opposites rather than variations. One hostname did not answer within the timeout.

Of the 66 that answered, 64 carried the platform comment on line 1 and the identical 28 line rule block. Two returned HTTP 200 with an HTML document in the body instead of a robots.txt, which parses to no groups and no rules and therefore closes nothing. That is a quieter version of the same problem: a 200 status makes a crawler treat the response as the site's robots.txt, and the parsing error rule in RFC 9309 tells crawlers to use whatever lines are parseable, which here is none of them.

61 of the 66 had the 26 AI tokens inside the wildcard group. Three carried a Disallow: / somewhere. That leaves a distribution worth stating plainly, because it is the number a Squarespace owner should compare their own site against: on this sample, 3 sites in 66 had asked an AI crawler to stay out, and 63 had not, whether by leaving the default alone or by serving a file with no rules in it. The crawler directory on this site lists which tokens belong to which vendor, and the methodology page records how a capture is taken and scored. What this measurement cannot tell you is anything about compliance, which is a separate question we have looked at where six of ten assistants never requested the file at all.

The single line that separates a name from a block

Two of the 66 files carried the block. Both were the platform file, both listed the same 26 tokens in the same order, and both differed from the other 62 by one line. At line 28 they carry User-agent: YouBot, the last of the 26. At line 29 they carry Disallow: / and nothing else. That one rule closes the group. The 26 AI tokens now sit in a group of their own with a rule that forbids the entire site, and the three advertising tokens and the wildcard open a second group below, which still receives the ordinary 28 line rule block.

Squarespace documents the switch that writes that line. Its help page on requesting that AI models exclude your site, last updated 8 January 2026 and published at support.squarespace.com/hc/en-us/articles/360022347072-Request-that-AI-models-exclude-your-site, says to open the Settings panel, click Crawlers, and check the box labelled Block known artificial intelligence crawlers. It then lists the bots the box covers, and that list is the same 26 names, in the same order, that we parsed out of the served files. The page also answers the question directly: the box is unchecked by default, and the stated reason is that Squarespace does not want to potentially impact your site's traffic by excluding it from chat answers and sources.

The third blocking file in the sample got there another way. It was not the platform shape at all: it opened with the Cloudflare content signals preamble, the block of comment lines that states the search, ai-input and ai-train signals and what each one grants, and it carried nine separate Disallow: / lines across nine groups. Somebody had replaced the generated file rather than ticking the box, which is the only case in 66 where the owner's own text reached the crawler.

That is a defensible default and the page argues for it openly, which is more than most platforms do. The gap is not the policy. The gap is that the unchecked state still writes the 26 names into the file, so the file carries the vocabulary of a block without the rule that makes one. Squarespace's own phrasing for the unchecked state is that it has not added any AI do not crawl requests to your robots.txt, which is exactly right and exactly what the names alone amount to. A reader who scrolls their own robots.txt cannot tell the two states apart without knowing that an edit and its effect are different events and that one missing line is the whole difference.

The crawlers that fetch a page to answer a question are not on the list

The 26 names are worth reading for what is missing from them. We searched all 66 files for six tokens and found zero occurrences of each. OpenAI's crawler documentation publishes four tokens and describes them separately: GPTBot is used to crawl content that may be used in training its foundation models, OAI-SearchBot is used to surface websites in search results in ChatGPT's search features, and ChatGPT-User is used when a user asks ChatGPT a question and it visits a web page. Squarespace's list names GPTBot and neither of the other two.

Anthropic's crawler page documents three: ClaudeBot for collecting web content that could contribute to training, Claude-User for when an individual asks Claude a question and it accesses a website, and Claude-SearchBot for improving search result quality. Squarespace names ClaudeBot, and also anthropic-ai, which is not among the three that page documents. It names neither of the two that fetch a page on a user's behalf, which is the pair we counted when we looked at three Anthropic bots behind one IP list. Perplexity's bot documentation publishes PerplexityBot, described as designed to surface and link websites in search results and explicitly not used to crawl for foundation models, and Perplexity-User for user actions. Neither appears in any of the 66 files.

The one entry on the list that spans both jobs is Google-Extended. Google's common crawlers page states that it is a standalone product token controlling whether crawled content may be used for training Gemini models and for grounding, which it defines as providing content from the Google Search index to the model at prompt time, and that it does not affect a site's inclusion in Google Search. So ticking the Squarespace box does reach Gemini grounding, and does not reach ChatGPT search, Claude answering or Perplexity at all.

Squarespace's help page says there is not currently a universal way to be excluded from training while still being featured in the same company's chatbot answers. There is no universal way, and that sentence is true as written. Three of the vendors on its own list publish a per vendor way, at their own documentation, today. Which of those you want is a policy question and not one a scanner answers, but the choice exists and the checkbox does not expose it. That is the same shape as OAI-SearchBot being named in far fewer files than GPTBot: the training token is the one everybody knows.

What to check on a Squarespace site, in the order that decides the answer

The check is short because the file is short, and it does not need a tool. Request /robots.txt on your own domain and read four things in sequence.

First, the status. A 200 with a robots.txt is the only case where anything you write matters, and two of our 66 returned a 200 with an HTML page, which is indistinguishable from a working file until you look at the body. Second, the line immediately after the last AI user-agent line. If it is another user-agent line, the names above it carry the wildcard's rules and nothing else. If it is Disallow: /, the box is ticked. Third, which tokens are on the list at all, because the answer time crawlers described above are not, so the box cannot express a preference about them either way. Fourth, whether robots.txt is even the layer deciding your outcome, because it is the weaker of the two layers that decide whether AI can read a site and a file that permits a crawler is not a server that serves it, as we found when sites banning GPTBot served it a 200 anyway.

None of this is an argument for ticking the box. Blocking a training crawler is a trade with real costs on both sides, and the honest position is that we measured a file and not an outcome. What we can say is that the default is legible only if you know the grammar: the difference between a Squarespace site that has asked AI crawlers to leave and one that has not is one line in a file most owners will never open, and 61 of the 66 we read were on the side of that line where the names are decoration. If you want the crawler's view of your pages rather than the file's, what GPTBot sees renders it, and AI visibility is the wider measurement this sits inside.