Wix AI crawlers: 0 of 69 robots.txt files named one, and Wix's own fix names BingAI

What is inside the robots.txt Wix writes for you

The default file runs from 488 to 542 bytes across this sample, the difference being the length of the site's own sitemap URL, and it holds five groups. The first is the wildcard: User-agent: *, then Allow: /, then a single Disallow of the query string pattern that opens a gallery lightbox. The second names AdsBot-Google-Mobile and AdsBot-Google on consecutive lines and disallows two internal paths, one of them the gallery web app. The third names PetalBot and disallows the whole site. The fourth and fifth carry Crawl-delay: 10 for dotbot and for AhrefsBot, under a comment describing them as overly enthusiastic bots. Then a Sitemap line, and a closing comment reading that the file is auto generated and that the SEO Tools robots.txt editor is where to change it.

That closing comment appeared in 67 of the 69 files that answered. 62 of the 69 were byte identical to each other once the Sitemap URL was masked, and 66 were identical once the number of Sitemap lines was allowed to vary as well: two sites carried two Sitemap lines, one carried three and one carried four. Files a platform composes look like this. It is the opposite of what we found on the Shopify storefronts measured the day before, where no two files in the sample were the same.

The three files that were not the platform default are worth naming individually, because they are the only evidence in this sample of anybody touching the setting at all. One was an older shaped Wix file of 285 bytes, carrying the same PetalBot block and the same two crawl delays but no auto generated comment and a slightly different wildcard rule set. One was 85 bytes long and its only user-agent line reads User-agent* with no colon, which under the grammar is not a start group line, so the Disallow beneath it belongs to no group.

The third had been edited by hand, and it is the most instructive object in the whole sample. It runs to 6,069 bytes, carries 148 user-agent lines and 138 bare Disallow: / lines, and the names it blocks are EmailSiphon, EmailCollector, CherryPicker, Teleport, WebZip, Xenu's Link Sleuth, Zeus, Wget, larbin and Nutch. That is a well travelled anti scraper blocklist from the middle of the last decade, pasted in whole. Someone on that site cared enough about crawlers to write 138 rules. Not one of the 138 addresses a crawler that any AI company operates today, which is the same failure mode as a renamed token leaving a robots.txt group matching nothing: the file is full of intent aimed at the wrong decade.

Are Wix AI crawlers blocked by the file the platform generates?

No, and the reason is mechanical rather than a matter of interpretation. RFC 9309 is the standards track document that defines robots.txt, published in September 2022, and it settles what happens to a crawler whose name is nowhere in the file: if no group matches the product token, the crawler must obey the group whose user-agent line carries the asterisk. Here that group grants Allow: / and withholds one query string pattern that opens an image lightbox. So a fetch by GPTBot and a fetch by an unnamed scraper receive the same answer, which is yes to everything except a gallery URL nobody wants anyway. That is the same shape we described when a robots.txt named the wildcard and not the AI crawler, arrived at here by a platform default rather than by an owner's choice.

The one token the generated file blocks outright is PetalBot. It appears in 68 of the 69 files, always with Disallow: /, and it is not among the crawlers the AI vendors document: OpenAI publishes GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User, Anthropic publishes ClaudeBot, Claude-User and Claude-SearchBot on a page dated 7 April 2026, and Perplexity publishes PerplexityBot and Perplexity-User. None of those nine strings occurs anywhere in any of the 69 files. The strongest instruction in the file a Wix site ships with is pointed at a crawler that belongs to none of them.

The two crawl delay groups are worth one line because they are not rules in the specification's sense at all. The string Crawl-delay does not appear anywhere in RFC 9309, so a directive 68 of these files carry is one no standards document defines, which is the same gap that lets a crawl delay line contribute to unverifying a crawler rather than slowing it down. Whatever dotbot and AhrefsBot do with those lines, they do it by private agreement.

Two hostnames answered the robots.txt request with HTTP 404, and both were served by the Wix edge on the same morning. Under the specification an unavailable status permits a crawler to access any resource on the server, which is the outcome we set out in full when we measured that a 404 on robots.txt allows every crawler while a 503 blocks them all. So those two sites are not an exception to the finding. They are the most permissive files in the sample, by being absent.

The four tokens Wix tells you to paste, and the one that does not exist

Wix does have an answer for owners who want AI crawlers out, and it is a help centre article called Blocking AI Crawlers from Your Site, published at support.wix.com/en/article/blocking-ai-crawlers-from-your-site and read there on 10 September 2026. There is no toggle. The article routes you to SEO and GEO in the dashboard, then Tools and settings, then the Robots.txt Editor, and gives you a block of text to type in. That block names four user agents, each with Disallow: / underneath it: CCBot, GPTBot, ChatGPT-User and BingAI. A tip below it says the list includes some known AI crawlers and that you can add more as needed.

Three of those four are real. The fourth is not a crawler anyone publishes. OpenAI's bots documentation names four tokens and BingAI is not among them. Microsoft's own announcement of controls for content in Bing Chat, published on 22 September 2023, tells webmasters to use NOCACHE and NOARCHIVE meta tags, not a robots.txt user agent, which is the finding we set out at length in Microsoft's AI opt out is a meta tag, not a robots.txt token. A group headed User-agent: BingAI matches no crawler, so it is four lines of a file doing nothing. It is not harmful. It is just the part of the instruction that cannot work, sitting in the same block as the three parts that can.

What the list leaves out matters more than what it gets wrong. There is no ClaudeBot, so Anthropic's training crawler is untouched. There is no PerplexityBot, no Claude-User, no Perplexity-User and no OAI-SearchBot, which is the token that decides whether a page can surface in ChatGPT's search results and which we found named in only 16 of 140 robots.txt files when we last counted. There is no Google-Extended either, and Google's own crawler documentation, last updated 14 July 2026, describes that token as the way to manage whether crawled content is used for training and grounding Gemini models while explicitly not affecting inclusion in Google Search. An owner who follows the article exactly ends up blocking one training crawler and Common Crawl, and leaving the rest.

There is one more thing in the block worth reading closely. ChatGPT-User is not a training crawler. OpenAI's documentation describes it as the agent that visits a page when a user asks ChatGPT a question, so disallowing it does not protect anything from a training corpus. It closes the path by which a person's question reaches your page. The article's stated purpose is preventing content being used to train large language models without permission, and one of its four tokens spends that intention on an answering fetch instead. Wix's own FAQ at the bottom of the article says something similar in its own words: blocking AI crawlers in robots.txt may limit NLWeb functionality, because agents need access to view structured content, and NLWeb is a server you run rather than a file a crawler finds.

The second route in the same article, and what 71 home pages carried

The Wix article offers a second method after the robots.txt one, and it is the better advice of the two for anyone worried about Google specifically. It tells you to open a page's SEO panel in the editor, go to the Advanced SEO tab, open the robots meta tag control and select the nosnippet checkbox, which it describes as preventing the page's content being used as a direct input for AI overviews. That is a real mechanism with a real effect, and it is the one route on the whole page that touches Google's answer surfaces at all, since Google-Extended is missing from the block above and Googlebot is what feeds AI Overviews.

So we looked. Across the same 71 Wix served hostnames, fetched on 10 September 2026, we counted robots meta elements in the home page HTML and X-Robots-Tag values in the response headers. The string nosnippet occurs zero times. So does max-snippet. Not one of the 71 responses carried an X-Robots-Tag header of any kind. Three home pages carried a robots meta element, and all three said noindex, which is a different instruction with a different consequence and is not what the article was recommending. Two of those three are the same two hostnames whose robots.txt answered 404, which reads like two sites that were built and never finished rather than two deliberate choices.

That is the same shape as the wider count we published four days earlier, when 0 of 81 home pages blocked their own snippet across a much broader frame, and it is consistent with what we found when we went looking for the header route and discovered that 5 of 391 home pages sent an X-Robots-Tag and all five were a captcha. Two vendor documented routes for keeping content out of AI answers, offered to millions of site owners on the same help page, and in this sample of 71 sites neither had been used once.

We should be careful about what that does and does not show. It is not evidence that Wix owners want AI crawlers blocked and failed, and it is not evidence that they want them allowed. It is evidence that the default shipped, and stayed. On a platform whose whole proposition is that you do not have to think about the plumbing, a default that grants full access to every AI crawler is the setting almost every site will be running, and the article that changes it is four clicks deep behind a dashboard section most owners open once.

What to check on your own Wix site

None of this requires a tool, and the first two steps take a minute. Open your own domain with /robots.txt on the end and read what comes back. If the last line says the file is auto generated and points at the SEO Tools editor, you are running the default this post measured, and the practical meaning of that is that every AI crawler is allowed everything. Then decide, deliberately, whether that is what you want, because for a good many businesses it is: being readable is how a page gets quoted, and blocking a search or answering crawler removes the page from answers without removing it from a training corpus that was collected years ago.

If you do want to change it, change it with the current token list rather than the one in the help article. The tokens that matter are published by the vendors themselves and we keep the current set in one table on the AI crawler reference, which lists every token in our registry with the operator behind it and what it is for, and the robots.txt tester answers the question a file cannot answer by eye, which is which AI crawlers your own robots.txt allows, crawler by crawler rather than in the aggregate. Blocking is not the only choice on offer either: a training token, a search token and a user fetch token can each be answered differently, and treating them as one switch is how sites end up invisible in ChatGPT while their old pages remain in a corpus nobody can recall.

The last thing to say is the limit of the measurement itself, because it is the part a reader should hold onto. We read files. We asked each hostname once, on one day, from one network location, with redirects followed and a single attempt. We did not observe a single AI crawler requesting any of these sites, we read no access log, and we did not fetch any page with a crawler user agent to see whether the answer changed. Nothing here establishes whether any crawler obeyed, ignored or ever saw these files, and what we do and do not measure is written down for exactly this reason. The frame is skewed too: showcase roundups favour retail, hospitality and creative portfolios, so treat the shape of these files as a description of the platform default, which is what it is, and not as a census of Wix.