Bytespider robots.txt: 130 of 1,077 files named it, and 70 shut it out at the root

What does a Bytespider robots.txt rule actually say? 130 files answered

130 of the 1,077 parsed files named Bytespider in a user-agent line. Of those, 70 ended up disallowing the root path for that token once the most specific matching rule was applied, 54 carried rules that blocked some paths and left the root reachable, and 6 named the crawler and then disallowed nothing at all. The six are bindwell.ai, youshift.com, societies.io, mediamarkt.de, visitdubai.com and geico.com, and in each case the group holds an explicit Allow rule rather than an empty one, which is a decision to let the crawler through rather than an oversight.

The 70 that close the door are mostly recognisable names: congress.gov, illinois.gov, britishmuseum.org, unesco.org, nature.com, sciencedirect.com, jamanetwork.com, drugs.com, healthline.com, aljazeera.com, abc.net.au, axios.com, bloomberg.com, cnbc.com, cnn.com and bostonglobe.com among them. The shape of the rule varies more than the verdict does. 62 of the 130 gave Bytespider a group of its own, and 68 put it in a group shared with other tokens. nature.com runs 23 groups and spends one of them on Bytespider alone. congress.gov runs two groups and puts Bytespider in a single block of 128 user-agent tokens followed by one Disallow line, which is the other way to write the same answer.

This matters for anyone reading their own file, because RFC 9309 says a crawler "MUST use case-insensitive matching to find the group that matches the product token and then obey the rules of the group", and that where several groups match, the "matching groups' rules MUST be combined into one group". A token in a list of 128 is as binding as a token with a group to itself. What is not binding is a token nobody sends, which is the subject of a later section.

Bytespider is named less often than GPTBot and refused more often

GPTBot is the more frequently named token: 175 files against 130. But naming is not refusing, and on the share of its own namers that shut it out at the root, Bytespider sits at the top of the set alongside CCBot. 70 of 130 is 53.8 percent. The same figure for GPTBot is 45.1 percent, for Meta-ExternalAgent 46.7 percent, for Amazonbot 45.9 percent, for Applebot-Extended 46.5 percent and for Google-Extended 38.8 percent. Common Crawl's CCBot matches Bytespider exactly at 77 of 143.

The direct comparison is cleaner still inside the 121 files that name both tokens. There, Bytespider is disallowed at the root in 61 files and GPTBot in 57. 57 of those overlap, so there are 4 files that refuse Bytespider and allow GPTBot at the root, and none that do the reverse. On this corpus, no site owner decided ByteDance should have more access than OpenAI.

Two cautions about reading the headline share too hard. The denominators are small, and the gap between 53.8 and 45.1 percent is 130 and 175 files rather than a population. And this corpus is a deliberately built sample, not a slice of the web: it is stratified by platform and industry to make crawlability comparable across site types, which over-represents large publishers and institutions. Lantad has already published the web scale figure for comparison, where the 2025 Web Almanac, written from HTTP Archive's July 2025 crawl of 16,213,084 websites, found gptbot in 4.5 percent of desktop robots.txt files. 175 of 1,077 here is 16.2 percent. The corpus is not the web, and the right reading of these shares is relative, not absolute.

110 of 121 files gave Bytespider and GPTBot identical rules

The interesting part of the comparison is how little of it is deliberate. 121 of the 130 files naming Bytespider also named GPTBot. In 110 of those 121, the two tokens received rule for rule identical groups: the same patterns, the same allows, the same disallows. That is 90.9 percent of the files that name both, and it is the signature of a pasted blocklist rather than a position on either vendor. The same effect shows up whenever this corpus is asked about a specific token, and it is why a census of crawler tokens across 1,004 files found 2,209 distinct tokens with a long tail nobody could have chosen individually.

Only 11 files gave the two crawlers different rules, and the difference ran in one direction. In 8 of the 11, GPTBot's group held more Allow rules than Bytespider's. In none of the 11 did Bytespider's hold more. github.com disallows Bytespider at the root and gives GPTBot a list of allowed paths including /about, /pricing, /features and /marketplace. healthline.com disallows Bytespider at the root and carves out /brand/ and a single drug page for GPTBot. bloomberg.com, thehill.com and canva.com all disallow both at the root and then re-open specific paths for GPTBot only.

That is the whole of the deliberate differentiation in this corpus: eleven files, eight of them letting OpenAI into somewhere ByteDance cannot go. It is a small number, and it is worth more than the large one, because the 110 identical files tell you about tooling while the 11 tell you about intent. If you are deciding what your own file should say, the useful question is which of those two groups you are in, and the robots.txt tester answers it per token rather than per file.

News sites did nearly all of the blocking, and platform sites did none

The corpus is stratified, which makes it possible to ask who is doing the refusing. The answer is concentrated to a degree that the aggregate hides. 37 of the 67 news hosts disallowed Bytespider at the root. That is 55.2 percent of one stratum, and 37 of the 70 root blocks in the entire corpus. Education contributed 7 of 105, SaaS 5 of 121, ecommerce 5 of 84, healthcare 4 of 91, government 3 of 82 and small business WordPress sites 3 of 35. Finance managed 1 of 94 and travel 1 of 80.

Six strata contributed nothing at all: zero of 29 Framer sites, zero of 38 Webflow sites, zero of 54 Wix and Squarespace sites, zero of 39 no code sites, zero of 34 Shopify storefronts and zero of 20 static documentation sites. These are the platform strata, where robots.txt is usually whatever the platform generates, and the platform does not generate an opinion about ByteDance. A Shopify storefront owner who wants one has to write it, which is the sort of thing the per stack guides exist for.

This is the same concentration Lantad found when it asked a broader version of the question and reported that 42 of the 77 files blocking a citation crawler were news sites. It is consistent, and the reason is not mysterious: news publishers have legal departments, licensing negotiations and a direct commercial interest in who reproduces their text. It does mean that any statement of the form "N percent of sites block AI crawlers" is mostly a statement about publishers, and reporting it as a fact about the web is the error. Two of the 130 files went further than a path rule and set a Crawl-delay inside a Bytespider group: rivm.nl at 60 seconds and canonical.com at 2. That is the same rarity Lantad measured when it found 11 of 152 Crawl-delay settings aimed at an AI crawler.

Cloudflare's new training control covers seven vendors, and ByteDance is not one

This is reporting rather than measurement, so it is worth being precise about whose claim is whose. Cloudflare's post of 15 September 2026 introduces a setting it calls Disallow AI Training, and explains that it "is named for the Disallow: directive it publishes in your robots.txt". The post says Apple, Google and Microsoft "honor or have committed (in a specified time frame) to honor this setting", notes that Microsoft's robots.txt support for training is "targeted for early 2027", and adds that Cloudflare "also categorize the relevant crawlers from Amazon, Anthropic, Meta, and OpenAI as Accountable". It states that "17% of sites choose to enable some mechanism to block training". The same post announces that Managed Robots.txt "will be deprecated in favor of Bot Preference Sync". Lantad has measured none of that and is quoting it.

What Lantad can add is the shape of the gap. The seven vendors named in that post are the seven whose robots-level training tokens this corpus already carries: Applebot-Extended, Google-Extended, GPTBot, ClaudeBot and anthropic-ai, Amazonbot and Meta-ExternalAgent, each documented by its own vendor at Amazon, Anthropic and Google respectively. Bytespider is the most refused token in the set and sits outside the scheme. ByteDance's own crawler documentation is at zhanzhang.toutiao.com/docs/intro/26899, which is not a host this site links.

The practical consequence is narrow and worth stating plainly. If a managed control writes your training rules for you, it writes them for the vendors in the scheme, and a token outside the scheme keeps whatever your file already said about it, which for 947 of these 1,077 files is nothing. Whether that matters depends on something nobody in this post has measured: whether Bytespider obeys the rule when it finds one. A robots.txt rule is a request, and Lantad has written before about why a declared user agent is a claim rather than an identity and why only 2 of 9 AI crawler vendors publish the key directory that would let a server verify one.

Thirteen spellings of one crawler, and the ones that bind to nothing

133 files named something from the ByteDance family, and they spelled it thirteen different ways. Counting distinct files, Bytespider appeared in 128, TikTokSpider in 48, lowercase bytespider in 2, ToutiaoSpider in 2, ByteDanceSpider in 2, Doubaobot in 2, and then one file each for DoubaoBot, ByteDance, ByteSpider, Bytespider-Image, ByteDance-AI, TikTokBot and a user-agent line reading "ByteSpider Desktop".

Case is not the problem. RFC 9309 requires case-insensitive matching on the product token, so bytespider and ByteSpider bind exactly as Bytespider does. The problem is everything else. The specification says the "product token SHOULD be a substring of the identification string that the crawler sends to the service", which means a group binds only if some crawler actually sends that token. This repository's bot registry in core/src/bots.ts records one ByteDance token, Bytespider, with the identification string the vendor publishes for it. A group headed ByteDanceSpider or ByteDance-AI is a rule written against a token that registry does not carry, and a rule no crawler matches has no effect on anything. The line reading "ByteSpider Desktop" cannot match Bytespider at all, because the group name is not the token.

TikTokSpider is the clearest case of a spelling behaving differently from its neighbour. All 48 files naming it also named Bytespider, and 41 put the two in the same group. Yet only 7 of the 48 disallowed TikTokSpider at the root, and in those same 48 files Bytespider was root-disallowed only 8 times. The 48 are overwhelmingly small business sites carrying a generic blocklist that disallows a handful of administrative paths, so the token is present and the rule is harmless. Naming a crawler is not blocking it, which is the distinction every one of these censuses keeps running into, from DuckAssistBot in 63 files to Reflectionbot in one. A file that names thirteen variants of one crawler and binds two of them has done less than it looks like it has done, and that is exactly what a token by token check is for.

What the 947 files that never named Bytespider leave it

The 130 are the minority. 947 of the 1,077 parsed files never mentioned Bytespider, and RFC 9309 is explicit about what happens next: "If no matching group exists, crawlers MUST obey the group with a user-agent line with the "*" value, if present." So the wildcard group is the real policy for 88 percent of these sites, and it is worth knowing what it says.

Measured on 12 October 2026, 8 of the 947 had a wildcard group that disallowed the root, 693 had one that disallowed some paths and left the root reachable, 193 had a wildcard group that disallowed nothing, and 53 had no group that applied to Bytespider at all, which under the specification means no rules apply and everything is permitted. Nine of the 1,077 files were empty. So for roughly 936 of these 947 sites, ByteDance's crawler is allowed to read the home page, and in almost every case that is not a decision anyone made. It is the default surviving.

That is the useful finding for a reader who came here to decide what their own file should say, and it cuts both ways. If you want Bytespider out, the wildcard will not do it for you and no managed setting currently covers it, so the token has to be named. If you do not mind it, nothing needs doing. What you should not do is assume the answer, which is the general argument behind the trailing wildcard measurement and the finding that 52 of 54 sites refusing Google-Extended still allow Googlebot: the rules people believe are in their robots.txt and the rules that are in it diverge constantly. Splitting a vendor's training crawler from its answering crawler is a separate decision again, which Lantad measured in the 38 files that refuse GPTBot and admit ChatGPT-User, and which matters more for getting cited by ChatGPT than any training rule does.

What this post did not measure, and should not be read as measuring: whether Bytespider obeys any of these rules, how often it requests anything, or what ByteDance does with what it reads. Every figure here describes what 1,077 files said on one day, read once, from one network location, with no JavaScript executed. Compliance is a server log question, not a robots.txt question, and this corpus holds no server logs.