BlogFindings
AI2Bot robots.txt: 68 of 1,022 files named it, and 6 refused only the token Ai2 does not document
Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 8 November 2026, every request sent as LantadBot/1.0 with redirects followed, and parsed the 1,022 files that resolved into at least one user-agent group. 68 named AI2Bot, the training crawler the Allen Institute for AI runs, and 32 refused it at the site root. Ai2's own crawling notice offers one control for that crawler, a User-Agent string to filter on, and does not mention robots.txt anywhere on the page.
Ai2, the Allen Institute for AI, publishes a page it calls a crawling notice. It names the crawler, says the content it collects is used to train open language models, and gives the User-Agent header the crawler sends. Then it offers one control, which is to filter or reject that traffic at the server. Read at source on 8 November 2026, the page does not contain the word robots.txt. This run measured what 1,022 real robots.txt files say about AI2Bot anyway, and the answer splits three ways: a small deliberate block, a much larger group where the token appears inside a website builder's default file and changes nothing, and six files that wrote a rule for a second token Ai2 does not document while leaving the documented one open.
In short
- An AI2Bot robots.txt rule is a control 68 of 1,022 corpus robots.txt files wrote anyway. Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 8 November 2026 and parsed the 1,022 that resolved into at least one user-agent group. 68 named the AI2Bot token and 32 refused it at the site root.
- Ai2's crawling notice at allenai.org/crawler, read at source on 8 November 2026, publishes exactly one control for the crawler: the header Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler), which the page says can be used to filter or reject traffic from our crawler if desired. The page does not mention robots.txt, publishes no IP ranges and carries no date.
- 44 of the 68 files naming AI2Bot left it allowed at the site root on 8 November 2026, and 39 of the 68 carried one identical list of 30 tokens under a first line reading Squarespace Robots Txt, in which the token shares its group with the asterisk. A naming count for this token is mostly a count of one website builder's default file.
- 24 of the 1,022 files refused AI2Bot in a group that does not include the wildcard, carrying a site-root Disallow, on 8 November 2026, and 12 of those 24 were news sites. A further 8 files refused it without ever naming it, by a wildcard group that disallowed everything.
- 6 of the 1,022 files wrote a rule for Ai2Bot-Dolma and never named AI2Bot: bostonglobe.com, derstandard.at, dr.dk, rappler.com, sfchronicle.com and statnews.com, read on 8 November 2026. All 6 refused Ai2Bot-Dolma at the root and left AI2Bot allowed there, and Ai2's crawling notice names AI2Bot and not Ai2Bot-Dolma.
What Ai2 publishes about AI2Bot, and what it leaves out
The crawling notice is short. Read at source on 8 November 2026 it runs to a heading reading AI2 Bot, one sentence of purpose, and one technical detail. The purpose sentence is plainer than most: the AI2 Bot explores certain domains to find web content, and this web content is used to train open language models. The technical detail is the header, given as Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler), followed by the only instruction the page gives a site owner, which is that this user agent string can be used to filter or reject traffic from our crawler if desired. There is no robots.txt token, no list of addresses to verify the crawler against, no crawl rate, and no date on the page. The notice lives at allenai.org/crawler, which is not a host this site links, so that URL is written here as plain text for a reader who wants to check it.
The token sites are writing is therefore an inference rather than a published instruction, and it is a reasonable one. RFC 9309, the Robots Exclusion Protocol, says in section 2.2.1 that a crawler's product token SHOULD be a substring of the identification string that the crawler sends to the service, and its worked example pairs the header Mozilla/5.0 (compatible; ExampleBot/0.1; https://www.example.com/bot.html) with the robots.txt line user-agent: ExampleBot. Ai2's header is that exact shape, so AI2Bot is the token the convention produces. What the convention cannot produce is a commitment to read the file, and that is the part only the operator can publish.
Three other operators whose crawlers this scanner tracks publish it. OpenAI's crawler documentation, read at source the same day, names four crawlers and recommends allowing OAI-SearchBot in your site's robots.txt file, while saying that disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models. Anthropic's crawler article states that Anthropic's Bots respect do not crawl signals by honoring industry standard directives in robots.txt, and points at a JSON list of its crawler addresses. Perplexity's bot documentation recommends allowing PerplexityBot in robots.txt and publishes two address files. None of those three carries a visible last-updated date either, which is its own small problem, but all three answer the question Ai2's page leaves open. The gap matters because the whole premise of a robots.txt rule for an AI crawler is a voluntary promise, and an unstated promise cannot be relied on or broken.
| Operator | Tokens named | Says it reads robots.txt | Publishes addresses | Date on page |
|---|---|---|---|---|
| OpenAI | 4 | Yes | Yes | None shown |
| Anthropic | 3 | Yes | Yes | Updated this week |
| Perplexity | 2 | Yes | Yes | None shown |
| Ai2 | 0 | No mention | No | None shown |
What does an AI2Bot robots.txt rule do, and how many sites wrote one?
Lantad asked each of the 1,419 hostnames in this repository's two committed corpus seed files for its robots.txt once on 8 November 2026, as LantadBot/1.0 with redirects followed and a twenty second timeout. 1,108 answered with a 2xx status, 1,100 of them a plain 200. 40 of those 1,108 responses were HTML documents rather than robots.txt files, which is a failure mode worth its own measurement and has had one. A further 46 parsed into no user-agent group at all. That leaves 1,022 files holding at least one group, and every count in this post is against those 1,022.
68 of the 1,022 named AI2Bot. 61 named Ai2Bot-Dolma. 948 named neither. Resolving the site root for the token with the matcher in core/src/robots.ts returns a refusal on 32 of the 1,022, and those 32 split in a way that matters more than the total: 24 named the token in a group that does not include the wildcard, carrying a site-root Disallow, and 8 never named it at all and were caught by a wildcard group that disallowed everything. The second kind is the ordinary way a crawler is refused, and this corpus has produced the same shape before, where 559 of the 581 pages GPTBot lost were closed by a rule that never named it.
For scale, AI2Bot ranks 26th by naming count among the 2,197 distinct tokens these 1,022 files name between them, a census this corpus has run before at a similar size. It sits well below GPTBot at 187 and ClaudeBot at 179, and a little below CCBot at 154, which is the comparison that fits best: CCBot and AI2Bot are both crawlers that collect text for datasets rather than crawlers that put a link in an answer. 67 of the 68 files naming AI2Bot also named CCBot. Only airbnb.com named AI2Bot without it, in a file naming 18 tokens in all. That near-total overlap is the first sign that this token is arriving in lists rather than in decisions, which is what the next section measures directly.
GET /robots.txt, 1,419 hosts, 8 November 2026
- User-Agent: LantadBot/1.0 (+https://lantad.co/bot) redirects followed, 20s timeout, one request per host
- 2xx response 1,108 hosts
- of which an HTML document, not a robots.txt file 40
- of which no user-agent group parsed 46
- of which at least one user-agent group parsed 1,022
- non-2xx or request error 311 hosts: 162 sent 403, 56 sent 503, 54 sent 404
Why 44 of the 68 namings changed nothing
44 of the 68 files naming AI2Bot left it allowed at the site root. The explanation is one file, repeated. 39 of the 68 carried an identical list of 30 user-agent lines whose first line is a comment reading Squarespace Robots Txt, and the list runs AI2Bot, Ai2Bot-Dolma, aiHitBot, Amazonbot, anthropic-ai, Applebot-Extended, Bytespider, CCBot, ClaudeBot, cohere-ai and twenty more, ending with the asterisk. All 30 tokens head one group, and the rules under it are path rules: Disallow /config, Disallow /search, Disallow /account, a set of query-string patterns, and nothing touching the site root. So a crawler reading that file finds its own name, reads the group, and is permitted everywhere a general crawler is permitted. The name is in the file and expresses no preference about AI at all.
That is not a defect in the builder's file, which is doing something sensible, and it is not a trick. It is a reason to distrust any headline of the form a share of sites now name crawler X, because a naming count mixes two different things: a publisher's decision, and a default shipped to every site on a platform. This corpus has measured the same distortion in the other direction, where a token sat inside the wildcard's own group and between a quarter and a third of several tokens' naming counts turned out to be platform defaults. 35 of the 44 permissive namings here sit in the corpus stratum built from Wix and Squarespace sites, which is also a warning about this corpus rather than about the web: that stratum exists by design, so it is over-represented relative to the real population.
Three of the remaining namings are the opposite of a block. blindpay.com, postman.com and sonarsource.com each name AI2Bot in a group whose first rule is Allow with the site root, blindpay.com alongside a Content-Signal line and 27 other AI tokens. Two more, visitdubai.com and westwing.de, name the token in a group carrying no rule that matches the site root at all, so the file mentions the crawler and then says nothing about it. Reading a rule as a decision requires resolving it, which is the difference between grepping a file and evaluating it, and it is why this corpus reports verdicts rather than mentions.
A decision: 24 of the 68
- User-agent: AI2Bot
- Disallow: /
- The group does not include the wildcard
- 12 of the 24 are news sites
A default: 39 of the 68
- # Squarespace Robots Txt
- User-agent: AI2Bot ... and 29 more, ending User-agent: *
- Disallow: /config, /search, /account, query patterns
- No rule touches the site root, so the verdict is allowed
The six files that refused Ai2Bot-Dolma and allowed AI2Bot
61 files named Ai2Bot-Dolma, and 6 of them named it without naming AI2Bot: bostonglobe.com, derstandard.at, dr.dk, rappler.com, sfchronicle.com and statnews.com. All 6 refuse Ai2Bot-Dolma at the site root and leave AI2Bot allowed there: on five the token falls through to a wildcard group that permits the root, and on bostonglobe.com no group matches it at all, which RFC 9309 reads as no restriction. On bostonglobe.com the rule sits in a block the file itself labels AI Bots, between a group for AmazonAdBot and one for Amazonbot, and there is no AI2Bot group anywhere in the file. Five of the six were re-read directly at source after the sweep and the rule was still there. rappler.com answered 403 to two later requests, so its file could only be read once, which is a reminder that a single read of a bot-defended host is a single read.
Whether that gap is a gap depends on the matcher, so here is what this one does. The implementation in core/src/robots.ts selects the longest group token that the crawler's own token begins with, so a group headed ai2bot binds a crawler calling itself ai2bot-dolma, and the reverse is not true: a group headed ai2bot-dolma does not bind AI2Bot, because AI2Bot does not begin with it. That asymmetry is the whole arithmetic of the two counts. 38 files refuse Ai2Bot-Dolma at the root, which is the 32 that refuse AI2Bot plus exactly these 6. Not one file refuses AI2Bot while allowing Ai2Bot-Dolma.
The specifications leave room for that reading rather than settling it. RFC 9309 requires case-insensitive matching to find the group that matches the product token, and says that where more than one group matches the same product token the groups must be merged, but it does not define what a crawler should do about a longer token that contains a shorter one. Google's robots.txt documentation is more explicit about the selection rule, stating that only one group is valid for a particular crawler and that its crawlers use the group with the most specific user agent that matches. Under that rule a crawler named Ai2Bot-Dolma would read an AI2Bot group when no better match exists, which is what this matcher does, and a crawler named AI2Bot has nothing to read in these six files. Token matching is quietly full of these edges, as the 84 mis-cased tokens that all bound anyway in an earlier run showed.
Flow: File names Ai2Bot-Dolma only to Crawler token ai2bot-dolma; File names Ai2Bot-Dolma only to Crawler token ai2bot; Crawler token ai2bot-dolma to Exact group match; Crawler token ai2bot (prefix fails) to No group names the token; Exact group match (6 files) to Refused at root; No group names the token (6 files) to Allowed at root.
What this run did not measure
It did not observe AI2Bot. Nothing here is a measurement of the crawler's behaviour, because this scanner reads sites rather than server logs, so no figure in this post says whether AI2Bot requests these files, parses them, or acts on them. That is the honest state of the question: Ai2 publishes no statement to test, and this run produced no observation to test it against. A user agent is a claim rather than an identity in any case, and when a vendor publishes no address list there is nothing to check a claimed identity against, which is a harder problem than it sounds even for crawlers that do publish one: 0 of 2,344 addresses OpenAI publishes answered a reverse DNS lookup in an earlier run.
The sample is a sampling frame, not the web. The 1,419 hostnames are the two committed seed files behind the AI Visibility Index study, stratified deliberately by platform and by industry, so the wix-squarespace stratum and the news stratum are both over-weighted against the real population. Every count here is a count of these named sites on one date, read once. 311 hosts gave nothing to read: 162 refused the file with a 403, 56 returned a 503 and 54 a 404, and those two are opposite instructions under the specification rather than two flavours of missing, which this corpus has measured and written up separately. A host that refuses the file to this scanner may well serve it to another client, so the 1,022 is a floor on how many of these sites have a usable robots.txt, not a census.
Finally, the finding about the Squarespace default is a finding about a count, not about those 39 sites. None of them did anything wrong, and nothing here suggests any of them wanted AI2Bot blocked. The point is narrower and applies to every token this corpus reports on, including the ones measured in recent runs: a number for how many files name a crawler is not a number for how many publishers decided anything, and the two are different by about a factor of two for this token. One other thing a file cannot tell you is whether a site that permits a crawler can actually be read by it, which is a separate measurement and the one the scanner itself exists to make.
-
Named and refused24 files The token sits in a group without the wildcard, carrying a site-root Disallow. 12 of the 24 are news sites. -
Refused without being named8 files A wildcard group disallowing everything caught it, so the file never mentions the crawler. -
Named and allowed44 files 39 of them carry one website builder's 30-token default, where the token shares its group with the asterisk. -
Not named, allowed946 files No rule reached the token, which RFC 9309 reads as no restriction at all.
Lantad
Published .
The sequence is familiar to anyone who reads their own access logs. An unfamiliar name appears, in this case AI2Bot, you look up what it is, and you write a line in robots.txt to say yes or no. The useful question is whether the line does anything, and that depends on something the person writing it rarely checks: whether the organisation running the crawler has ever said it reads the file.
Common questions
What should my AI2Bot robots.txt rule be?
That is a decision this post cannot make for you, but two facts narrow it. First, Ai2's crawling notice, read at source on 8 November 2026, offers only a User-Agent string to filter on and does not mention robots.txt, so a robots.txt rule for AI2Bot rests on no published commitment to read it. Second, the token the RFC 9309 convention produces from Ai2's published header is AI2Bot, and a group headed AI2Bot also binds a crawler calling itself Ai2Bot-Dolma under a most-specific-match reading, while a group naming only Ai2Bot-Dolma does not bind AI2Bot. 6 of the 1,022 corpus files measured were in that second state.
Does AI2Bot respect robots.txt?
No source read for this post answers that, which is the finding rather than an evasion. Ai2's crawling notice does not mention robots.txt at all, and no Ai2 statement about robots.txt compliance was found anywhere this run looked. Third-party crawler directories publish verdicts on it, none of which is an Ai2 statement, so none is repeated here as fact. By contrast OpenAI, Anthropic and Perplexity each state in their own documentation that their crawlers read the file. Lantad measured the files, not the crawler, so nothing here observes what AI2Bot does with them.
How many sites block AI2Bot?
Of the 1,022 robots.txt files Lantad read from this repository's corpus seed files on 8 November 2026 that parsed into at least one user-agent group, 32 refused AI2Bot at the site root. 24 of the 32 named it in a group that does not include the wildcard, carrying a site-root Disallow, and 8 refused it through a wildcard group without naming it. 68 files named the token in total, so most namings are not refusals. The corpus is a stratified sampling frame rather than a random sample of the web, so treat this as a measurement of 1,022 named sites on one date.
What is Ai2Bot-Dolma, and is it the same crawler?
Ai2's crawling notice names one crawler and one header, AI2Bot, and does not name Ai2Bot-Dolma, so this post makes no claim about what the second token is or whether Ai2 sends it. What is measurable is that both tokens appear widely in real files: 68 of 1,022 corpus files named AI2Bot and 61 named Ai2Bot-Dolma on 8 November 2026, 55 named both, and 6 named only Ai2Bot-Dolma. The 30-token default file that accounts for most of the naming lists them one after the other on its first two user-agent lines.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
AI visibility
AI visibility is whether AI systems can access, read, and faithfully represent your web pages, the prerequisite for being cited in an AI answer.
CCBot robots.txt: 140 of 1,013 files named it, and 15 that blocked GPTBot left it allowed
Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 17 October 2026, as LantadBot with no JavaScript executed. 1,013 parsed into at least one user-agent group. 140 named CCBot and 81 disallowed it at the root. 15 files that closed the root to GPTBot left CCBot allowed, and not one of those 15 named CCBot anywhere in the file.
AI crawler list: 201 tokens on one blocklist, and 7 of the 9 files naming 50 or more refused every citation crawler
A widely shared community blocklist, read at source on 6 November 2026, carries 204 User-agent lines resolving to 201 distinct tokens under the case-insensitive matching RFC 9309 requires, followed by a single Disallow rule for the whole site. Its own README says it holds AI related crawlers of all types, regardless of purpose, and that is literal: every one of the 15 tokens in this scanner's registry is on it, including the three whose vendors document them as the crawlers that surface and link a site in AI answers. Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 6 November 2026 and parsed the 1,031 files that resolved into at least one user-agent group. 753 named none of the 201. Nine named 50 or more, and 7 of those 9 refused all three citation crawlers at the site root.
Crawler user agent: 1,004 robots.txt files named 2,209 tokens, and 120 name one Anthropic no longer documents
A user-agent line in robots.txt is an exact product token, and a token that matches nothing is a rule that does nothing. Lantad requested /robots.txt once from each of 1,419 hostnames on 17 September 2026. 1,004 files parsed into at least one group and named 2,209 distinct tokens between them, 1,098 of which appear in a single file, 813 of which contain a character RFC 9309 does not let a crawler put in its own name, and 120 of which name anthropic-ai or Claude-Web, neither of which appears on Anthropic's crawler page.
