BlogFindings
Applebot-Extended: 110 of 1,014 robots.txt files named Apple's training opt out, and 61 left it allowed
Lantad requested /robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 14 October 2026, as LantadBot/1.0, redirects followed, no JavaScript executed. 1,093 answered HTTP 200, 1,059 of those served something other than HTML, and 1,014 parsed into at least one group. 110 of the 1,014 named Applebot-Extended. 49 of the 110 disallowed it at the root and 61 left it allowed. On 63 of the 110 the root verdict was the same one the wildcard group would have given with the line deleted.
That makes it a different kind of rule from everything around it in the same file. A Disallow aimed at an AI crawler is meant to stop a request. A Disallow aimed at Applebot-Extended is meant to change what happens to bytes that were already collected under a different name. Nothing about the result is visible in a server log, which means the file itself is the only evidence a site owner has that the thing they intended is the thing they wrote. This post reads that evidence across a corpus of 1,419 hostnames and counts how often the intent and the file agree. If you want the same answer for one host before reading on, the robots.txt tester resolves a group per crawler token.
In short
- Lantad requested /robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 14 October 2026. 1,014 files parsed into at least one group, and 110 of them named Applebot-Extended, the robots.txt token Apple documents for opting out of training.
- Of those 110 files, 49 disallowed Applebot-Extended at the root and 61 left it allowed. 39 wrote exactly Disallow: / for the token and 7 wrote exactly Allow: /.
- On 63 of the 110 files, naming the token produced the same root verdict that this scanner's evaluator returns for a bot with no group of its own, which falls back to the wildcard. Writing the line changed nothing that could be measured from the file.
- Apple's own Applebot documentation, which carries a published date of 4 September 2026, states that Applebot-Extended does not crawl webpages and that it is only used to determine how to use the data crawled by the Applebot user agent.
- 31 of the 1,014 files disallowed GPTBot at the root while leaving Applebot-Extended allowed, and not one of those 31 named Applebot-Extended anywhere in the file.
| Group that binds Applebot-Extended at / | Files | Root allowed | Root disallowed |
|---|---|---|---|
| An explicit Applebot-Extended group | 110 | 61 | 49 |
| The wildcard group | 886 | 880 | 6 |
| An Applebot group, by most specific match | 11 | 10 | 1 |
| No group matched at all | 7 | 7 | 0 |
What is Applebot-Extended, and what does disallowing it actually do?
Apple runs one crawler and two tokens. The crawler is Applebot, and Apple's page says the data it collects powers search technology "including Spotlight, Siri, and Safari". The second token is Applebot-Extended, which Apple describes as giving "web publishers additional controls over how their website content can be used by Apple", specifically the choice to "opt out of their website content being used to train Apple's general purpose foundation models", naming those products as "including Apple Intelligence, Services, and Developer Tools". The example the page prints is two lines: a user-agent line naming Applebot-Extended, then Disallow: /private/.
The mechanics matter because they are unlike the rest of the file. RFC 9309, the Internet Standards Track specification of the robots exclusion protocol published in September 2022, is a document about fetching. It tells a crawler how to find rules, how to match its own product token case insensitively, and which rule wins when several match: "The most specific match found MUST be used", where "the most specific match is the match that has the most octets", and if an allow and a disallow are equivalent then "the 'allow' rule SHOULD be used". Every one of those sentences governs whether a request is made. None of them governs what a vendor does with a response it already holds.
Applebot-Extended sits in that gap. A site writes a rule in the grammar of fetching, and asks for an outcome that is not about fetching at all. There is no status code that confirms it, no log line, and no second request to observe. Apple's page does not say what happens if a publisher disallows Applebot itself rather than Applebot-Extended, so whether blocking the crawler also withdraws the training use is not answerable from the documentation. That silence is worth holding onto, because the most common thing a file in this corpus does about Apple is name the crawler and not the token, and our earlier reading of how Applebot falls back to Googlebot groups showed how far a file can drift from what its author meant.
Flow: Applebot requests the page to GET /robots.txt; GET /robots.txt to Crawl decision for Applebot; Crawl decision for Applebot (if allowed) to Spotlight, Siri, Safari; GET /robots.txt to Applebot-Extended rule read; Applebot-Extended rule read (governs use, not access) to Training use of stored data; Applebot-Extended rule read (nothing to observe) to No request, no log line.
How many robots.txt files named it, and how many blocked it?
Of the 1,419 hostnames in the two committed corpus seed files, 1,093 answered HTTP 200 for /robots.txt. 34 of those answered with an HTML document rather than a robots file, which leaves 1,059 plausible text responses, and 1,014 of those parsed into at least one user-agent group. 110 of the 1,014 named Applebot-Extended in a group heading. That is 10.8 percent of the parsed files, and it places Apple's training token third among the tokens this scanner knows: GPTBot was named in 169 files, Google-Extended in 141, Applebot-Extended in 110, and plain Applebot in only 23.
The ratio inside each of those populations is the part worth reading twice. Of the 110 files that named Applebot-Extended, 49 produced a disallowed verdict at the root and 61 produced an allowed one. Google-Extended behaves similarly in this corpus: 51 of the 141 files that named it disallowed it at the root. So on both of the two big training tokens, a clear majority of the sites that went to the trouble of typing the token did not end up refusing it. That is not a contradiction, and it is not necessarily an error. Some of those files allow deliberately, and seven of them say so in the strongest available form by writing nothing but Allow: / under the token, geico.com and mediamarkt.de among them.
The rest of the gap is more interesting than deliberate permission. 39 of the 110 wrote exactly one rule, Disallow: /, which is the unambiguous full opt out; nature.com, science.org, sciencedirect.com, cnn.com, forbes.com, bloomberg.com, techcrunch.com and airbnb.com are in that group. That leaves 64 files whose Applebot-Extended group carries more than one rule, and those are where the intent gets harder to read from the file. Counting named tokens alone would have reported 110 sites with an Apple training position, which is how a robots.txt census overstates adoption. The same arithmetic applies to the per crawler counts we published for Amazonbot and for DuckAssistBot, and the full capture procedure is on the methodology page.
On 39 files a platform wrote the token, not the site
39 of the 110 files carry a signature that is not editorial at all. Each one contains the same unusual pair of rules, an Allow for /api/ui-extensions/ and a Disallow for /commerce/digital-download/, alongside a long run of query string disallows for format=json, format=ical and reversePaginate. That is a hosting platform's default robots.txt, shipped identically to every site on it, and the corpus picked up 39 instances of it across dental practices, notaries, salons, florists and small charities. All 39 leave Applebot-Extended allowed at the root.
What those files do with the token is the detail that makes the count misleading. The platform does not give Applebot-Extended a group of its own. It lists it as one user-agent line in a single block of roughly thirty consecutive user-agent lines that also includes anthropic-ai, ccbot, claudebot, bytespider, meta-externalagent, youbot, omgili, facebookbot, several Google tokens, and the wildcard itself. Because the wildcard shares that block, every rule applied to Applebot-Extended is a rule applied to every other crawler that reads the file. The token is present, and it carries no Apple specific meaning whatsoever.
42 of the 110 files put Applebot-Extended in the same group as the wildcard this way. A reader counting tokens would score all 42 as sites with an articulated AI policy; a reader resolving groups finds that the policy is the site's ordinary crawl hygiene with more names attached to it. This is the same failure mode as the files we found that rule only on the whole site when they name AI crawlers, and it is why the token census in 2,209 crawler tokens across 1,004 files is a measure of what platforms ship rather than what owners decided. None of this is a criticism of the sites. A dentist did not choose to have an opinion about Apple Intelligence, and the file does not claim they did.
Platform default, 39 files, allowed at root
- User-agent: anthropic-ai
- User-agent: applebot-extended
- User-agent: ccbot
- User-agent: gptbot
- User-agent: *
- Disallow: /config
- Disallow: /search
- Allow: /api/ui-extensions/
nature.com, disallowed at root
- User-Agent: anthropic-ai
- Disallow: /
- User-Agent: Applebot-Extended
- Disallow: /
- User-Agent: Bytespider
- Disallow: /
- User-Agent: GPTBot
- Disallow: /
Naming the token changed nothing on 63 of the 110 files
There is a cleaner way to ask whether a rule did any work, and it does not depend on recognising a platform signature. For each of the 110 files, this scanner resolved the root verdict twice: once for Applebot-Extended, and once for a token with no group of its own, which under the specification falls back to the wildcard. If the two verdicts agree, then deleting every Applebot-Extended line from the file would have left the outcome at the root exactly where it is.
They agree on 63 of the 110. On 47 they differ, and those 47 are the files where naming Apple's training token moved the answer. The 63 include the 42 platform cases already described, plus files whose Applebot-Extended group restates rules the wildcard group already carried. The honest reading of 110 named is therefore 47 files where the line changed the root verdict, which is 4.6 percent of the 1,014 parsed files rather than 10.8 percent.
This test has a limit worth stating. It compares verdicts at the root path only, so a file that gives Applebot-Extended a narrower rule deeper in the tree than its wildcard group gives can register as unchanged here while differing on some other path. 53 files in the corpus allow Applebot-Extended at the root and still carry a real path disallow for it, and this test says nothing about those paths. What the root comparison does catch is the case a site owner most often believes they have avoided: a token written down and then undone, or never doing anything, which is the pattern behind both the wildcard dominance in 559 of 581 blocks and the defects in 232 of 1,059 files. Rules that resolve to nothing are the quiet half of AI visibility, because they leave an owner confident about a position their file never took.
-
Verdict differs from the wildcard fallback47 files Naming Applebot-Extended changed the root answer. -
Verdict identical to the wildcard fallback63 files Deleting every Applebot-Extended line would leave the root verdict unchanged. -
Token shares its group with the wildcard42 files Every rule for Apple's token is a rule for every other crawler too. -
Exactly one rule, Disallow: /39 files The unambiguous full training opt out. -
Exactly one rule, Allow: /7 files Deliberate permission, stated in the strongest form the grammar offers.
Eleven files never named it and still bound it
Group selection is a matching problem, and the token Apple chose makes it an awkward one. Applebot-Extended begins with the string Applebot. Google's robots.txt documentation describes group selection as taking "the group with the most specific user agent that matches the crawler's user agent", says that "all non-matching text is ignored", and gives the example that "both googlebot/1.2 and googlebot* are equivalent to googlebot". This scanner's evaluator implements that behaviour, so an Applebot group is a more specific match for Applebot-Extended than the wildcard is, and it binds.
Eleven files in the corpus are decided that way: they name Applebot, never name Applebot-Extended, and so Apple's training token inherits the crawler's rules. Ten of the eleven come out allowed. The eleventh, snu.ac.kr, writes Disallow: / under an Applebot heading in a long list of per crawler blocks, and so on this reading its training opt out is a side effect of a crawl block it wrote for a different purpose. arxiv.org, louvre.fr, booking.com, bestbuy.com, instacart.com and temu.com are among the other ten.
This is the one finding in this post that rests on a behaviour nobody has promised. RFC 9309 does not specify prefix matching of the group token at all: it says only that the product token "SHOULD be a substring of the identification string that the crawler sends", which describes how a crawler names itself rather than how a file is matched. Apple's page does not say how its own matcher resolves a group, and we did not probe Apple to find out. So treat those eleven files as ambiguous rather than as a measured Apple behaviour: on Google's documented rule they bind to the Applebot group, on a strict exact match reading they fall to the wildcard, and the file alone cannot distinguish the two. Token shape has caused this before, in mis-cased tokens that still bound and in stale rules left by renamed crawlers.
Group selection for Applebot-Extended on snu.ac.kr
- GET /robots.txt user-agent: LantadBot/1.0 (+https://lantad.co/bot)
- 200 OK text/plain
- file names: Applebot, Amazonbot, GPTBot, Baiduspider, *
- file does not name: Applebot-Extended
- candidate groups matching Applebot-Extended: applebot, * 2
- most specific match by Google's documented rule applebot
- rule found in that group for / Disallow: /
- verdict at / for Applebot-Extended disallowed
Blocking GPTBot is not blocking Apple
The cross tabulation between the two most written training tokens is where the practical advice sits. Across the 1,014 parsed files, 49 disallow both GPTBot and Applebot-Extended at the root, 31 disallow GPTBot while leaving Applebot-Extended allowed, and 7 do the reverse. The 31 are the number to look at, because not one of them names Applebot-Extended anywhere in the file. Their AI position was written against OpenAI's token and left Apple's to the wildcard. congress.gov, jstor.org, nejm.org, webmd.com, medscape.com, cbsnews.com, aljazeera.com, irishtimes.com and france24.com are in that set.
OpenAI's crawler documentation is explicit that its own token carries its own meaning, stating that "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models". Nothing in that sentence reaches another vendor, and there is no shared registry of training tokens that a single rule can address. Each vendor publishes its own string, and a file speaks to exactly the strings it contains. That is why GPTBot being the most blocked AI crawler we have measured tells you so little about any other crawler on the same hosts.
The 7 files that go the other way split into two different shapes, and only one of them is a choice about Apple. Five of them, fornidental.com, kidzandteendental.com, aftenposten.no, sfchronicle.com and washingtonpost.com, name Applebot-Extended with Disallow: / and never name GPTBot at all, so GPTBot falls to a wildcard group that leaves the root open. The washingtonpost.com file goes further and carves two paths back out, allowing /creativegroup/ and /advertising/ under the same heading. The other two, sap.com and gmarket.co.kr, are structurally the opposite: their wildcard group disallows the whole site, and a separate group naming GPTBot carries only path rules, so GPTBot is let in past a blanket block while Apple's token stays caught by it. A reservation written only in prose has the same reach problem, which is what 15 sites reserving text and data mining rights showed, and the current token list is on the AI crawlers page.
| GPTBot at / | Applebot-Extended at / | Files | What the file did |
|---|---|---|---|
| Disallowed | Disallowed | 49 | Both tokens written and refused |
| Disallowed | Allowed | 31 | None of the 31 names Applebot-Extended |
| Allowed | Disallowed | 7 | 5 name Apple's token only, 2 exempt GPTBot from a site wide block |
| Allowed | Allowed | 927 | Neither token refused at the root |
What this measured, and what it did not
This is a reading of files, not of behaviour. Every figure above comes from one GET of /robots.txt per hostname on 14 October 2026, sent as LantadBot/1.0 from a single network location with redirects followed and no JavaScript executed, parsed with this repository's own parseRobotsTxt and evaluated with its evaluateRobots. No Apple traffic was observed, no page was fetched as Applebot, and nothing here establishes whether Apple honours any rule in any of these files. A file is a statement of intent, and this post counts statements.
Four limits are worth naming precisely. Verdicts are computed at the root path only, so a token allowed at / may still be disallowed deeper, and 53 files in the corpus are exactly that case. The 326 hostnames that did not answer 200 are absent from every denominator, and a 403 or a timeout is not evidence of a permissive file, as the difference between a 404 and a 503 makes clear. The eleven prefix matched files rest on Google's documented group selection rather than on anything Apple has published. And a single capture says nothing about change over time, which matters because an edit takes a while to reach a crawler, as one measured robots.txt edit showed.
The one recommendation this supports is narrow. If a site intends to opt out of Apple's foundation model training, the token has to appear in a group of its own, with a rule that resolves against the paths that matter, and it should be checked rather than assumed, because 63 of the 110 files that named it would read identically with the line removed. Our scanner's own policy is published for the same reason, the corpus and its strata are described under research, and the per token counts for other vendors sit alongside this one in posts such as Bytespider in 130 of 1,077 files.
- Token appears in a group of its own Checkable in the file. 68 of the 110 did not share the group with the wildcard.
- Root verdict differs from the wildcard fallback Checkable by evaluating the file twice. True on 47 of the 110.
- Rule covers the paths the site cares about Checkable per path. 53 files allow the root and still disallow some path for the token.
- Apple honoured the rule Not measurable from a file, and not measured here. No Apple request was observed.
- Blocking Applebot also withdraws training use Apple's page does not state this. It says only that Applebot-Extended governs use of data Applebot crawled.
Lantad
Published .
A site that wants out of Apple's model training has one documented lever, and it is a line in a text file. Apple's support page for its crawler describes a second product token, Applebot-Extended, and says publishers opt out by disallowing it in robots.txt. The page is at support.apple.com/en-us/119829 and carries a published date of 4 September 2026. It states plainly that this token "does not crawl webpages" and that it "is only used to determine how to use the data crawled by the Applebot user agent". So the token is not a crawler to be kept out. It is a flag attached to data Apple has already fetched.
Common questions
What is Applebot-Extended?
It is a robots.txt product token Apple documents for controlling whether a site's content is used to train Apple's general purpose foundation models. Apple's support page, published 4 September 2026, states that it does not crawl webpages and is only used to determine how to use the data crawled by the Applebot user agent, so it governs use rather than access.
Does disallowing Applebot-Extended stop Apple crawling my site?
No. Apple's documentation describes Applebot as the crawler and says the data it collects powers Spotlight, Siri and Safari. Applebot-Extended issues no requests of its own, so a rule aimed at it changes nothing about what is fetched. A site that wants to stop the crawling has to address Applebot instead.
How many sites block Applebot-Extended?
In this corpus, 56 of the 1,014 parsed robots.txt files produced a disallowed verdict for it at the root on 14 October 2026: 49 through a group that named the token, 6 through a wildcard group, and 1 through an Applebot group matched on prefix. 110 files named the token in total.
Does blocking GPTBot also opt me out of Apple's training?
No. Each vendor publishes its own token and a file speaks only to the strings it contains. 31 files in this corpus disallowed GPTBot at the root while leaving Applebot-Extended allowed, and none of those 31 named Apple's token anywhere.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
Applebot robots.txt: 64 files hand Apple's crawler to their Googlebot group, and 44 of those answer differently
Apple documents that when a robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot instructions. Lantad requested /robots.txt once from each of 1,419 hostnames on 19 September 2026. 1,023 parsed into at least one group, 22 named Applebot, and 64 named Googlebot without naming Applebot. On 44 of those 64 the Googlebot group and the wildcard group give different answers, and on five the wildcard refuses the whole site while the Googlebot group lets Apple in.
Amazonbot robots.txt: 108 of 1,016 files named it, and 38 that blocked GPTBot let it in
Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 14 October 2026. 1,016 parsed into at least one user-agent group. 108 of them named Amazonbot, 48 of those 108 disallowed it at the root, and 38 files across the whole set disallowed GPTBot at the root while leaving Amazonbot in, 35 of them without ever naming Amazonbot. Amazon's own crawler page documents three user agents, says they do not support crawl-delay, and says they may use a cached robots.txt copy from the last 30 days, against the 24 hours RFC 9309 recommends.
Text and data mining opt out: 15 sites reserved the right in prose, and none served the machine-readable file
Lantad requested robots.txt, then the home page, then every legal page the home page linked, for all 1,419 hostnames in this repository's two committed corpus seed files on 14 October 2026, executing no JavaScript. 656 hostnames delivered a readable legal page. Reading all 32 candidate clauses by hand confirmed 15 whose own terms restrict the use of site content for AI or machine learning training, and 11 of those 15 leave all nine training crawler tokens this scanner evaluates allowed at the root. Six hostnames in the whole corpus serve a TDMRep file at /.well-known/tdmrep.json, and none of the six is one of the 15.
Most blocked AI crawler: GPTBot in 82 of 718 robots.txt files, and its search twin in 24
Lantad requested /robots.txt once from each of 1,027 hostnames across eight industry sectors on 12 September 2026, and 718 of the responses parsed as robots.txt. GPTBot was disallowed at the site root by 82 of them, seven more than the fourth placed token. The wider gap is inside each vendor: 59 of the 718 refused GPTBot while leaving OAI-SearchBot allowed.
Crawler user agent: 1,004 robots.txt files named 2,209 tokens, and 120 name one Anthropic no longer documents
A user-agent line in robots.txt is an exact product token, and a token that matches nothing is a rule that does nothing. Lantad requested /robots.txt once from each of 1,419 hostnames on 17 September 2026. 1,004 files parsed into at least one group and named 2,209 distinct tokens between them, 1,098 of which appear in a single file, 813 of which contain a character RFC 9309 does not let a crawler put in its own name, and 120 of which name anthropic-ai or Claude-Web, neither of which appears on Anthropic's crawler page.
