Applebot robots.txt: 64 files hand Apple's crawler to their Googlebot group, and 44 of those answer differently

Applebot robots.txt rules: which group applies when you write none?

Two documents answer that question and they do not agree, which is the whole of this post. RFC 9309, the Robots Exclusion Protocol, tells a crawler to find the group whose product token matches its own name, and if no group matches, to use the group headed by an asterisk. Google's robots.txt specification describes the same selection from the implementation side, adding that only the single most specific matching group applies and the rest are ignored entirely. Neither document contains any provision for one crawler borrowing another crawler's group. Under both, a file that never writes the word Applebot gives Applebot the wildcard group, and that is what this scanner's parser returns, as the methodology sets out.

Apple's own page says otherwise. The About Applebot article, which is published at support.apple.com/en-us/119829 and carries a last updated date of 4 September 2026, was read at source on 19 September 2026. Under the heading about customising robots.txt rules it states that Applebot respects standard robots.txt directives in general search crawls that are targeted at Applebot, and then, in the next paragraph, that if robots instructions do not mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions. The same page states that Applebot does not follow crawl-delay, and that publishers can opt out of generative model training by disallowing Applebot-Extended.

The consequence is specific rather than philosophical. On a file with a wildcard group and a Googlebot group and no Applebot group, there are two defensible answers to the question of what Applebot may fetch, and they are produced by reading two different documents. If the two groups say the same thing, nothing turns on the choice. If they differ, the site owner has written one set of rules for search engines in general and a second for Google, and Apple has quietly put its crawler in the second bucket. Nothing in the file records that decision, no validator reports it, and the owner has no reason to look. This is the same class of problem as a group whose token no crawler can match, which is why a renamed crawler token leaves the rules behind it matching nothing: the file parses, the rules are real, and they govern a different crawler than the author had in mind.

Five files refuse every unnamed crawler and let Apple in

The starkest cases are the ones decided at the site root, because there the two readings do not differ by a directory or a query string but by the whole site. Five of the 64 files carry a wildcard group that refuses everything and a Googlebot group that does not refuse the root: amsterdam.nl, sciencedirect.com, scielo.org, sap.com and gmarket.co.kr. Read under RFC 9309, Applebot is refused the entire site on all five. Read under Apple's documented fallback, Applebot is allowed. There is no case anywhere in the 64 that runs the other way at the root, so where the root verdict moves at all, it moves in Apple's favour.

The files are small enough to quote in outline. gmarket.co.kr is 347 bytes in total: a wildcard group carrying Disallow: /, then a second group whose eight consecutive user-agent lines name GoogleBot, BingBot, Yeti, Daumoa, Twitterbot, GPTBot, AdsBot-Google and Ads-Naver together, followed by a crawl-delay and two Allow rules. Apple is not in that list and Applebot is nowhere in the file, so a site that has named the eight crawlers it wants, including one AI crawler by name, has admitted a ninth it never considered. sap.com has the same shape at greater length, with a wildcard Disallow: / followed by a group of twenty user-agent lines covering Googlebot and its variants, Bingbot, Baiduspider, DuckDuckBot, MojeekBot, SeznamBot and others. sciencedirect.com opens with Disallow: / under the wildcard and then writes a Googlebot group of narrow path exclusions. scielo.org writes Disallow with an empty value under Googlebot, which RFC 9309 defines as imposing no restriction at all, and a wildcard Disallow: / further down the file.

These are not careless files. They are files written by people who thought about which crawlers to admit and enumerated them, which is the opposite of the far commoner pattern where 559 of 581 disallowed paths were decided by a wildcard group that never named the crawler. The enumeration is exactly what creates the exposure: a file that names neither Applebot nor Googlebot has nothing for Applebot to fall through to, and 937 of the 1,023 files are in that position. It is the deliberate, crawler-by-crawler file that hands Apple a key, and the more carefully the Googlebot group is written the further it can diverge. Anyone wanting to see which group their own file gives a named crawler can run it through the robots.txt tester, with the caveat in the last section about what that tester currently does with Applebot.

On 44 of the 64 files the two groups disagree

Below the root the comparison needs a set of paths, and the honest set is the one the file itself supplies. For each of the 64 files every rule pattern in every group was turned back into a concrete path, with a trailing end anchor removed and each asterisk replaced by a literal character, and the site root was added. Each of those paths was then evaluated twice against the same parsed file, once for Applebot, which with no Applebot group resolves to the wildcard, and once for Googlebot. Where the two verdicts differ, the file is one where Apple's documented fallback changes the answer. 44 of the 64 files have at least one such path.

The direction is not symmetrical. Across all the differing paths, 513 comparisons gave Applebot more access under the Googlebot group than under the wildcard, and 221 gave it less. Both directions are failures of the author's intent, and the second is the one nobody expects. time.com is the clearest example of it: its wildcard group is a single Allow: /, and its Googlebot group carries 80 rules, most of them disallowing query string patterns such as a search parameter and various tracking parameters. 78 of the 80 paths tested came back different. A site that has explicitly opened itself to every crawler has, without writing a word about Apple, handed Applebot the crawl restrictions it wrote for Google.

The opposite direction is better represented and easier to produce by accident. newegg.com writes 72 rules under the wildcard and eight under Googlebot, and 62 of the 74 paths tested differ. nhs.uk writes 31 under the wildcard and one under Googlebot, and 32 of 33 differ. edreams.com writes 60 and five. abc.net.au writes 22 and one. In each case the wildcard group is the real ruleset and the Googlebot group is a short list of exceptions, so an unnamed crawler routed to the Googlebot group escapes almost all of the site's restrictions. This is the second layer of the problem this blog has described before, where robots.txt and the server can disagree about the same crawler, except here the disagreement is inside the file rather than between the file and the edge, and it is one host's file governing one host rather than a subdomain problem.

It is worth saying what these 64 files are not. They are not files written by people unaware of AI crawlers: 27 of the 64 name at least one of the 15 AI crawler tokens this scanner evaluates. They are files whose authors addressed AI crawlers by name, addressed Google by name, and never wrote the name of the crawler that feeds Siri and Spotlight. The same corpus sweep two days earlier found that 1,004 files named 2,209 distinct user-agent tokens between them, which is a large vocabulary to have assembled while leaving Applebot absent from 1,001 of those files.

106 files name the opt-out and 22 name the crawler

The token counts are the other half of this measurement and they are lopsided in a way worth stating plainly. 106 of the 1,023 files name Applebot-Extended. 22 name Applebot. Only 11 files name both, so 95 of the 106 sites that wrote a rule about Apple wrote it exclusively about the token that does not fetch anything. Apple's page is explicit that Applebot-Extended controls whether crawled content is used to train generative models, and that Applebot is the crawler doing the crawling. Writing the first and not the second leaves the fetch governed by whatever group the fallback selects.

Of the 106, 44 actually disallow Applebot-Extended at the site root, which is the opt-out working as documented; the other 62 name it in a group that does not refuse the root. The 44 include most of the large publishers and journals in the corpus, among them nature.com, sciencedirect.com, science.org, jamanetwork.com, cnn.com, bloomberg.com, forbes.com, bostonglobe.com and nzz.ch, which matches the pattern this blog measured when 52 of 54 sites refusing Google-Extended still allowed Googlebot. Refusing training while keeping search is a coherent and common position, and Apple supports it with a separate token in the way Google does. The gap is that Google-Extended controls a crawler that does not exist either, while Apple's search crawler has a documented fallback that the training token does nothing about.

sendbird.com is the case that puts both halves together. Its file names 22 user-agent tokens, one per group, including Applebot-Extended with Disallow: /, along with GPTBot, ClaudeBot, CCBot, Bytespider, Amazonbot, meta-externalagent, cohere-ai, Diffbot and omgili all refused the same way, and OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, DuckAssistBot and MistralAI-User all admitted with an empty Disallow. Its wildcard group carries 52 rules restricting documentation reference paths. Its Googlebot group carries a single Disallow with an empty value, which is no restriction at all. Applebot is not named. So the site has refused Apple's training use, thought carefully enough about AI crawlers to rule on 22 tokens, and left the Apple crawler that actually issues requests following an unrestricted Googlebot group past 52 rules it wrote for everybody else. That only nine vendors publish crawler documentation at all, and six of them publish exactly one token, is part of why files like this one accumulate tokens faster than they accumulate accuracy, and it is the same shape as writing a rule for a crawler a vendor does not publish, which is what happened when one of 323 files named a Mistral token and used a name Mistral does not document.

What this measurement found wrong in our own scanner

The parser used throughout this post is the one this product ships, and it does not implement Apple's fallback. It implements RFC 9309 with Google's documented refinements, which means that asked about Applebot on a file with no Applebot group it selects the wildcard group and reports that verdict with no qualification. On the 44 files identified above, that verdict is the wrong one by Apple's own description of its crawler, and on the five root cases it is wrong in the most consequential way available: it will report that Applebot is refused the entire site when Apple documents that it is not.

That is a defect in this scanner and it is recorded here rather than quietly fixed and forgotten, because the shape of it generalises. A robots evaluator that is correct against the standard can still be wrong about a specific crawler, since the standard describes what crawlers ought to do and each vendor's documentation describes what one of them says it does. Applebot is the case in this corpus, but it will not be the only one, and the honest position for a tool that sells per-crawler verdicts is that a vendor deviation documented on the vendor's own page beats a correct reading of the RFC. The robots.txt validator run two days ago checked files against the specification and this is the complementary failure: a file can be perfectly valid and still be read by a crawler in a way the specification does not describe.

One further Apple deviation is measurable in the same sweep and costs nothing to report. Apple's page states that Applebot does not follow crawl-delay. 162 of the 1,023 files carry at least one crawl-delay line, and 15 of those are among the 64 files the fallback governs, including amsterdam.nl, which sets a crawl-delay of 3 inside the very Googlebot group Applebot is documented as following. Crawl-delay is not in the four fields Google supports either, and a crawler can lose its verified status over the directive rather than gain a slower crawl from it. A site relying on it to pace Apple is relying on something Apple says in writing it does not read. Which crawlers a file names, and what each vendor documents about the token, is what the AI crawler reference is for.

What this did not measure

No access log was read on any site in this corpus and no request from Applebot, or from any other named crawler, was observed. Everything above is a reading of published files and of one vendor's published documentation, so it evidences what those files say and what Apple states in writing, and it evidences nothing about what Apple's crawler actually does when it arrives. A vendor's documentation is a claim, and this blog has made the point about other vendors often enough that it applies here too: a user agent is a claim rather than an identity, and so is a sentence on a support page. The fallback is reported here as documented behaviour, not as verified behaviour.

The corpus is an editorial sampling frame of large organisations across government, education, healthcare, news, software, ecommerce, travel and finance, plus a set grouped by publishing platform. It is not a random draw from the web, so every rate here supports a statement about these 1,419 hostnames and nothing wider. 306 hosts did not answer with a 2xx status and 30 failed at the transport layer, and those are absent from every figure, which means sites most defensive about unfamiliar clients are under-represented. One request was sent per host from one network location, so a file that varies by geography or by requester would not show it here, and asking a site's own file for a verdict is a different exercise from asking whether the crawler can reach the file at all.

The path comparison also deserves its limit stated. Paths were generated from the rule patterns in each file, which is the right sample for finding disagreement between two groups in that file, and the wrong sample for estimating how much of a real site is affected. A file with 80 rules produces 80 test paths and a file with two produces two, so the count of differing paths measures the size of the divergence in the file rather than the share of a site's URLs that move. The 44 figure is the one that generalises, because it is a count of files rather than of paths, and no claim is made here about what proportion of any site's content changes hands.

Finally, nothing in this post says a site is worse off for any of it. Five sites admitting Applebot to a site otherwise closed may be exactly what their owners want, since Applebot feeds Spotlight, Siri and Safari, and a site that wanted to be found on Apple devices would be right to allow it. The finding is not that these files are wrong. It is that none of them says anything about Apple, that what happens to Applebot on all 64 is therefore decided by a sentence on Apple's support site rather than by anything the author wrote, and that no tool the author is likely to run will mention it. Our own crawlability study and the broader question of AI visibility both rest on per-crawler verdicts, which is why a vendor deviation this size is worth a post rather than a footnote.