Amazonbot robots.txt: 108 of 1,016 files named it, and 38 that blocked GPTBot let it in

What does an Amazonbot robots.txt rule control, and what does Amazon ignore?

Amazon publishes one page for its crawling, at developer.amazon.com/amazonbot, and it covers three user agents rather than one. The page opens by saying it describes how webmasters can control Amazonbot, Amzn-SearchBot and Amzn-User interactions with their site, that each user agent setting is independent of the others, and that a change may take about 24 hours for Amazon's systems to reflect. Each of the three carries its own example user-agent string and its own published IP address list at a separate URL. The Amazonbot list, fetched on 14 October 2026, holds 1,292 distinct IPv4 addresses written as bare addresses rather than CIDR ranges, no IPv6 at all, and a single creationTime field dated 8 September 2026 at 05:24 UTC. A published list is the only way to check the claim a user-agent string makes, and this blog has found those lists thin before: none of the 2,344 addresses OpenAI publishes answered a reverse DNS lookup, and the general point is that a user agent is a claim rather than an identity. Amazon's page documents no reverse DNS method for Amazonbot, only the address list.

The three are not interchangeable, and the page is careful about which does what. Amazonbot is described as used to improve Amazon's products and services, with the sentence that matters for anyone deciding whether to allow it: this helps Amazon provide more accurate information to customers and may be used to train Amazon AI models. Amzn-SearchBot is described as improving search experiences in Amazon products and services, with content becoming eligible to appear in experiences such as Alexa, and the page states that it does not crawl content for generative AI model training. Amzn-User supports user actions such as answering Alexa queries that need current information, and the page states that because its actions can be initiated by a user it may not follow all robots.txt directives.

The robots section is where the specifics are. Amazon states that automated crawling from these agents respects the Robots Exclusion Protocol, honouring the user-agent and the allow or disallow directives. That is the complete list of what is claimed. The page then states that they do not support the crawl-delay directive, and this blog's own read of nine crawler documentation pages on 20 September 2026 found only Anthropic documenting support for the field, with Apple and Amazon both documenting the opposite. It is a field that 152 of 1,056 files in this corpus set anyway. It adds that the agents respect the rel=nofollow link attribute and the page-level robots meta tags noarchive, noindex and none, glossing noarchive specifically as do not use the page for model training. That gloss makes Amazon one of the small number of vendors for whom a tag 7 of 1,077 home pages send is a training opt out rather than a cache instruction. Nothing on the page states a position on the wildcard and end-anchor characters that RFC 9309 section 2.2.3 requires crawlers to support, which matters because 50 of the 108 files naming Amazonbot use one of those characters in a rule written for it.

What was measured on 14 October 2026, and what was not

The population is the 1,419 hostnames in the two committed corpus seed files in this repository, worker/seeds/corpus-seeds-platform.json with 392 hostnames in ten platform strata and worker/seeds/corpus-seeds-industry.json with 1,027 hostnames in eight industry strata. Each was asked for its robots.txt once, as LantadBot/1.0, from a single network location, with redirects followed, a fifteen second timeout and no browser involved.

The funnel matters here more than usual, because every proportion below rests on which denominator it uses. 1,094 hostnames answered HTTP 200. 1,061 of those returned something that was not an HTML page served at the robots.txt address, a defect this blog has counted across the same corpus before. 1,016 of those parsed into at least one user-agent group, and that 1,016 is the denominator used for every file count in this post. Files that returned a 200 and parsed into no group at all were excluded because a file with no group cannot name a token. Of the remainder, 238 hostnames answered a status in the 400 to 499 range, 178 of them a 403, 34 answered 503, 8 answered 202, and 45 requests failed before any status arrived, through a timeout, a DNS failure or a TLS error. A 403 on the robots file is its own finding rather than a gap in the data, because a server that refuses the file to a crawler is a stronger block than anything the file could contain, and 79 of 115 hostnames measured earlier refused a crawler what their own file allowed.

Every file was then parsed with parseRobotsTxt and evaluated with evaluateRobots, both from core/src/robots.ts, which is the code that ships in the product rather than a throwaway script written for this post. A token counts as named when a user-agent line's value lowercases to that exact token. A root verdict is the answer evaluateRobots gives for the path slash, which is the strongest form of the question: a file can leave the root open and still close most of what matters. The method is the one described on the methodology page and used for the crawlability study.

Three limits apply to everything that follows. No Amazon crawler was observed doing anything: there is no traffic data here, only files and the vendor's own documentation, so every claim about Amazon's behaviour is a claim about what Amazon publishes and not about what its crawlers do. Second, one URL per hostname was requested, at the apex, so nothing here describes subdomains, and Amazon's page states its agents read robots.txt at the host level. Third, these 1,419 front doors are drawn from two committed sampling frames rather than from the web, so no percentage here generalises beyond them.

108 files named Amazonbot, and thirteen Amazon tokens appeared in all

108 of the 1,016 files named Amazonbot. 48 of those 108 disallowed it at the root and 60 left it allowed there. Across the whole 1,016, 54 files disallowed Amazonbot at the root, so six closed the root to Amazon's crawler through a wildcard group without ever naming it. For scale, 170 files named GPTBot and 81 disallowed it at the root. Against the other crawlers this blog has counted one at a time, Amazonbot is both better known and more often refused: 63 of 1,063 files named DuckAssistBot and 48 of those 63 let it in, where 60 of the 108 naming Amazonbot let it in.

How the 108 wrote the rule is as interesting as how many wrote it. 44 gave Amazonbot a group of its own. 64 put it in a group shared with other tokens, which means the rule was not written about Amazon specifically but about a list that happened to include it. 65 of the 108 carry at least one Allow line in the Amazonbot group, and 50 use a wildcard or an end anchor in a rule for it, characters Amazon's documentation does not mention. Exactly one file gave Amazonbot a Crawl-delay, coralvilleanimalhospital.com with a value of 5, which Amazon's page says it does not support.

Where the 108 came from undercuts the idea that this is a considered decision. 35 of the 54 hostnames in the corpus wix-squarespace stratum named Amazonbot, and 39 of the 108 carry the distinctive query-parameter rules of a Squarespace-generated file. That is platform default rather than authorship, the same mechanism behind 61 of 66 Squarespace sites naming an AI crawler without blocking it, and it is why the raw count of files naming a token is a weak measure of intent. The counterpart is the group nobody chose either: 559 of the 581 pages GPTBot lost in an earlier study were closed by a rule that never named it.

Thirteen distinct Amazon-named tokens appeared across the 1,016 files, named by 115 files between them. Three of the thirteen are on Amazon's crawler page. The other ten are not, and they are a reasonable picture of what people believe Amazon runs: amazon-kendra, AmazonBuyForMe, amazon-qbusiness, AmazonProductDiscoveryBot, Amazonbot-Video, AmazonSellerInitiatedListing, Amazon-Advertising-Ad-Standards-Bot, AmazonProductDiscovery, and a line reading Amazon AdBot with a space in it. That last one cannot bind to anything: RFC 9309 states the product token MUST contain only uppercase and lowercase letters, so a value with a space in it is not a product token and no crawler will match it.

Two of those ten are worth separating from the rest, because they are not Amazon reading your site. AWS documents the amazon-kendra token in its Kendra Web Crawler guide, at docs.aws.amazon.com/kendra/latest/dg/stop-web-crawler.html, which states that the crawler respects standard robots.txt directives like Allow and Disallow and that AWS customers are required to obtain authorization before indexing any particular website. That makes amazon-kendra a crawler an AWS customer points at a site, not an Amazon service collecting training data, and the same guide now carries a notice that Kendra is no longer open to new customers. amazon-qbusiness reads as the same kind of thing. Five files and two files respectively wrote rules for them, which is a reasonable response to a token appearing in logs and a poor substitute for the three agents Amazon's own crawler page names. If you want to see which of your own tokens bind, the robots.txt tester evaluates one path against each token separately rather than reporting the file as a whole.

38 files blocked GPTBot at the root and left Amazonbot in

The comparison the files support best is the one between the crawler people argue about and the crawler they do not. 38 of the 1,016 files disallow GPTBot at the root and allow Amazonbot there. 43 disallow both. 11 go the other way, allowing GPTBot and disallowing Amazonbot, among them washingtonpost.com, axios.com, aftenposten.no, va.gov, illinois.gov, notion.com and chewy.com. The remaining 924 allow both. So the lean is not unanimous, and that matters: enough sites have decided Amazon is the one to stop that the 38 cannot be read as everybody agreeing. The ratio is a little over three to one toward letting Amazon in, and the interesting part is what sits behind each side of it.

35 of those 38 never name Amazonbot anywhere in the file, while 37 of the 38 name GPTBot explicitly. That pair of numbers is the finding. These are not sites that weighed Amazon and decided to allow it; they are sites that wrote a list and Amazon was not on the list. The hosts are not obscure. 404media.co, aljazeera.com, cbsnews.com, hollywoodreporter.com, derstandard.at, france24.com and eltiempo.com all disallow GPTBot at the root while an Amazonbot request falls to a wildcard group that lets it in. So do airbnb.com, alibaba.com, canva.com and figma.com. OpenAI's own documentation describes GPTBot as the crawler whose fetches may be used to train its models, which the bots page at developers.openai.com states directly, and Amazon's page says the same thing about Amazonbot in almost the same words. A file that blocks one and not the other is making a distinction between vendors that neither vendor's documentation supports.

The most striking single case is amazon.com. Its robots.txt, fetched on 14 October 2026, carries 101 groups and 100 distinct user-agent tokens, and disallows 99 of them at the root: GPTBot, ClaudeBot, CCBot, PerplexityBot, Google-Extended, Bytespider, DuckAssistBot, Google-NotebookLM, Gemini-Deep-Research, Meta-ExternalAgent and 89 others. Not one Amazon or Amzn token appears in the file, so Amazonbot, Amzn-SearchBot and Amzn-User all fall to the wildcard group and are allowed at the root. Whatever the reason for it, that file is the clearest demonstration in this corpus that a long block list is a list of other companies rather than a policy about being crawled.

For a site deciding what to do, the useful framing is not which vendor to trust but which purpose to permit, because the purposes differ and the rules do not track them. Amazon has also attached a price to the answer. The same crawler page closes by saying that allowing Amazonbot in robots.txt may make a site eligible for Amazon Content Partners, which it describes as giving independent content creators a 1 percent affiliate commission boost on eligible sales, free hosting credits and AI traffic management tools, at contentpartners.amazon.com. That turns a robots.txt line into a commercial term, and it is the sort of arrangement that leaves no trace in the file itself: publisher tolls are invisible in robots.txt, so a reader cannot tell a site that allows Amazonbot on principle from one that is paid to. This blog has measured that gap before from the other side: seven of 22 files written to block AI training blocked nobody, and GPTBot remains the most blocked AI crawler, on 82 of 718 files, which says more about which name is in the news than about which fetch costs the site something.

Amzn-SearchBot is documented to follow other search bots' rules, and 9 files change answer

One sentence on Amazon's page describes behaviour the standard makes no provision for. If robots.txt files do not mention Amzn-SearchBot but allow other search bots, Amazon states, Amzn-SearchBot will crawl in accordance with the robots.txt directives given to other search bots.

That is not how the standard works. Under RFC 9309 a crawler whose product token is not named falls to the group whose token is the asterisk, and if no group matches at all it is unrestricted. There is no provision for borrowing another crawler's group, and no way for a site owner to predict which other crawler would be borrowed from. A file that allows Googlebot explicitly and disallows everything else through the wildcard group therefore has two readings: under the standard, Amzn-SearchBot is disallowed, and under Amazon's documented behaviour it crawls as Googlebot does, which is allowed.

How often that matters is measurable, and the answer is small but not zero. 71 of the 1,016 files named Googlebot. 12 disallowed the root to an unnamed crawler through a wildcard group. 9 files do both while never naming Amzn-SearchBot, which is the exact shape where the two readings give opposite answers at the root: amsterdam.nl, botcity.dev, coralvilleanimalhospital.com, gmarket.co.kr, instacart.com, sap.com, sciencedirect.com, scielo.org and theregister.com. No file in the corpus diverges the other way, so where the two readings differ, Amazon's documented behaviour is always the more permissive one.

coralvilleanimalhospital.com is the clearest example, because it does three things at once that the documentation answers differently. Its first group names 39 tokens together, Googlebot and Amazonbot among them, with path rules and a Crawl-delay of 5. Its second group is the wildcard with a bare disallow of everything. So Amazonbot is allowed by name, the crawl-delay written for it is a field Amazon says it does not support, and Amzn-SearchBot is disallowed by the wildcard under the standard while Amazon's own page says it would follow the rules given to Googlebot, which this file allows. Two of the three answers therefore come from Amazon's page rather than from the file, which is the opposite of what a robots.txt is for. The general problem is older than Amazon's version of it: a rule binds only to the token it names, which is why 1,004 files naming 2,209 distinct tokens mostly name nothing that crawls, and why a new token like OAI-AdsBot, named by 4 of 1,014 files, arrives into files that cannot anticipate it.

Two windows for an edit, and the 79 hostnames where unreachable decides

The last disagreement is about time, and it is the one that affects every site rather than nine of them. Amazon's page gives two windows for how long a robots.txt change takes to bind, in two different sentences, and does not say which governs. The opening paragraph says a setting change may take about 24 hours for Amazon's systems to reflect. The robots section says the agents will fetch host-level robots.txt files or use a cached copy from the last 30 days.

RFC 9309 section 2.4 is specific on this: crawlers SHOULD NOT use the cached version for more than 24 hours, unless the robots.txt file is unreachable. Thirty days appears in the RFC too, but in a different place and about a different thing. Section 2.3.1.4 offers 30 days as an example of a reasonably long period after which a crawler MAY treat a file that has been unreachable throughout as unavailable, or continue with a cached copy. Amazon's 30 days is a cache window for a file that was fetched successfully, which is the case section 2.4 covers and caps at 24 hours. The practical consequence for a site owner is the one this blog has measured from the crawler side: a robots.txt edit does not take effect when you save it, and on Amazon's documentation the gap between saving and binding is somewhere between one day and one month.

The second half of the same paragraph is sharper. When a file cannot be fetched, Amazon says, the agents will behave as if it does not exist. RFC 9309 splits that case in two and treats the halves oppositely. Section 2.3.1.3 covers unavailable, meaning a status in the 400 to 499 range, and says the crawler MAY access any resources on the server, which is Amazon's reading. Section 2.3.1.4 covers unreachable, meaning a server error in the 500 to 599 range or a network failure, and says the file is undefined and the crawler MUST assume complete disallow, which is the opposite of Amazon's reading. The two statuses pointing in opposite directions is a property of the standard that this blog has written about before, and Amazon's documentation collapses them into one rule.

That is not hypothetical in this corpus. On 14 October 2026, 34 of the 1,419 hostnames answered 503 for their robots.txt and 45 more failed before a status arrived. For those 79 hostnames, a conforming crawler is required to crawl nothing, and a crawler behaving as Amazon's page describes would crawl everything. Both numbers come from one network location at one moment and a 503 is often transient, so this is a statement about what the two documents require in a state these hosts were observed in, not a claim that 79 sites are being crawled against their wishes. If a robots.txt matters to your site, the thing to check is not its contents but whether it is reliably served at all, per host, since Amazon's page states the agents read it at the host level and 73 of 277 subdomains in this corpus gave an AI crawler a different answer from their apex.