Detect GPTBot: 0 of 2,344 addresses OpenAI publishes answered a reverse DNS lookup

How do you detect GPTBot in your own server logs?

Three checks exist and they are not interchangeable. The first is to match the User-Agent header against the string the vendor publishes, which fails as proof for the obvious reason that any client can send any header, and fails in a second way this blog has already counted: of the fifteen tokens in the registry, only six carry a complete published User-Agent string to match against, because a robots.txt product token and an HTTP header are different strings and vendors do not always publish both. Matching what is left is also an ordered problem rather than a lookup, and every failure mode in that ordering returns a confident wrong label rather than an error.

The second check is to compare the source address against a list the vendor publishes. That is a real check, because the list lives on a host the vendor controls, and it is the check OpenAI's own documentation points site owners at: the page recommends allowing OAI-SearchBot in robots.txt "and allowing requests from our published IP ranges below". The third is the forward confirmed reverse DNS lookup, which Google's page on verifying requests from its crawlers, last updated 2026-03-20 UTC, sets out as two command line steps: run a reverse lookup on the address, confirm the name sits in a domain the operator owns, then run a forward lookup on that name and confirm it returns the address you started with. The second step is what makes it proof, because a PTR record can be set by whoever controls the address block while the forward record can only be set by whoever controls the domain.

None of this is visible from outside a site, which is why this blog has reported the mechanism before without measuring it: when Cloudflare rebuilt its bot submission form, the finding was that its automated review fetches an IP list, confirms reverse DNS or validates a signature while taking the operator's description of its own behaviour on trust, and that post said in its own takeaways that Lantad had measured none of it. What can be measured, and is measured here for the first time, is whether the two checkable signals actually work on the addresses the vendors name. Note what is not in this list: robots.txt is not a detection mechanism at all. RFC 9309 governs what a well behaved crawler should not fetch rather than who is allowed to ask, and its own section 3 says so in one line, that "the Robots Exclusion Protocol is not a substitute for valid content security measures".

Which AI crawler vendors publish a list of their addresses?

Six of the nine do, across fifteen files holding 4,589 IPv4 prefixes and 1,068 IPv6 prefixes between them. OpenAI's crawler documentation links four files, one per crawler. Anthropic's help centre article links one file at claude.com/crawling/bots.json covering all of its bots together, which is the same single list arrangement this blog found when it read the file that cannot tell Anthropic's three named bots apart, now stamped 7 October 2026 and holding 38 prefixes rather than 26. Perplexity's bots page links two, Apple's support article links one, Google's crawling documentation links four, and Amazon's crawler page links three.

The three that publish nothing a machine can read are worth stating precisely, because the reasons differ. Meta's web crawlers page carries one instruction on the subject, reading "Add to your allow list either the user agent strings or the IP addresses (more secure) used by the crawler", and it makes the words "user agent strings or the IP addresses" a link to the anchor identify on the same page. That anchor is declared on four separate headings in the page source, and the page carries no IPv4 literal anywhere in it, no CIDR block and no reference to a whois record or an autonomous system number. ByteDance's Bytespider page at zhanzhang.toutiao.com/docs/intro/26899 answered HTTP 200 and delivered a document whose only readable text was its title, with the body requiring JavaScript, so Bytespider is recorded here as not assessed rather than as publishing nothing. That is the same crawler whose robots.txt treatment this blog counted yesterday, when 130 of 1,077 files named Bytespider and 70 shut it out at the root.

Common Crawl's CCBot page is the third and the most frustrating. It documents reverse DNS properly, states that the crawler "is now run on dedicated IP address ranges with reverse DNS", gives a worked dig example, and says the ranges "are also provided as JSON at https://index.commoncrawl.org/ccbot.json". Three requests to that URL from this network location on 12 October 2026 were each cut off before any HTTP response arrived, twice with a connection reset and once with an empty reply, so the file was not read and nothing is claimed about its contents. That is a gap in this measurement rather than a finding about Common Crawl.

Amazon is the only vendor that publishes the same thing three different ways. All three of its lists are a JSON object embedded in an HTML page rather than served as JSON, and the schema is not consistent between them: the Amzn-SearchBot page uses the key ip_prefix with a /32 suffix on every value, which is AWS's own shape, while the Amazonbot and Amzn-User pages use the key ipv4Prefix with bare addresses carrying no prefix length at all. A parser written against any one of those three rejects the other two.

How old is the list you are checking against?

Every one of the fifteen files carries a creationTime, and the spread between them is the first thing that decides how much the IP check is worth. Google's four were all stamped 2026-10-09, three days before they were read. Anthropic's and OpenAI's chatgpt-user.json were five days old. At the other end, perplexitybot.json carried 2025-02-07T16:56:00, which is 612 days before this measurement, and the file holds eight prefixes covering eighteen addresses in total. Perplexity's smaller perplexity-user.json was stamped 360 days earlier, Amazon's Amzn-User page 342 days, and OpenAI's searchbot.json 283 days.

Staleness cuts in a specific direction and it is worth being exact about which. A list that has not changed is not necessarily wrong: a crawler fleet that has not moved needs no new file, and OpenAI's searchbot.json may be accurate at 283 days old. What a long gap does is remove the site owner's ability to tell the two cases apart, because a list that is never updated and a list that needs no update produce the identical file. The cost lands asymmetrically: an address the vendor has added since the stamp will be refused by a server allowlisting from that file, and the crawler is turned away with no error anybody will read. This blog has looked at the other half of that problem before, finding that these files answer the question of when they last changed twice, and on six of thirteen the two answers differed by between 13 and 942 days, because the creationTime inside the file and the Last-Modified header on the response are set by different things.

The sizes have also moved since this blog last counted them. When OpenAI's four files were opened and counted on 29 July 2026, chatgpt-user.json covered 37,328 IPv4 addresses against GPTBot's 3,472. On 12 October 2026 the same two files cover 36,496 and 2,704. The direction is worth noticing: both shrank, and gptbot.json now holds eighteen prefixes. The user triggered fetcher is still the largest list OpenAI publishes, covering 36,496 addresses against 2,704 in the next largest and 5,520 across the other three together.

Does a reverse DNS lookup work on the addresses vendors publish?

For four of the six publishers it works almost everywhere, and for one it does not work at all. Lantad asked 11,362 distinct published addresses for a PTR record. Every address published as an exact single address was asked directly, which covers Amazon's 3,131, the 32 exact entries in Anthropic's file and eight of Perplexity's twelve. For every address published as a block, the first usable address was asked, and for OpenAI's 293 blocks, Anthropic's six and all 24 of Apple's, up to eight further addresses per block were asked as well, Apple's exhaustively.

The results split cleanly. All 3,131 of Amazon's addresses answered with a PTR record and all 3,131 forward confirmed. 4,641 of Apple's 4,704 addresses answered and every one of those forward confirmed, leaving 63 without a record. All twelve of Perplexity's answered and confirmed. Google's four lists answered on 694 of 1,091 addresses asked, and 652 of those forward confirmed. Anthropic's file answered on its 32 exact addresses and on none of the 48 addresses sampled from its six blocks. OpenAI's four files answered on nothing: 2,344 addresses asked, 2,344 NXDOMAIN responses, which is 144 for GPTBot, 312 for OAI-SearchBot, 1,872 for ChatGPT-User and 16 for OAI-AdsBot. Three of those addresses were then re-checked by hand against two independent public resolvers, which agreed.

One methodological point belongs in the post rather than in a footnote, because getting it wrong would have produced a published falsehood. Asking only the first usable address of each block said that 3 of Apple's 24 blocks had reverse DNS. Asking all 4,704 addresses in those same blocks said 4,641 did. The first address of a block is very often the one nobody has assigned, so sampling the edge of a range understates PTR coverage badly, and that is why the Apple figure here comes from an exhaustive pass and why OpenAI's zero is reported against 2,344 addresses rather than 293. It is also why the honest claim about OpenAI is bounded: its blocks cover 42,016 addresses and 2,344 of them were asked, so what is measured is that no sampled address has a PTR record, consistent with there being no reverse DNS scheme rather than proving one cannot exist. The same bounding applies to anything a reader checks with the crawler view tool or reads in the methodology: a measurement states its sample.

What does the reverse DNS hostname actually prove?

A PTR record that forward confirms proves that whoever controls the domain in the name also controls the address. It does not say who that is, and the names divide into two populations that are worth very different amounts. 5,812 of the 8,510 answering addresses resolved into a domain the crawler's own operator holds: 4,641 under applebot.apple.com, which is exactly the mask Apple's support article describes, 165 under googlebot.com, 387 under google.com, 95 under gae.googleusercontent.com, and 524 under crawl.amazonbot.amazon, a name inside Amazon's own top level domain. On any of those, the lookup settles identity. A server seeing crawl-66-249-66-1.googlebot.com resolve back to the address that sent the request knows it is talking to Google.

The other 2,698 resolve to the hosting provider. All twelve of Perplexity's published addresses resolve to a name of the form ec2-107-20-236-150.compute-1.amazonaws.com, which identifies an EC2 instance and nothing further; the same name shape would be returned for any Amazon Web Services customer's machine. 29 of Anthropic's 32 exact addresses resolve under bc.googleusercontent.com and the remaining three under the same EC2 pattern. 2,607 of Amazon's own 3,131 resolve to the EC2 name rather than to crawl.amazonbot.amazon, which makes Amazon the one vendor whose fleet is split across both populations: 524 addresses provably Amazonbot, 2,607 provably an EC2 instance.

There is a sharper consequence in the Anthropic case and it is nobody's fault. Google's verification instructions tell a site owner to run the reverse lookup and then "verify that the domain name is either googlebot.com, google.com, or googleusercontent.com". Anthropic's addresses resolve under bc.googleusercontent.com, which is a subdomain of the third of those. An implementation that follows those instructions as a domain suffix test, which is how they are written, will forward confirm Anthropic's crawler and conclude it has verified a Google request. The narrower masks in the same page's table, which specify gae.googleusercontent.com for the fetchers that use that domain, do not have the problem; the prose instruction does. For AI visibility work this is the practical upshot: a generic cloud name is a fact about hosting, and treating it as identity is how a wrong allow decision gets made. It is also why the signature based alternative matters, and when this blog counted who had deployed it, 2 of 9 vendors published a Web Bot Auth key directory. The user agent swap measurement that found 103 of 1,089 sites served an unknown bot and refused GPTBot is the view of the same problem from outside the server.

IPv6, and the gap between the documentation and the measurement

Three of the nine vendors document reverse DNS as a way to identify their crawler. Google sets out the masks and the two step check. Apple's support article, carrying a published date of 4 September 2026, says traffic from Applebot "is generally identified by using reverse DNS in the *.applebot.apple.com domain" and gives the host commands. Common Crawl's page documents it, and adds the one caveat nobody else states: the ranges carry reverse DNS "except over IPv6 where reverse DNS is not yet supported". The crawler documentation of OpenAI, Anthropic, Perplexity and Amazon contains the string DNS zero times, and Perplexity's page instead advises combining "both User-Agent string matching and IP address verification".

Two of those four are then contradicted by the measurement in opposite directions. Amazon documents no reverse DNS and all 3,131 of its published addresses have a forward confirming PTR record, 524 of them under a name that says Amazonbot: a site owner who read only the documentation would not know the strongest check available to them exists. Apple documents reverse DNS and 63 of its 4,704 addresses have no record, which is a small enough share to be unremarkable except for the direction, since a documented mechanism that silently misses an address is worse than an undocumented one that works.

IPv6 is the hole that matters most and it is not close. Of the 1,068 IPv6 prefixes in all fifteen files, every single one is Google's. OpenAI, Anthropic, Perplexity, Apple and Amazon publish 4,589 IPv4 prefixes and not one IPv6 prefix between them. A site reached over IPv6, which is most sites behind a modern CDN, therefore has no published list to check a connection from any of those five against, and no documented reverse DNS mask either. That leaves the User-Agent header, which is the string this whole post started by discarding. Checking what your own file says to each token is a separate job and the robots.txt tester does it, but it answers a different question: robots.txt decides what a crawler should fetch, and none of it decides who is asking. Lantad publishes its own crawler's identity and contact at the bot page, which is the same courtesy measured here and the cheapest one a crawler operator can extend.