Crawler IP range files carry two dates, and six of thirteen disagree

What a crawler IP range file actually contains

No specification governs this file. There is no RFC for it and no registry of where each vendor keeps one, yet all five have converged on the same object anyway. It has two keys. The first is creationTime, a string holding an ISO timestamp. The second is prefixes, an array in which every entry is an object carrying exactly one key, either ipv4Prefix or ipv6Prefix, whose value is a CIDR block. That is the whole format. Across all thirteen files we found no version number, no expiry, no signature, no checksum and no field describing what changed since the last edition.

The thirteen files hold 1,417 IPv4 prefixes and 1,057 IPv6 prefixes between them, and the IPv4 side covers 156,080 addresses. The IPv6 prefixes are not distributed the way you might expect. All 1,057 of them are Google's. The other eight files, four from OpenAI, two from Perplexity, one from Apple and one from Microsoft, contained no IPv6 entry of any kind. A site reached over IPv6 by something claiming to be GPTBot, PerplexityBot, Applebot or bingbot has nothing in these files to check the address against, which is a gap rather than a permission.

The address counts are also spread over four orders of magnitude, and the spread does not track how prominent the crawler is. The GPTBot range file carried 21 prefixes covering 3,472 addresses. Apple's applebot.json carried 33 prefixes covering 7,056. Microsoft's bingbot.json carried 28 covering 4,736. Perplexity's perplexitybot.json carried 8 prefixes covering 18 addresses, because every entry in it is a single host. The largest single file was OpenAI's chatgpt-user.json at 36,016 addresses, and Google's user-triggered-agents.json covered 67,840 from just three IPv4 prefixes, one of which is a /16.

None of that tells you a crawler is real. It tells you an address is inside a published range, which is a much weaker statement, and the difference is the subject of a user agent is a claim, not an identity. The tokens each vendor documents are listed on the free AI crawler reference.

The two dates a file gives you do not agree

A site that verifies crawlers by address has to refetch these files, because a list it copied once is a list that goes stale. The obvious way to decide when to refetch is to ask the file when it last changed. The problem is that each file answers that twice and the answers are not the same.

Take Microsoft's bingbot.json. The creationTime inside Microsoft's bingbot range file reads 2024-01-03, which was 959 days before we fetched it. The Last-Modified header on the same response read 3 August 2026, sixteen days before. Those two fields are 942 days apart and they are describing the same 1,580 bytes. One of them is wrong about something, and nothing in the file says which.

The pattern held across the set. Five of the thirteen files agreed to within a day: Google's four regenerated files, whose creationTime and Last-Modified were stamped within thirty seconds of each other, and OpenAI's chatgpt-user.json, where the two matched to the second. Six disagreed, by 13 days on Apple's applebot.json, 99 on OpenAI's adsbot.json, 169 on Google's user-triggered-agents.json, 216 on searchbot.json, 280 on gptbot.json and 942 on bingbot.json. The remaining two, both Perplexity's, sent no Last-Modified header at all, and no ETag either, so there is no cheap conditional request that can ask them whether anything moved. Perplexity's bot documentation names the ranges without naming a refresh mechanism.

Neither field is doing the job a consumer of these files needs. creationTime is content: it is generated when the vendor's pipeline writes the file, and Google's user-triggered-agents.json shows it can sit still for months while the object is re-served. Last-Modified is transport, set by whatever handed you the bytes. Google's page on verifying requests from its crawlers and fetchers, last updated 2026-03-20 UTC, describes matching an address against these lists as the automatic alternative to a reverse DNS lookup, and says nothing about how often to refetch them. That silence is the norm rather than Google's alone. A related timing problem, where a published directive and the crawler's copy of it drift apart, runs through crawl delay can unverify a crawler.

A Last-Modified header can move when the ranges have not

The clearest case in the set is OpenAI's smallest file. The OAI-AdsBot range file holds two prefixes covering 256 addresses and carries a creationTime of 2026-05-12, ninety-nine days before we read it. Its Last-Modified header read 19 August 2026 at 17:24:01 GMT, which is to say roughly an hour before we asked for it.

The rest of the response explains why, and the arithmetic is worth doing because the conclusion generalises. Alongside that header the response carried x-vercel-cache with the value HIT and an age of 5,021 seconds, against a Date of 19 August 2026 at 18:47:43 GMT. Subtract the age from the Date and you get 17:24:02, which is the moment that edge cache entry was filled. The Last-Modified header read 17:24:01. The two are one second apart, so what that header is reporting is when a CDN edge fetched the object, not when OpenAI last touched the ranges. The two prefixes inside the file are the same two we recorded on 29 July 2026.

That is not a criticism of the CDN, which is behaving normally. It is a statement about what the header can support. The MDN reference for Last-Modified describes it as the date and time at which the origin server believes the resource was last modified, and notes it is used as a validator and is less accurate than an ETag. On a file served through an edge cache, the origin server in that sentence may be the edge. A refetch policy built on it will refetch a file that has not changed, which is merely wasteful, and it will also report a change that did not happen, which is worse, because it teaches whoever reads the alert to ignore the next one.

The direction that actually costs something is the opposite one, and Google's user-triggered-agents.json shows it. That file was re-served on the day we read it while its creationTime stayed at 2026-03-03. A checker keyed on creationTime would have concluded the file had not moved in 169 days. A checker keyed on the header would have concluded it moved today. Both are reading the file correctly and reaching different answers, and the only way to know which is right is to download the bytes and compare them to the copy you already had. The same reasoning about the lag between publishing a rule and a crawler acting on it is set out in when a robots.txt edit reaches a crawler.

The one file we can show changed, and it got smaller

Everything above concerns what a file says about itself. There is one file in the set where we can compare against our own earlier reading rather than against a header, because the figures were published here at the time. On 29 July 2026 we opened OpenAI's four range files and recorded the counts in four OpenAI crawler IP lists, opened and counted. Three of the four are byte-for-byte the same story three weeks later: gptbot.json still holds 21 prefixes covering 3,472 addresses at creationTime 2025-10-30, searchbot.json still holds 35 covering 2,496 at 2026-01-02, and adsbot.json still holds 2 covering 256 at 2026-05-12.

The fourth moved, and it moved down. On 29 July the chatgpt-user.json file carried 286 prefixes covering 37,328 IPv4 addresses at a creationTime of 2026-07-23. On 19 August it carried 204 prefixes covering 36,016 addresses at a creationTime of 2026-08-14. That is 82 fewer prefixes and 1,312 fewer addresses in twenty-one days.

The direction matters more than the size. Most writing about stale allowlists assumes the failure is under-permission: the vendor adds capacity, your copy is old, and you start rejecting a crawler you meant to admit. A list that shrinks fails the other way. A site still holding the 29 July copy is admitting 1,312 addresses that OpenAI no longer publishes for ChatGPT-User, and admitting an address on the strength of a withdrawn range is precisely the case an allowlist exists to prevent. Nothing announced the change. There is no changelog for these files, no feed, and no mailing list we could find.

Two limits on that finding, both real. We compared counts, not sets, because what we published on 29 July was a count, so we cannot say how many of the 204 current prefixes are new rather than survivors, and the 82 net loss could conceal both removals and additions. And we do not know why the file shrank. Consolidating adjacent blocks into fewer, larger prefixes would reduce the prefix count while covering more addresses, but here the address count fell too, so a pure consolidation is not the explanation. What any of this means for whether a crawler can reach your pages is a separate question from address verification, and what AI visibility means keeps the two apart.

Google moved these files and renamed one, and the old path still answers

A verification list can also break without changing a single prefix, by moving. Google's announcement of the new location for its crawler IP range files, published in March 2026, moved them from a path under /search/apis/ipranges/ to one under /crawling/ipranges/, and renamed googlebot.json to common-crawlers.json in the process. Google's crawling documentation changelog is where these edits are recorded, and it is the closest thing any vendor in this set offers to a feed of changes.

We tested both paths on 19 August 2026. A request for googlebot.json at the old path returned 301 with a Location header pointing at /static/crawling/ipranges/common-crawlers.json, so a client that follows redirects still gets the file. A client that does not follow redirects gets a 301 body, and if it parses that as JSON it fails, and if it treats a parse failure as an empty list it now has an allowlist of nothing.

The sharper case is the other way round. A request for common-crawlers.json at the old /search/apis/ipranges/ path returned 404. That is the URL somebody would construct by reading the rename announcement and applying the new filename to the path they already had, and it produces a hard failure rather than a redirect. Between those two, the redirect is the kinder outcome and the 404 is the honest one; the failure mode with no warning attached is a client that swallows either and carries on with a stale copy.

The verification page now names five files rather than four: common-crawlers.json, special-crawlers.json, user-triggered-fetchers.json, user-triggered-fetchers-google.json and user-triggered-agents.json. A site that copied the old list of four and never revisited it is not checking the fifth against anything. Renaming is a recurring source of quiet breakage in this area, and the robots.txt version of the same problem is written up in renamed crawler tokens leave stale robots rules.

What a site can check instead of trusting either date

The practical answer is that neither date is a change signal, so a checker should stop asking for one. Download the file, hash the sorted set of prefixes, and compare that hash to the one you stored last time. It is a few lines, it costs one request against a small file, the thirteen coming to about 159 kilobytes in total and only Google's user-triggered-fetchers.json exceeding 40 of them at 70, and it is the only method in this post that answers the question actually being asked, which is whether the ranges you are enforcing are the ranges the vendor publishes. Store the fetch timestamp yourself rather than relying on the file to carry one.

The second answer is to prefer reverse DNS where the vendor supports it, because it does not go stale. Apple's About Applebot page, which carries a published date of 8 June 2026 and lives at support.apple.com/en-us/119829, states that traffic from Applebot is generally identified using reverse DNS in the applebot.apple.com domain, and gives matching the address against a CIDR prefix in its JSON file as the other way. The file itself is at search.developer.apple.com/applebot.json. We name both as plain text because neither host is on this site's registered outbound list. Notably, that page states no refresh interval for the file, and we checked: the words daily and refresh do not appear on it. A forward-confirmed reverse lookup asks the vendor's own DNS about one address at the moment the request arrives, so there is no copy to keep current.

Reverse DNS is not free, though, and the honest version of this advice says so. It adds a DNS round trip to the request path, it fails when the vendor's DNS is slow or unreachable, and it only works for vendors who publish the pointer records. Of the five here, Apple and Google document it, Google describing a reverse lookup as the manual method on its verification page. We found no mention of reverse DNS on OpenAI's bots page when we read it on 19 August 2026. That is why the address files exist at all, and why they cannot simply be abandoned.

The longer-term answer is cryptographic rather than topological, and it is being specified now: a crawler that signs its requests does not need an address list, because the signature travels with the request. We wrote up the current state of that work in what web bot auth actually specifies. Until it is deployed, an address check is what most sites have.

For our own part, how Lantad identifies itself sets out the token and the contact route we publish, on the principle that a scanner asking sites to be legible should be legible itself. What we grade and what we decline to grade is set out in the scoring methodology, and if the question you actually have is which crawlers your robots.txt admits rather than which addresses to trust, the robots.txt tester answers that one directly.