Four OpenAI crawler IP lists, opened and counted

The four files, and what is actually inside them

Each file is small JSON with the same two keys: a creationTime string and an array called prefixes, where every entry is an object carrying a single ipv4Prefix in CIDR notation. There is no version field, no expiry, and no signature. Read on 29 July 2026, the four looked like this.

The GPTBot range file carried 21 prefixes covering 3,472 IPv4 addresses, with a creationTime of 2025-10-30. The OAI-SearchBot file carried 35 prefixes covering 2,496 addresses, creationTime 2026-01-02. The OAI-AdsBot file carried 2 prefixes covering 256 addresses, creationTime 2026-05-12, and both of those prefixes are a /25. The ChatGPT-User file carried 286 prefixes covering 37,328 addresses, with a creationTime of 2026-07-23T02:02:53, six days before we read it and by some distance the freshest of the four.

Two things are worth noticing before any interpretation. The first is that prefix count and address count do not move together. OAI-SearchBot publishes more prefixes than GPTBot, 35 against 21, while covering fewer addresses, 2,496 against 3,472, because the OAI-SearchBot list is made of smaller blocks. Anyone building a containment test cares about addresses; anyone eyeballing a file for a sense of scale sees prefixes, and the two answers disagree.

The second is that not one of the four files contained an IPv6 prefix. All 344 entries across all four files were IPv4. That is a fact about the published lists rather than a claim about what OpenAI's infrastructure can do, but it has a direct operational consequence: a site reached over IPv6 by something claiming to be an OpenAI crawler cannot be verified against these files at all, because there is nothing in them to match against. That is not a hypothetical for a site behind a modern CDN.

If the vocabulary here is unfamiliar, what an AI crawler is is the reference entry, and the free AI crawler reference lists the tokens Lantad evaluates.

What an OpenAI crawler IP range count does not prove

A published address range is capacity, not activity. It says where requests from a crawler may originate, and it says nothing whatever about how many requests any of them sends, how often, or to whom. ChatGPT-User publishing more than ten times GPTBot's address space is not evidence that ChatGPT-User fetches more pages than GPTBot, and we have not measured that. We have opened four files and added up the blocks in them.

There are ordinary infrastructure reasons a range can be large without the traffic being large. A fleet spread across more regions needs more blocks. A provider that allocates in small units publishes more of them. A list that has grown by accretion since 2025 can carry ranges no longer in service, because none of these files carries an expiry and nothing in the format marks a prefix as retired. The ChatGPT-User file being both the largest and the most recently regenerated is consistent with a fleet that is actually expanding, and it is equally consistent with a list that is simply maintained more actively than the other three. Nothing in the file distinguishes those.

What the files do support is precisely one operation, and it is the one they exist for: given a request that claims to be an OpenAI crawler, you can test whether its source address falls inside the published blocks for that crawler, and a request from outside them is not that crawler whatever its header says. That is a containment test against a list you refresh on a schedule, which is cheap. We argued the general case for it in a user agent is a claim, not an identity, and what is new here is only that the lists have now been opened and counted rather than described.

The honest statement of Lantad's own position has not changed since that post. Lantad does not verify crawler identity by IP today, every per-crawler figure it shows is labelled as a count of requests claiming that crawler, and the reason it is not built is ordering rather than difficulty. That labelling convention is not specific to crawlers: it is the rule applied to every AI visibility figure the product prints, because a number whose provenance goes unstated is a number a reader will over-read. Our scanner identifies itself as LantadBot and the page explaining what our bot does says so. Cloudflare's bot documentation covers the managed alternative for sites already behind it, and OpenAI is not unusual in publishing ranges at all: Perplexity's bots guide publishes them too.

OpenAI applies robots.txt to two of the four crawlers

The crawler overview opens by naming which tokens the file is for, and the sentence is narrower than the four-crawler table underneath it. It reads: "OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI." Two tokens, not four. The page then adds that the settings are independent of each other, so a site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot to keep its content out of foundation model training.

Read the per-crawler rows and the asymmetry holds. GPTBot carries a plain statement that "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models". OAI-SearchBot carries a recommendation to allow it in robots.txt, plus a consequence worth quoting because it is routinely overstated in both directions: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links". Opting out of the search crawler removes you from the answer, not from the interface.

ChatGPT-User carries the opposite kind of statement. The page says it "is not used for crawling the web in an automatic fashion", and then: "Because these actions are initiated by a user, robots.txt rules may not apply." That is the token with 286 published prefixes behind it. The page also states that ChatGPT-User is not used to determine whether content may appear in Search, and directs anyone managing search opt outs to OAI-SearchBot instead.

OAI-AdsBot says nothing about robots.txt at all. The row describes what the crawler does, that it "only visits pages submitted as ads", and that the data it collects "is not used to train generative AI foundation models". There is no sentence about disallowing it and no sentence about honouring a disallow. Third party write ups of OAI-AdsBot published this month assert both that it honours robots.txt and that OpenAI has not confirmed whether it does, which is a good reason to read the vendor page rather than a summary of it. The absence is the finding: on 29 July 2026 the documentation stated a robots.txt consequence for three of its four crawlers and none for the fourth.

None of that is unusual for the protocol. RFC 9309 specifies how a crawler that chooses to consult robots.txt should parse and apply it, and specifies nothing that compels anyone to consult it. The edge layer, not the file, is where refusal is actually enforced, which is the argument in two layers decide whether AI can read your site. One more detail from the same page is worth carrying into any before and after test: OpenAI states that for search results "it can take ~24 hours from a site's robots.txt update for our systems to adjust", so a change checked ten minutes later has been checked too early. Our free robots.txt tester evaluates your file against each token immediately, which answers what the file says rather than what OpenAI has ingested.

OpenAI sends a different user agent when it fetches robots.txt

Buried in the OAI-SearchBot row is a detail with more operational consequence than its placement suggests. OpenAI publishes two user agent strings for that crawler, not one. The ordinary string ends "compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot". The string used when fetching robots.txt has an extra segment: "compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot". The page gives the reason plainly, that the marker "helps site owners more easily distinguish requests for the robots.txt file from requests for other resources, especially when their logs do not include paths". The GPTBot row carries the same note without printing the variant string.

This is a considerate piece of engineering and it is also a trap for anything doing exact string comparison. A log filter written against the published page fetch string will not match the robots.txt fetch, so a site counting OpenAI requests by exact match undercounts by exactly the robots.txt fetches, which are the ones that tell you whether your file is being read at all. A matcher that tests for the substring OAI-SearchBot catches both. A matcher that tests for equality with a documented string catches one. If you want to see what your own page returns to a crawler user agent rather than to a browser, what GPTBot sees on your page performs exactly that fetch and shows the response.

The general failure this belongs to is that turning a user agent string into a crawler label is an ordering and matching problem with named ways to be confidently wrong, which is set out in AI crawler detection is an ordering problem. The specific lesson here is narrower and easy to act on: match on the product token, never on the whole string, because the vendor reserves the right to change everything around the token and has now documented a case where it does so within a single crawler.

It is also worth being clear about what the marker is not. It is not a robots.txt directive, not a second product token, and not something to add to your file. There is nothing to configure. It exists so that a site whose access logs omit the request path can still tell which requests were for robots.txt, and a site whose logs do include paths already had that information.

Our own copies of these strings are out of date

Reading a vendor page carefully invites the same reading of your own files, so here is what the comparison found. Lantad's crawler registry is a single array called BOT_REGISTRY in core/src/bots.ts. It holds fifteen tokens, three of them OpenAI's: GPTBot, OAI-SearchBot and ChatGPT-User. There is no OAI-AdsBot row, so no Lantad surface prints a robots.txt verdict for it.

The stored user agent strings have drifted. Our GPTBot entry records GPTBot/1.2 against the documented GPTBot/1.4. Our OAI-SearchBot entry records OAI-SearchBot/1.0, and it has drifted further than a version number: the documented string is now a full browser style user agent beginning "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)" and carrying a Chrome token, where ours is the older short form with no browser segment at all. Our ChatGPT-User entry matches the documented string exactly. The docsUrl on all three entries points at platform.openai.com/docs/bots, which returned a 301 redirect to developers.openai.com/api/docs/bots when we requested it on 29 July 2026.

The consequence is real but smaller than it first looks, and overstating it would be the same error as ignoring it. That stored string is sent as the request user agent by the differential probe in core/src/pipeline.ts, the step that fetches a page as a named crawler to see whether the site treats it differently. Since almost any site-side rule keys on the product token rather than the full string, a stale version number is unlikely to change a probe result. What it does mean is that a file whose comment describes it as the vendor's published request user agent no longer holds the vendor's published request user agent, and that is worth fixing before it matters rather than after.

The missing OAI-AdsBot row is the more interesting gap, because adding it is not a one line data change. BOT_PURPOSES in the same file has exactly three members: training, search and user_agent. Validating a landing page an advertiser submitted is none of those three. The purpose classes are load bearing rather than descriptive, since PENALIZED_PURPOSES names the classes whose disallow costs a site points in the Access sub-score, and the mapping from purpose to audience is written to be total, so adding a fourth class is a deliberate compile error until somebody decides what the report would say about it. That is the same design that stopped a Googlebot gap being papered over, described in the one crawler our scanner does not model.

Two of those four gaps are corrections and two are decisions, which is why this post reports them rather than quietly shipping a patch alongside it. The scoring rules are published on our methodology page, the principle that a number we cannot explain does not ship is in why we will not grade a page we could not measure, and if you want the platform specific version of all this, how to get cited by ChatGPT is the evergreen page underneath this dated argument.