OAI-AdsBot: 4 of 1,014 robots.txt files named it, and one wrote a rule shaped like its job

What is OAI-AdsBot, and what does OpenAI say it does?

It is the newest of the four product tokens on OpenAI's crawler page and the only one whose job is not reading the open web. The documented sequence runs the other way round: an advertiser submits an ad, and the crawler then visits the landing page that advertiser nominated. The full user agent string the page publishes is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot, and it is one of the two rows OpenAI publishes as a full user agent string rather than as an example whose version number may change.

Its published address list is correspondingly tiny. The adsbot.json file, fetched on 13 October 2026, carries two IPv4 prefixes, 130.131.229.128/25 and 20.168.90.128/25, covering 256 addresses, and a creationTime of 2026-05-12. That is the same content and the same stamp we read when we opened and counted all four of OpenAI's crawler address files on 29 July 2026, so the file has not moved in the two and a half months since. For comparison, the same reading found 286 prefixes for ChatGPT-User.

The reason this token is worth a measurement rather than a footnote is the asymmetry in its own documentation. OAI-SearchBot's row recommends allowing it in robots.txt to appear in search results. GPTBot's row states that disallowing it indicates a site's content should not be used in training. ChatGPT-User's row states that because its actions are user-initiated, robots.txt rules may not apply. OAI-AdsBot's row contains the string robots.txt zero times. A site owner reading that page learns the purpose of all four and the robots.txt consequence of three, and this is the vendor's own page rather than a summary of it, which matters because third-party write-ups have asserted both that it honours robots.txt and that OpenAI has not said. The documentation is the thing to read, and it is still silent.

That silence is also why a disallow aimed at this token is a strange instrument. Under the robots exclusion protocol, published by the IETF in September 2022, a crawler obeys the group matching its own product token and falls back to the wildcard group only when no group names it. A rule is an instruction to a crawler that has agreed to read it. For three of OpenAI's tokens the vendor states what reading it means. For this one it does not, which puts the four files below in a different position from the 170 that name GPTBot: they are writing rules whose documented effect is unstated. Across the nine vendors in our registry this concentration is unusual, since six of them publish exactly one crawler token and therefore have nothing to be asymmetric about.

How many robots.txt files name OAI-AdsBot?

Four, out of the 1,014 that parsed. The four are youshift.com, slate.com, ebay.com and expedia.com. By contrast 170 of the same files name GPTBot, 101 name ChatGPT-User and 81 name OAI-SearchBot, and 180 name at least one of the four. That last number is the one that makes the shortfall deliberate rather than accidental: 68 files name GPTBot and OAI-SearchBot and ChatGPT-User, and do not name this one. Those 68 are not sites that ignored OpenAI. They are sites that sat down, worked through the vendor's token list, wrote three separate groups, and stopped one short.

Two files name all four. One of those is youshift.com, which writes four single-line groups each reading Allow colon slash, and the other is ebay.com, which is the only file in the corpus that treats the four as four different problems. Seventy-two files name GPTBot and none of the other three, which is the shape you get when a robots.txt was edited once, in 2023 or 2024, in response to the news story about training data.

A count of four is small enough that the population matters more than the number. These 1,014 files are 1,014 front doors from two committed sampling frames, stratified by industry and by the platform a site is built on, so they are not a sample of the web and no percentage here generalises beyond them. The figure is also sensitive to who answers: 239 hostnames replied in the 400s, 34 in the 500s and 44 never answered at all, and those two outcomes are not interchangeable, since a 4xx leaves a crawler unrestricted and a 5xx is commonly read as a site-wide disallow. What the four does sit comfortably alongside is every other per-token count we have taken on this corpus: 130 of 1,077 files named Bytespider, 63 of 1,063 named DuckAssistBot, and 1 of 1,068 named Reflectionbot within days of its documentation appearing. Token awareness in robots.txt tracks news coverage, not vendor documentation, and an ad-safety crawler generates no news. The same pattern shows up in the spelling: the 1,014 files name 2,186 distinct tokens between them, and we counted 2,209 in a comparable pass on 17 September 2026, most of them inherited from a list somebody copied.

The four files, read line by line

Only one of the four writes a rule that does any work, and the test for that is mechanical: delete the group naming the token, re-evaluate the site root with the same parser, and see whether the answer moves.

On youshift.com it does not. The file is 600 bytes and holds fifteen groups, one of which names this token and carries the single rule Allow colon slash. Remove it and the wildcard group allows the root anyway. The same is true of expedia.com, whose group bundles the token with five other advertising agents, four of them Google's and the fifth Microsoft's adidxbot, and then writes an Allow colon slash followed by six disallows covering internal tracking and typeahead endpoints. That is a thoughtful piece of filing, since it classifies the crawler by what it does rather than by who runs it, and it still changes no verdict at the root.

slate.com is the clearest case of naming without ruling. Its group heading carries 328 user-agent tokens on consecutive lines, a list that runs from a360-search to zoominfobot and takes in Ahrefs, Semrush, DataForSEO, Criteo, Taboola and several hundred more, and this token sits in it alphabetically between nostocrawlerbot and omvionleadlake. The eight rules that follow disallow /search, /comments/, /_, /css, /fonts, /media, /piano and /static. Nothing in that group was written about OpenAI's advertising crawler. It was written about scrapers in general, and the token was swept into it. That is the ordinary case across this corpus, and it is the same mechanism behind the finding that 559 of 581 blocked paths were closed by a rule that never named the crawler losing them, and behind the broader pattern that files naming an AI crawler usually rule only on the whole site.

ebay.com is the exception, and it is worth reading closely because it is the only file here that appears to have read the vendor documentation. Its group for this token holds three rules in this order: Allow colon /b/*?*mkcid=32*, Allow colon /itm/*?*mkcid=32*, and Disallow colon slash. Everything is shut except two URL shapes carrying an mkcid=32 query parameter, which is an eBay campaign tracking parameter, and those two paths cover its category and item pages. Evaluated with our parser, /itm/123?mkcid=32 is allowed to this token and disallowed to GPTBot, which the same file blocks at the root in a group of ten AI crawler tokens. OAI-SearchBot and ChatGPT-User get a fourth group with query-string restrictions and an open root. One file, four different answers, matched to four documented purposes. If you want to see the same arithmetic run against your own file, our robots.txt tester evaluates a path per token with the parser these figures came from, and the pattern of a specific allow sitting under a blanket disallow is exactly where the wildcard group stops deciding and the named one starts.

Who blocks it, and which rule actually did it

Thirteen of the 1,014 files disallow this token at the site root. Twelve of the thirteen do it through the wildcard group, the one headed User-agent colon star, and in all twelve the deciding rule is a flat Disallow colon slash that closes the site to every crawler not named elsewhere in the file. Those twelve are coralvilleanimalhospital.com, botcity.dev, amsterdam.nl, wa.gov, helsinki.fi, sciencedirect.com, scielo.org, eluniversal.com.mx, theregister.com, sap.com, gmarket.co.kr and instacart.com. None of them decided anything about OpenAI's advertising crawler. They closed the front door and this token was behind it with everyone else.

The thirteenth is ebay.com, whose disallow is deliberate and scoped. So the honest summary of the blocking picture is one file in 1,014 that chose, and twelve that inherited.

The cross-tabulation against GPTBot is the part worth acting on. Of the 81 files that disallow GPTBot at the root, 70 allow this token, 56 allow OAI-SearchBot and 39 allow ChatGPT-User. A site that believes it has blocked OpenAI has usually blocked the training token and left the other three open, which is the same split we found when 38 files refused GPTBot at the root and let ChatGPT-User in. Whether that is the right outcome depends on what the site owner wanted, and for two of the three remaining tokens there is a clear answer in the vendor's own documentation: blocking OAI-SearchBot costs the site its appearance in ChatGPT search answers, which is the opposite of what most people blocking GPTBot say they are trying to do.

For this token the same question has no documented answer, and that is the practical point of the whole measurement. If your organisation does not advertise on ChatGPT, the crawler has, by OpenAI's own description, no page of yours to visit, and a rule for it changes nothing you can observe. If your organisation does advertise there, the pages it visits are pages your own marketing team submitted, and a blanket disallow is a rule aimed at your own ad operation. Either way the rule is not a visibility control, which makes it unlike almost everything else in the file. One thing to check before writing any of this is simply whether your rules still address tokens that exist: a renamed token leaves a group matching nothing, and silent staleness is the normal failure mode of these files.

What this measurement does not say

It does not say that any crawler obeyed anything. Everything above is a reading of text files, parsed with the scanner's own parser and evaluated for one path, the site root. We sent no request as OpenAI and observed no request from OpenAI, so nothing here is evidence about this crawler's behaviour, its volume, or whether it reads robots.txt at all. OpenAI's page does not claim it does, and we are not filling that gap with an inference.

It does not say a rule is pointless because it changed no verdict today. A group that merely agrees with the wildcard still records an intention, and it starts deciding the moment the wildcard changes. The test we ran answers a narrower question: on 13 October 2026, for the site root, did deleting the group move the answer. On three of four files it did not.

It does not measure whether these sites actually serve the crawler. Permission in robots.txt is the cheapest of the gates and the least predictive: we have measured 79 of 115 hostnames refusing a crawler their own robots.txt permits, and a published address list plus a user agent string is weaker identification than it looks, since a user agent is a claim rather than an identity and none of the 2,344 OpenAI addresses we queried answered a reverse DNS lookup. A rule naming a token binds whoever chooses to send that token.

And it does not generalise. One URL per hostname, one date, two committed frames, 1,014 files. The four and the thirteen are counts within that population and nothing more. If you want the whole method rather than the headline, our methodology page states what a scan measures and what it refuses to grade, and the crawler registry page lists the tokens we evaluate, which still does not include this one: our registry classifies a crawler as training, search or user agent, and validating an advertiser's landing page is none of the three. That missing row is a known gap rather than an oversight, and it is the kind of thing worth knowing about a tool before you trust its AI visibility verdict.