Can GPTBot read your robots.txt? 92 of 1,056 sites refused it the file

What 1,419 sites were asked, three times each

The corpus is the two committed seed frames this repository scans from, 392 hostnames stratified by publishing platform and 1,027 stratified by industry sector, 1,419 distinct hosts between them. Each host was asked for https://host/robots.txt three times over HTTPS, redirects followed, one attempt per request, a twenty second timeout, no JavaScript executed, from one network location. 4,257 requests in total, run within a few minutes of each other so that a change at the origin between the three is unlikely rather than impossible.

The three requests differed in exactly one header. The first carried this scanner's own token, LantadBot/1.0, which is documented at our crawler page and is an ordinary declared bot. The second carried the example string OpenAI publishes for GPTBot, Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot. The third carried a current desktop Chrome string. Nothing else changed: same client, same address, same TLS stack, same second.

1,388 of the 1,419 hosts answered all three requests without a transport failure. 1,102 answered the browser string with HTTP 200, and 1,056 of those returned a body that parses as robots.txt rather than as HTML. Those 1,056 are the base for every figure below, and the 46 we dropped matter: a site that answers a browser with a 200 carrying an HTML challenge page has not served anyone a robots.txt, and counting it as a baseline would have inflated the result. The methodology page describes the same discipline applied to scans.

That base is deliberately conservative in the other direction too. The 283 hostnames whose file never reached this client at all under any string are absent from every number here, and defensive sites are over-represented among them, so the rates below describe sites that answer an outside request rather than the web.

Can GPTBot read my robots.txt, and would you ever find out?

92 of the 1,056 returned a non-200 status to the GPTBot string. 69 of the 92 answered 403 Forbidden, 11 answered 503, and the remaining twelve spread across seven other codes including one 402 Payment Required and one 429. These are not obscure hosts. The list includes canada.ca, nyc.gov, un.org, cornell.edu, nature.com, cnbc.com, nbcnews.com, salesforce.com, nike.com, homedepot.com, marriott.com and blackrock.com.

The interesting control is the other bot. 63 of those same 92 hostnames returned HTTP 200 and the site's real file to the LantadBot token, in the same sweep, from the same address. A further 33 of the 1,056 refused LantadBot as well, and 31 of those refused both. So the population splits cleanly: a small group refuses declared bots generally, and a larger group refuses this particular product token while handing the file to another one without complaint. Whatever is making that decision is reading the user agent string, not the address, and the string is the only thing that changed.

That is the part worth sitting with, because it breaks the feedback loop that site owners rely on. A user agent is a claim rather than an identity, which is why bot defence exists at all, and why verifying a crawler costs a lookup against published addresses. But a defence that keys on the string alone cannot tell an impostor from the real crawler, so it returns the same 403 to both. The owner opens the file in a browser, sees the rule, and has no way to learn that the crawler the rule was written for never received it. Nothing appears in a dashboard. The only place it shows up is an access log, filtered by user agent, which is not where anyone looks.

This is also the reason the robots.txt tester on this site fetches the file rather than accepting a paste. A rule evaluated against a file you supplied answers a different question from a rule evaluated against the file the server actually hands out.

Why a 403 and a 503 point a crawler in opposite directions

A refusal is not one outcome. RFC 9309, the Robots Exclusion Protocol, published by the IETF in September 2022, splits the failures into two classes with opposite consequences, and the split falls on the first digit of the status code.

Section 2.3.1.3 defines the unavailable status: "Unavailable" means the crawler tries to fetch the robots.txt file and the server responds with status codes indicating that the resource in question is unavailable, and it names the 400 to 499 range as the HTTP example. The instruction that follows is one sentence long: the crawler MAY access any resources on the server. Section 2.3.1.4 defines the unreachable status, names the 500 to 599 range, and says the file is undefined and the crawler MUST assume complete disallow.

Apply that to the 92. 78 landed in the 400 range, which under the specification is the site telling the crawler that no rules exist and everything is open. 14 landed in the 500 range, which is the site telling the crawler to treat the whole host as closed. The same defensive reflex, expressed through a different status code, produces the opposite policy, and in 78 cases out of 92 it produces the permissive one. This site has written before about how a 404 and a 503 on robots.txt are opposites; what is new here is that the choice between them is being made by a bot defence layer rather than by anyone who thought about crawler policy.

There is a second edge in section 2.4. Crawlers SHOULD NOT use a cached robots.txt for more than 24 hours, unless the robots.txt file is unreachable, which means a 5xx can also freeze whatever the crawler last managed to fetch. A site that switched from serving its file to refusing it may be governed for some time by a copy it no longer publishes, and a robots.txt edit already takes time to reach a crawler before any of this applies.

None of this describes what GPTBot did. It describes what the specification tells any conforming crawler to do with the response these servers produced, which is the only checkable statement available from outside. Whether OpenAI's implementation follows the letter of section 2.3.1.3 on a 403 is not something an outside measurement can see, and we did not see it.

Three publishers answered the GPTBot string with a different file

964 of the 1,056 answered the GPTBot string with HTTP 200. Seven of those 964 returned bytes that differ from what the browser string received, and six of the seven also differ from what LantadBot received a moment earlier. Three of those 964 returned a 200 carrying an HTML page rather than robots.txt at all, being khanacademy.org, inria.fr and schwab.com, whose response opens with a Charles Schwab error page apologising that it was unable to complete the request.

Three of the seven are worth reading in full, because they are the same file. Forbes, The Hollywood Reporter and Variety each returned a byte identical 597 byte document to the GPTBot string, identical down to the hash across three separately operated publishers. It disallows 23 named search and SEO crawler tokens, among them Googlebot, bingbot, Applebot, AhrefsBot and SemrushBot, and then closes with a wildcard group reading User-agent: * followed by Allow: /. GPTBot matches no named group in that file, so it falls to the wildcard, and the wildcard says everything is open.

Now read what the browser string received from the same three hosts the same second. Forbes returned 33,649 bytes naming 11 of the 15 AI crawler tokens this scanner evaluates. Variety returned 1,283 bytes naming 13. The Hollywood Reporter returned 1,859 bytes across 116 lines naming 13, and at lines 57 to 59 of that file sits a group headed User-agent: GPTBot, carrying Allow: /lifestyle/shopping/ and then Disallow: /. Somebody wrote that rule deliberately: it opens one commercial section to GPTBot and closes the rest of the site. It is in the copy a browser is handed and it is not in the copy the GPTBot string is handed.

We are not in a position to say why, and the honest reading is that this is a configuration artefact rather than a strategy. A 597 byte file shared byte for byte across three publishers points at a shared layer sitting in front of all three rather than at three editorial decisions. But the effect on the file's meaning does not depend on the intent. The 11 to 13 AI crawler groups these publishers wrote, which is far more care than the 82 of 718 files that disallow GPTBot at all, are absent from the version served under that token. This is a different failure from the wildcard group deciding 559 of 581 blocked pages and a different failure again from a subdomain disagreeing with its own apex. In those cases the rule reaches the crawler and does something other than what the author expected. Here the rule does not reach it.

One case runs the other way and is worth naming for balance. theregister.com served the real file, an explicit default deny policy dated 18 June 2026, to both bot strings, and served the browser string an HTML page titled "Are we human?". On that host the crawler got the rules and the reader got the challenge.

Which sectors refused, and which platforms never did

The 92 are not spread evenly, and the shape of the distribution says more about who runs the infrastructure than about who holds an opinion on AI crawling. Education leads at 14 of 102 hostnames, with ecommerce also at 14 from a smaller base of 83, then government at 11 of 76, news at 10 of 65, finance at 10 of 92, healthcare at 9 of 89 and travel at 9 of 80. Software as a service sits lowest of the large strata at 5 of 122.

The cleaner division is between the two seed frames. The industry frame, which samples large organisations by sector, produced 82 refusals from 709 baseline hostnames. The platform frame, which samples sites by how they are built, produced 10 from 347, and all ten came from its two self hosted strata, local media sites and small business WordPress installations. Every one of the 282 hostnames across the eight hosted platform strata, covering Framer, Webflow, Wix, Squarespace, Bubble, Shopify storefronts, static documentation sites and marketing sites for software companies, returned identical bytes to all three user agent strings.

That is a useful negative result and it points at the cause. A site on a managed platform does not have a rule layer of its own in front of its robots.txt, so the file it publishes is the file everybody gets. Nothing in the Shopify guide or any of the other stack guides needs to change because of this measurement. The exposure belongs to organisations large enough to run bot management in front of the origin, which is the same population that writes the most considered crawler policy, and that is the uncomfortable pairing: the more seriously a site takes crawler policy, the more infrastructure is likely to sit between that policy and the crawler it addresses.

It is worth being precise about how narrow this finding is against the wider crawlability study. We measured one path, /robots.txt, on the apex host, once per user agent, from one location. We did not measure whether these same sites refuse the home page, which a separate sweep found 115 of 700 doing, and a site that serves robots.txt happily may still refuse everything else. The two failures compound rather than substitute.

If you want the practical version: the check is one command and you already have it. Request your own /robots.txt with a crawler user agent string and compare the bytes to what your browser gets. What GPTBot sees does the fetch for you, and the AI crawler reference lists the tokens worth trying. If the two differ, the policy you think you published is not the policy in force, and no amount of work on AI visibility or on generative engine optimisation reaches past that.

What this does not show

The largest limit is the one that has to be stated first. These requests carried GPTBot's product token but they did not come from OpenAI. OpenAI's crawler documentation publishes the addresses its crawlers fetch from at a JSON file on its own domain, and this client is not in it. So every 403 above is a server's answer to an unverified client presenting that string, and a site that verifies by address may well answer the real crawler differently. Nothing here counts a request from OpenAI, and nothing here is evidence that GPTBot was refused anything.

What the measurement does establish is narrower and still worth having: on 63 of these hostnames the same unverified client, from the same address, in the same sweep, got the file under one bot token and a refusal under another. That difference cannot be explained by address reputation, because the address was constant. It is a decision about the string.

Two further caveats sit on the GPTBot string itself. The example string OpenAI publishes currently names version 1.4 and says the version number may change; we sent 1.2, and a filter matching the substring GPTBot would treat both alike while a filter pinned to an exact string might not. And the same documentation states that when fetching robots.txt files OpenAI may add a robots.txt marker to the user agent string to help site owners distinguish those requests from requests for other resources. A real GPTBot robots.txt fetch therefore carries a slightly different string from the one we sent, and a rule keyed to the exact example string would see neither of them.

Three smaller limits. No access log was read on any site, so nothing here reports what any crawler received or did; every figure is a claim about a response to this client. The corpus is an editorial sampling frame of large organisations and platform cohorts rather than a random draw from the web, so these figures support statements about these 1,056 hostnames and nothing wider. And a single sweep cannot distinguish a standing policy from a transient rate limit, which is particularly relevant to the one 429 and the eleven 503s, though the 403s were stable across the re-fetches we ran on the substituted files.

The reason to publish it anyway is that it cuts under a lot of advice this site and everyone else gives. Guidance on being cited by ChatGPT, by Claude and by Perplexity all begins with the crawler reading your rules and then your pages, and so does most of the answer engine optimisation literature. If the rules file does not arrive, the first step never happens, and it fails silently in a way the owner cannot see from a browser. That is worth one command to rule out.