Can ChatGPT read my website: on 4 of 29 sites the answer changed with the agent

Can ChatGPT read my website, and which ChatGPT is asking?

Three clients, three tokens, three separate decisions. OpenAI's documentation, read at source on 5 September 2026, states that GPTBot is used to crawl content that may be used in training its generative AI foundation models, and that disallowing GPTBot indicates a site's content should not be used in training. OAI-SearchBot is described as surfacing websites in search results in ChatGPT's search features, and the same page recommends allowing it in robots.txt. ChatGPT-User covers certain user actions in ChatGPT and Custom GPTs.

The third of those carries a caveat that no test result can express. The page states that ChatGPT-User is not used to crawl the web automatically and that, because these actions are initiated by a user, robots.txt rules may not apply. A tester that reports ChatGPT-User as allowed is reading the file correctly and telling you less than it looks like, because the vendor has said in its own documentation that the file may not govern that traffic in the first place.

Splitting one brand into three tokens is what makes the training decision separable from the citation decision. A publisher who wants to stay out of a training corpus but remain quotable in an answer has to write two rules that say opposite things, and get both tokens spelled right. That is a different task from the one most robots.txt files were written for, and it is the reason what to allow for ChatGPT is not a single switch.

None of this describes behaviour. A robots.txt verdict is a statement about what a file grants to a client that identifies itself with a token and chooses to honour the file. We publish the user agent this scanner sends and the addresses it sends from so that permission and behaviour can be told apart in your own logs, which is the only place they can be. When somebody else fetched as well as read, 234 of 592 sites that disallowed GPTBot served it a 200 anyway.

What 29 robots.txt files said to the three OpenAI agents

Of the 29 files, GPTBot was allowed to fetch the site root on 23 and disallowed on 6. OAI-SearchBot and ChatGPT-User were each allowed on 27 and disallowed on 2. Those numbers are worth stating in the other direction as well: the training bot was refused the front page three times as often as the search bot, on a sample where both were asking the same 29 files for the same path on the same day.

The more useful figure is how few of the files had an opinion. 21 of the 29 named none of the three tokens anywhere, so all three verdicts came from the wildcard group. Eight named at least one: GPTBot appeared in 8 files, OAI-SearchBot in 5 and ChatGPT-User in 5. A verdict that arrives through the wildcard is not a decision about ChatGPT. It is the answer a site gives every client it has never heard of, and it will keep giving that answer when the token list changes underneath it.

Two of the eight named a token and allowed it anyway. www.cloudflare.com names GPTBot with an Allow: / at line 15 and ChatGPT-User with an Allow: / at line 18, which is a deliberate yes rather than a default one. www.squarespace.com declares a GPTBot group whose rules do not match the site root, so the answer is allowed and the group still exists to carry narrower rules.

The pattern also explains why a file's size predicts nothing. www.bloomberg.com carries 98 user-agent groups in 7,048 bytes and refuses all three. medium.com carries 2 groups in 884 bytes and refuses one of the three. developer.mozilla.org, docs.python.org, react.dev, nodejs.org and kubernetes.io each declare a single group and allow everything, which is the same posture a Shopify storefront ships with by default. We found the same thing when we ran the same parser against the six sites in this repository's fixtures and not one of their 48 groups named an AI crawler either. If you want the current token list rather than a snapshot, our crawler directory carries it.

The four sites that gave the three agents different answers

Four of the 29 returned different verdicts to the three tokens, and all four have the same shape: GPTBot disallowed at the root, OAI-SearchBot allowed. Stay out of the training corpus, stay eligible for the citation. Whether the sites intended precisely that is not something a robots.txt read can establish, but it is what the files say.

www.forbes.com is the clearest instance because the two rules sit ten lines apart. In the 33,649 byte body we parsed, a group headed GPTBot carries Disallow: / at line 351, and a group headed OAI-SearchBot carries Allow: / at line 361. ChatGPT-User is not named at all, so it falls through to the wildcard and is allowed by default rather than by decision.

www.ebay.com and www.tripadvisor.com reach the same outcome by grouping. eBay heads one group with AmazonBot, meta-externalagent, GPTBot and Applebot-Extended and gives it Disallow: /, then heads a separate group with OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User whose rules are path specific and leave the root allowed. Tripadvisor does the same with a GPTBot Disallow: / at line 37. medium.com is the thinnest version: GPTBot is grouped with FacebookBot, GoogleOther and meta-externalagent under Disallow: / at line 30, and neither of the other two OpenAI tokens appears in the file.

Set against that, www.cnn.com and www.bloomberg.com refuse all three, which is the only configuration in this sample that answers the question in the title with a plain no. CNN does it with a single group listing all three tokens together, resolved at line 89. Bloomberg spreads them across its 98 groups, at lines 76, 83 and 139. Both are decisions; only one of the two is legible at a glance, and legibility matters when the person maintaining the file is not the person who wrote it. Our standing measurements keep finding that the failures which cost sites access are this plain.

Ten hostnames refused to serve us robots.txt at all

Ten of the 40 returned HTTP 403 to both of our requests for /robots.txt: www.bbc.co.uk, www.theguardian.com, www.nytimes.com, www.reuters.com, www.ft.com, www.wsj.com, stackoverflow.com, www.reddit.com, www.mayoclinic.org and www.allrecipes.com. webflow.com failed to connect on both attempts. That is eleven of forty hostnames where we have no reading of the file at all, and the honest statement of what that means is narrower than it looks.

It is a fact about our request, not about OpenAI's crawlers. We asked from a datacentre address with a user agent nobody has an allowlist entry for. A crawler operating from published address ranges, as OpenAI's four IP range files describe, is a different client arriving at the same edge and may well be treated differently. Nothing here establishes that these ten sites refuse GPTBot.

What it does establish is that the refusal is not scoped to the robots file. We asked the same ten hostnames for their homepage with the same user agent on 5 September 2026, and all ten returned 403 there as well. So for this client the file and the pages behind it were equally unreadable, and a permission question never arose.

That matters because of what the standard says about the gap. RFC 9309 states that an unavailable robots.txt imposes no restrictions on crawling, and this scanner implements exactly that: a 4xx status yields an allow-all policy with the reason recorded, while a 5xx yields an assumed disallow, because crawlers commonly treat a failing robots.txt as disallow-all until it recovers. So a 403 on robots.txt is, on paper, the most permissive answer a site can give, delivered by infrastructure whose purpose is to be restrictive. The file says everything is allowed because the file was never served. How this scanner reaches a verdict is written down for cases exactly like this one, and a degraded reading is reported as degraded rather than graded.

Permission is the first question and not the last one

An allowed verdict says a file grants access. It does not say the page held anything to read once the request succeeded, and that second half is where most of the loss actually happens.

This repository holds a measurement that puts a number on it. Two golden fixtures produce an identical composite score of 34.4 and an identical grade of F. One ships zero words in its initial HTML; the other ships all 485 of them, inside a hydration JSON payload rather than in prose. We published that as text in hydration JSON scoring the same as no text at all, and the reason it lands here is that both of those fixtures would pass a robots.txt check without difficulty. Permission was never their problem.

The same separation shows up in what a fetch returns against what a browser assembles. When we counted links across five real pages captured on 15 July 2026, JavaScript added no new crawl paths that were not already in the raw response, and of the twelve links that disappeared once the page rendered, nine were one page's translations of itself. A framework that moves prose out of the server response is making a decision about prose parity whether or not anybody framed it that way, which is the thing a Next.js site has to get right before a robots.txt rule can matter to it.

Citation is a third question again, and it is the one buyers usually mean. Similarweb's figures, which we reported when we found that ChatGPT citations are rare and land on the homepage, put citation presence in US ChatGPT prompts at roughly 6.8 percent by May 2026. Access is necessary for that and nowhere near sufficient, which is why AI visibility is measured in more than one place.

What to check on your own site, in order

Four checks, and they have to run in this order, because each one makes the next meaningful.

First, confirm the file is served. A 403 or a connection failure to the client doing the asking is not a permissive robots.txt, it is no robots.txt, and ten of the forty hostnames here were in that state for us. Ask from outside your own network, because an office address on an allowlist will show you a file that a crawler never sees.

Second, ask per token rather than per brand. Run GPTBot, OAI-SearchBot and ChatGPT-User separately against the paths you care about, and do the same for the other vendors' training and search pairs. The robots.txt tester here returns one verdict per token for this reason. If your file names none of them, you have not made a decision, you have inherited one, which is the position 21 of these 29 sites are in.

Third, read which group decided it. A verdict via the wildcard and a verdict via a named group look identical in a summary and mean opposite things about intent. Grouping is also where the errors hide: a token misspelled in a group heading falls through silently, and a declared user agent is a claim rather than an identity in the log where you might go looking for the consequence.

Fourth, look at what the allowed page actually returns. What GPTBot sees fetches the raw response rather than the rendered one, which is the version a crawler that executes no JavaScript is working from. An allowed page with its prose in a script tag is a page you have given permission to read something that is not there, and no amount of correcting the robots.txt will change that.