How does ChatGPT find my website? A Cloudflare pilot on about 1 percent of its network indexed hundreds of millions of pages

How does ChatGPT find my website?

Four routes, and only one of them is new. The first is the ordinary crawl: OpenAI's crawler documentation names OAI-SearchBot as the agent used to surface websites in search results in ChatGPT's search features, and recommends allowing it in robots.txt. The second is a sitemap, which hands a crawler an inventory rather than making it walk a link graph. The third is a user-initiated fetch, which the same documentation assigns to ChatGPT-User and describes as not crawling automatically, so it reaches a page because a person asked for that page and not because any index held it.

The fourth is the one Cloudflare's post adds. Rather than waiting to find an address by following a link to it, OpenAI's search crawler receives signals from Cloudflare's network about which addresses exist and which have changed, and then, in the post's words, independently evaluates and ranks how those pages appear based on how relevant and helpful they are. The ranking stays with OpenAI. What moves to the network is the knowledge that a page is there and worth a look.

That is a change of discovery mechanism, not of anything downstream, and it is worth being precise about the difference because the two get conflated constantly. A protocol that tells a crawler sooner that a page exists does not change what the crawler receives when it arrives, which is the argument this site made about IndexNow in August 2026 and which holds here unchanged. If the page answers with a bot challenge, or with an empty shell that needs JavaScript, a faster route to it produces a faster arrival at the same wall. Everything in how to get cited by ChatGPT that was true before this pilot is still true after it.

What the pilot reports, and what it does not claim

Three figures are stated plainly and all three are counts of scale rather than of outcome. The pilot launched in July 2026. It ran on a participating cohort of about 1 percent of Cloudflare's total network. It enabled OpenAI to discover and index hundreds of millions of new web pages. That is the result, and it is a real one: an index that did not hold those addresses now holds them.

What the post does not say is worth as much to a site owner as what it does. There is no referral figure, no citation figure and no traffic figure, because the post lists measuring what happens after content is discovered among the things the next phase will do. Cloudflare also writes that finding new pages can initially increase crawling as those pages are evaluated and indexed, which is the honest direction of travel for a site that joins: more requests first, and a benefit that has not yet been measured. A reader deciding whether this matters to their business should notice that the company announcing it has not claimed the business outcome.

Two limits are stated as limits rather than buried. A signal about a page does not grant permission to access it. And the research uses metadata about the web for search and discovery rather than sharing page content for model training. Both sentences are doing real work: they separate the question of whether a crawler knows about a page from the question of whether it may fetch it, and they keep this pilot out of the training argument entirely.

None of that tells you whether being found converts into being named in an answer. On that question this site's position has not moved: Similarweb figures reported here on 3 August 2026 put citations in US ChatGPT prompts at roughly 6.8 percent by May 2026, rising from about 1.6 percent a year earlier, with the homepage taking about 60 percent of ChatGPT referrals in the week after 7 May 2026. Discovery feeds a funnel whose later stages are narrow and measured elsewhere.

The pages links never reached

The premise of the pilot is that useful pages exist which few other sites link to, and Cloudflare gives the kind of example that makes it concrete: the manual for one washing machine model, the current passport fees, a restaurant's latest menu. Those pages are specific, they answer a real question, and nothing much points at them.

That gap is measurable from the outside and this site has measured it. On 29 August 2026, six pages captured in July exposed 985 unique internal paths between them, while the sitemaps those same six sites declared listed 200,712 URLs. The comparison is not exact, because 985 paths came from six captured pages rather than a full crawl, and the post that reports it says so. But two orders of magnitude is not a rounding difference. It is the shape of the problem: what a crawler reaches by walking from a home page and what a site actually publishes are different inventories, and the declared inventory is far larger.

The walk is worse than that figure implies, because on some sites it cannot start. Of 1,091 home pages that answered with HTML on 21 September 2026, 84 carried no internal link a crawler could follow at all, and 55 of those 84 contained no anchor element anywhere in the document. For those sites a link-following crawler has one page and no second step.

The declared route is in better shape, though not uniformly. Of 1,016 robots.txt files that parsed on 17 September 2026, 839 declared a sitemap. Fetching the declared maps on 1 October 2026 found 827 of 871 delivering a body, while 38 answered a non-200 status, one timed out, and five named a value that is not an absolute URL. So most sites do hand over an inventory, and a few dozen point at an address that answers with nothing. A network-level signal routes around that failure for whoever is behind the right network, and leaves it exactly where it was for everybody else.

Three patterns that waste a crawl, and what we measured on our own corpus

Cloudflare names three things it learned make a nonexistent page look interesting, and all three are measurable without any access to a crawler's logs. Bots repeatedly ask for files that are not there. Some sites redirect almost any address to their home page. Other addresses lead to copies of the same content. Recognising those patterns, the post says, cuts wasted work before a crawler fetches the page.

Each of the three has been counted here, on this repository's committed corpus of 1,419 hostnames, on a stated date. Asking every host for a path that cannot exist on 20 September 2026 produced 1,094 answers, of which 936 returned 404 and 125 returned HTTP 200 for a page that does not exist. On a second nonexistent path of a different shape, 111 of those 125 did it again, which rules out a one-off. A crawler asking those 125 sites whether an address is real gets yes every time.

Redirection is the ordinary case rather than the exception. With automatic redirect following switched off on 11 October 2026 and every Location header walked by hand, 878 of the 1,322 home pages that answered were reached only through at least one redirect. Among those, 1,031 redirect responses carried five different status codes and 62 chains mixed a permanent code with a temporary one, so a crawler deciding whether an address is canonical gets inconsistent guidance from the status line itself.

Duplication is close to universal at the apex. Of 313 home pages that answer at their own apex, asked for four more URL forms of the same page on 15 October 2026, 305 served the same home page at two or more live URLs, and 45 of those 305 declared no canonical at all. The wider canonical picture from 30 September 2026: of 1,069 home pages, 906 declared a canonical link element and 163 declared none, while 63 of the 904 with a usable href named a different address, among them a staging host and an address on localhost.

The point of listing them together is not that sites are badly built. It is that the three signals Cloudflare says it now filters on are real and common, so filtering them is worth doing, and a site which exhibits all three is handing every crawler the same confusion whether or not any network is filtering on its behalf.

A change signal needs something to validate against

The second half of the pilot is freshness. Cloudflare writes that keeping up by fetching every page more often would be expensive for everyone, that a crawler which repeatedly downloads an unchanged page adds load without bringing any benefit, and that change signals tell a crawler when a page is worth another visit and when it is not.

A signal delivered at the network layer is one way to answer that. The protocol has had another way for thirty years, and most sites in this corpus leave it on the table. Asking 1,419 hostnames for their home page twice on 6 October 2026 and then a third time with a conditional request carrying whatever validator the first response supplied, 1,101 answered with HTML, and 391 of those sent neither an ETag nor a Last-Modified header, so a returning crawler had nothing to ask against. Of the 710 that did offer a validator, 514 answered HTTP 304 Not Modified when asked properly, which is the mechanism working exactly as designed.

Those two numbers are the honest frame for the freshness claim. For 514 sites the question of whether a page changed already had a cheap answer that any crawler could have obtained with one request, and a network-level change signal replaces a mechanism that was working. For the 391 that offered no validator, there was no cheap answer at all, and a signal from the network is a genuine addition rather than a substitute. Which group a site is in is a one-request check, and it is the sort of thing what GPTBot sees exists to show.

It is worth being careful about what any of this proves regarding crawler behaviour. Every figure above describes what a server sent to a scanner identifying itself honestly as LantadBot. None of it observes GPTBot, OAI-SearchBot or any vendor's crawler doing anything, and this site does not impersonate a vendor user agent to find out.

Discovery is not permission, and not readability

The sentence in Cloudflare's post that matters most to a site owner is the plainest one: a signal about a page does not grant permission to access it. Permission still lives in robots.txt, and that layer is in the state it has been in all year.

Of 1,025 robots.txt files that parsed from this corpus on 28 September 2026, 172 named GPTBot and only 80 named OAI-SearchBot, the token OpenAI ties to appearing in ChatGPT search answers. On 38 of those same files a rule refuses GPTBot at the root while leaving ChatGPT-User allowed. A narrower sample read the same way: of 140 files that parsed from 170 hostnames on 8 September 2026, 25 named GPTBot and 16 named OAI-SearchBot. OpenAI states that each setting is independent of the others, so a site that blocked training and expected search to follow got neither the block it wanted nor the exposure it assumed, and the same documentation notes it can take about 24 hours from a robots.txt edit for OpenAI's systems to adjust. Faster discovery does not shorten that lag. A robots.txt tester answers the per-crawler question in one request, which is worth doing before concluding anything from a traffic graph.

Then there is the layer above robots.txt, which answers first and often disagrees with it. On 26 September 2026, asking 1,089 hostnames for their home page while changing nothing but the user agent string found 103 that served an unknown bot and then refused the GPTBot string, and on 76 of those 103 the site's own robots.txt permits GPTBot at the root. A page discovered through a network signal still meets that edge rule on arrival.

Compliance is the open end of all of this, and Cloudflare moved on it the following day. Its post of 16 October 2026 introduces a system to evaluate whether verified bots keep honouring site preferences, scoring whether a bot's traffic is consistent with published robots.txt preferences over time and whether it slows after an HTTP 429, and turning that into what it calls an explainable trust band. The post says the system is being used internally for now, with public scores on Radar planned, so there is no published figure to quote. That restraint is appropriate: a controlled study of ten consumer AI assistants, posted to arXiv on 16 July 2026, found six of the ten never requested robots.txt in any of 200 retrieval trials, and four of those six answered correctly from pages the file disallowed. Better discovery and reliable obedience are separate problems, and only one of them got news this week.