How to get cited by ChatGPT: 980 of 1,419 sites cleared every gate, and robots.txt closed 25

How to get cited by ChatGPT: the four gates, in the order they bind

A citation is the end of a chain and every link in it is a separate failure mode. Writing the chain out in order is most of the diagnostic work, because the conditions are not equally likely to be the one that broke and the cheapest one to check is not the one that usually fails.

The first condition is permission in robots.txt for the specific token that governs search, which is OAI-SearchBot and not GPTBot. The second is that the server actually serves the page when a crawler asks for it, which is a different system from robots.txt and frequently disagrees with it. The third is that the bytes returned contain the text, rather than a shell that a browser would fill in later. The fourth is that the page names itself, since the citation surface shows a title. Only the first of those four is a line anyone edits in a text file, which may be why it absorbs most of the advice.

The order matters for debugging rather than for truth: a site that fails the second gate fails whatever its robots.txt says, and a site that passes all four has cleared only the floor. Clearing the floor is not a prediction of a citation. It is the condition without which a citation is impossible, which is the part a site owner can actually inspect and fix. What happens above the floor, which pages get retrieved and which get quoted, is governed by systems nobody outside OpenAI can observe, and this blog has reported before that citations reached 6.8 percent of ChatGPT prompts and the visit landed on the homepage. The vocabulary for the whole exercise is collected under generative engine optimization.

What OpenAI documents: four crawlers, and one of them decides ChatGPT search

OpenAI's crawler documentation, read at source on 26 October 2026, names four. GPTBot crawls content that may be used to train its generative AI foundation models. OAI-SearchBot is used for search and surfaces websites in ChatGPT's search features. ChatGPT-User handles certain user-initiated actions such as visiting a page to answer a question, and the page states it is not used for automatic crawling and is not used to determine whether content appears in Search. OAI-AdsBot checks the safety of landing pages submitted as ads, visits only pages submitted as ads, and its data is not used to train foundation models.

Two statements on that page decide how this measurement had to be built. On opting out of search, it says that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links. So the consequence of a disallow is specific and partial rather than total erasure, and a site owner weighing the decision should read it that way. On timing, it notes that it can take about 24 hours from a site's robots.txt update for OpenAI's systems to adjust, which is the same class of lag this blog measured when it asked when a robots.txt edit reaches a crawler.

The page carries no publication or update date, which is worth stating plainly because every claim above is a reading of it on one day. It also publishes a JSON file of addresses for each crawler, which matters for the limits at the end of this post. Lantad's own registry of tokens lives in core/src/bots.ts and holds 15 entries across nine vendors, three of them OpenAI's: GPTBot, OAI-SearchBot and ChatGPT-User. OAI-AdsBot is documented by OpenAI and is not in that registry, so this product does not model it, a gap this blog has already published in its finding that 4 of 1,014 robots.txt files named OAI-AdsBot.

Gate one: a robots.txt rule closed 25 of 1,419 sites

1,115 of the 1,419 hostnames returned a robots.txt response in the 2xx range that parsed, 1,107 of them at HTTP 200 and 8 at 202. Evaluating the site root for OAI-SearchBot across those 1,115 files returned a disallow on 25 and an allow on 1,090. 20 of the 25 were closed by a group that named the token; the other 5 were closed only because a wildcard group disallowed the whole site, so no rule on those sites mentions OpenAI at all.

The 25 are mostly news. cnn.com, nbcnews.com, cnbc.com, thehill.com, variety.com, hollywoodreporter.com, scientificamerican.com, dr.dk, france24.com, nzz.ch, nzherald.co.nz, stuff.co.nz, theglobeandmail.com, eltiempo.com, eluniversal.com.mx, news24.com, rappler.com and qcitymetro.com all refused it, alongside amazon.com, congress.gov, wa.gov, amsterdam.nl, helsinki.fi, scielo.org and gmarket.co.kr. The mechanism on cnn.com is worth describing because it is representative: its file carries 84 User-agent lines, and OAI-SearchBot appears in a group of 83 tokens whose single rule is Disallow: / on line 97. The site did not write a rule about OpenAI. It wrote one rule about a long list that OpenAI is on.

Only 83 of the 1,115 parseable files mention OAI-SearchBot at all, and 63 of those 83 mention it and allow it anyway, usually by naming it in a group that permits crawling. So the token that actually governs ChatGPT search appearance is absent from 1,032 of these files, and the question of whether ChatGPT may index the site is being answered by a wildcard group written for something else. That is consistent with the earlier finding that 16 of 140 robots.txt files named OAI-SearchBot on a smaller sample.

The split between training and search is the clearest intentional signal in the data. 91 of the 1,115 files disallowed GPTBot at the root and 67 of those 91 left OAI-SearchBot allowed, so two thirds of the sites refusing the training crawler kept the search crawler. 44 disallowed ChatGPT-User, the token OpenAI says may not be bound by robots.txt at all because the fetch is user-initiated. Exactly one site, gmarket.co.kr, refused OAI-SearchBot while allowing GPTBot, which is the inverse of the usual choice. You can check your own file against each token with the robots.txt tester.

Gate two: 217 sites allowed the crawler in the file and refused it at the door

Of the 1,309 hostnames whose robots.txt did not refuse OAI-SearchBot, 1,057 answered the home page with HTTP 200. 252 did not, and 217 of those 252 answered 403 Forbidden. The remaining 35 spread across 202, 429, 404, 503, 406, 400, 401, 498, 500 and three transport failures.

The 217 divide into two groups that mean different things. 62 of them served a robots.txt that parsed cleanly and explicitly permitted OAI-SearchBot at the root, then refused the home page. For those sites the published rule file and the deployed bot rules disagree outright, and the rule file is the one a reader would check. The other 155 refused the robots.txt request itself with a 403 and then refused the home page the same way. That second group is a quirk of the standard worth naming: RFC 9309 section 2.3.1.3 states that when the server responds with status codes indicating the resource is unavailable, which it defines as the 400 to 499 range, then the crawler may access any resources on the server. A 403 on robots.txt therefore reads, in the specification, as no restrictions at all, while the same server is refusing every page. 155 sites in this corpus were in that state on 26 October 2026.

This is the same shape as earlier findings on this blog, now at corpus scale: 79 of 115 sites refused a crawler their robots.txt allowed, and 103 of 1,089 sites served an unknown bot and refused GPTBot. The reason the two layers disagree is that they are two layers: robots.txt is a file a publisher writes, and the 403 comes from edge bot rules a platform ships, which is the argument of two layers decide whether AI can read your site.

The refusals concentrate by sector. Of the 220 hosts across the whole corpus that answered the home page with 403, 61 were news, 47 ecommerce, 29 travel, 18 finance, 18 healthcare, 17 education and 17 government, against 9 in the SaaS stratum and single figures or zero across every platform stratum. News sites both write the rules and shut the door, which is one subject appearing twice rather than two findings.

One more measurement on that door: 85 of the 1,419 never produced a robots.txt verdict at all, 40 because the file returned a 5xx, which RFC 9309 section 2.3.1.4 says leaves the file undefined so the crawler must assume complete disallow, and 45 because the request failed in transport, 24 timing out at 15 seconds and 21 failing to connect. 39 of the 40 were a 503 and one, iisc.ac.in, a 502. Those 85 are counted as not cleared because the verdict is unknown, not because anybody refused anything, and the difference between a 404 and a 503 on that file is the subject of a 404 on robots.txt allows every crawler, a 503 blocks them all.

Gates three and four: 72 sites answered with no words, and 5 with no title

1,057 sites answered the home page with HTTP 200. Running extractFromHtml over the returned bytes gave 985 with at least one extractable word and 72 without: 71 returned HTTP 200 and no extractable words at all, and one, trendyol.com, returned 200 with an empty content type header so nothing was extracted from it. A page with no extractable words in the served HTML is not an empty page in a browser. It is a page whose text arrives later, from JavaScript the crawler did not run, which is the condition this site calls a prose parity failure and which it has measured repeatedly, most directly in finding that 17 of 380 home pages sent an AI crawler zero words.

Whether that matters for ChatGPT specifically is a question OpenAI's documentation does not answer: the bots page makes no statement about whether any of its crawlers execute JavaScript. That silence is general rather than particular to OpenAI, and this blog counted it: 2 of 9 operators say either way. So the honest form of gate three is not that ChatGPT cannot read these 72 pages. It is that the text is not in the bytes, and no vendor documentation says it will be fetched by a second mechanism.

The 72 cluster where client-side rendering clusters: 12 in ecommerce, 10 in the no-code stratum, 10 in travel, 8 in education, 7 in healthcare, 7 in finance and 6 among the single-page-app startups, against 2 in the Webflow stratum, 2 in SaaS and one each in the Wix and Squarespace stratum, static documentation, local media and news. Ten of the forty-two no-code sites in this corpus returned a 200 with nothing in it, which is the finding 10 of 42 no code home pages served a crawler zero words reported on the same stratum.

The fourth gate removed five more. calltree.ai, sciencespo.fr, contpaqi.com, gymshark.com and hoshinoresorts.com each returned words with no title element the extractor could find. A missing title is a small defect with a specific cost here, because the title is the string a citation chip displays, and it is the cheapest of the four gates to fix. Across the 980 that cleared everything the median extractable word count was 606, the minimum 2 and the maximum 10,843, and 91 of the 980 carried fewer than 100 words, so clearing the gate is not the same as having something to quote.

Which kinds of site cleared every gate, and which did not

The pass rate across the whole corpus was 980 of 1,419, or 69.1 percent, and that single figure hides the only part of this measurement that is actionable. Split by stratum, the rate runs from 34.4 percent to 100 percent, and it sorts almost perfectly by how the site was built and who runs it.

The hosted site builders are at the top. All 34 Shopify storefronts cleared every gate. So did 53 of 54 Wix and Squarespace sites, 30 of 31 static documentation sites, 40 of 42 Webflow sites, 37 of 39 SaaS marketing sites and 30 of 32 Framer sites. The reason is unglamorous: those platforms serve server-rendered HTML from their own edge and do not ship aggressive bot rules on the default plan, so both the door and the words look after themselves. That is the same conclusion the per-stack guides reach, and the detail per platform is in the crawlability study.

The bottom is news at 44 of 128, ecommerce at 51 of 130 and travel at 67 of 130. News fails for a reason that is partly a decision: 17 of its 128 sites refused OAI-SearchBot by rule, which is a publisher exercising the opt-out OpenAI documents, and 62 more refused at the door, with 4 producing no robots.txt verdict and 1 answering with nothing to read. Ecommerce fails for a different reason, splitting its 79 losses across 54 at the door, 13 on words or a title, 10 with no robots.txt verdict and 2 by rule. A site owner reading this should take the stratum as a prior and then check their own site, since the useful question is which of the four gates is theirs, not what their sector averages. What GPTBot sees answers that for one page.

Worth stating against the product's own interest: for 980 of these 1,419 sites the floor is already cleared, and for them a scanner has nothing to fix at this layer. The measurement that matters there is a different one, about whether the content gets retrieved and named, which is what AI visibility tracking tries to observe and which no measurement on this page touches.

What this measurement does not show

The largest limit is the user agent, and it cuts directly against the headline. Every request was sent as LantadBot/1.0, a declared crawler on nobody's allowlist, and not as OAI-SearchBot. A site that refused LantadBot may well admit OpenAI's crawlers, because OpenAI publishes a JSON file of addresses for each of its four bots and edge bot rules commonly verify a crawler by address before admitting it. So the 217 door refusals are a measurement of what an unlisted declared crawler receives, which is the general case and the one a new entrant faces, and they are not a measurement of what OAI-SearchBot receives. Lantad did not test with OpenAI's user agent string and this post does not claim ChatGPT is blocked on those sites. The reason a user agent cannot settle the question either way is the subject of a user agent is a claim, not an identity, and the address side is counted in 0 of 2,344 addresses OpenAI publishes answered a reverse DNS lookup.

Six further limits bound everything above. The unit is one page, the apex home page, so a site whose home page clears every gate may still refuse an article path, and nothing here describes a whole site. One request per host on one date cannot separate a standing policy from a transient error or a rate limit, so the 24 timeouts and 7 of the 429s may well answer on a retry. The 1,419 hostnames are two sampling frames stratified by platform and sector for the crawlability study rather than a random draw from the web, so the strata describe those lists and the smallest of them hold about thirty sites. No engine was asked anything, no answer was collected and no citation was observed, so no figure here is evidence that any of these sites is or is not cited by ChatGPT. The four gates are Lantad's reading of what OpenAI's documentation implies rather than a list OpenAI publishes, and OpenAI makes no statement about JavaScript execution, so gate three tests what is in the bytes and not what its crawlers can do. And the robots.txt evaluation is Lantad's matcher applied at path / only, so a site disallowing OAI-SearchBot on a subdirectory while allowing the root counts as allowed here.

What survives all of that is narrow and worth having. On 26 October 2026, across 1,419 real hostnames, a robots.txt rule explained 25 failures and the server door explained 252. Any guide to how to get cited by ChatGPT that stops at the file is answering the question that fewest sites get wrong. The same four-gate check applied to other assistants is on how to get cited by Claude and how to get cited by Perplexity, and the crawler vocabulary is defined under AI crawler.