Do AI crawlers read RSS feeds? 86 of 762 sites declared one, and 159 served an llms.txt

Do AI crawlers read RSS feeds?

Six vendor documentation pages were opened at source on 19 September 2026: OpenAI's crawler page, Anthropic's crawler article, Perplexity's crawlers page, Google's crawler overview, Common Crawl's CCBot page and Amazon's Amazonbot page. The string atom appears on none of the six. The string RSS appears exactly once across all of them, on OpenAI's page, in a line offering the reader a feed of changes to that documentation. That is OpenAI publishing a feed for you to subscribe to. It is not a statement that GPTBot reads yours.

Google is the only one of the six vendors that documents a feed reading fetcher at all, and its page is careful about what that fetcher is for. Feedfetcher is how Google crawls RSS or Atom feeds for Google News and WebSub, the Feedfetcher documentation states, carrying a last updated date of 11 February 2026, and it adds that only podcast feeds get indexed in Google Search. It also tells a site owner that the way to stop it is to serve a 404 or a 410 to the Feedfetcher-Google user agent, which is a different control surface from the one every AI opt out uses. Feedfetcher sits among the user triggered fetchers, the group this blog has already measured as the fourteen Google agents a robots.txt wildcard does not stop.

So the honest answer to the question in the heading, on the evidence available to a person outside these companies, is that nobody has said so. That answer has a shape worth being precise about. An absence from documentation is a claim about documentation and nothing more. We read no access logs on any site, we observed no request from any crawler, and none of this rules out an internal pipeline that consumes feeds without anyone writing it down. What it does rule out is the version of the advice that says to publish a feed because a named AI crawler is documented to fetch it. No vendor page we read says that, and a site owner who acts on it is acting on a claim nobody has made.

The same standard applies in the other direction, and it is the more uncomfortable half. None of those six pages documents a crawler reading an llms.txt either. Two of them name the file only to point at their own: OpenAI's page tells a reader to see llms.txt for the complete documentation index, and Perplexity's opens by telling agents to fetch the complete documentation index at /llms.txt. Asked for /llms.txt directly on 19 September 2026, five of seven vendor documentation hosts served one, being developers.openai.com, docs.perplexity.ai, support.claude.com, docs.anthropic.com and developer.amazon.com, while developers.google.com and commoncrawl.org answered 404. So the companies building these crawlers publish the file for their own documentation and document nothing about reading yours.

The two surfaces are therefore level on vendor evidence: neither is documented as an input to any named AI crawler. What separates them is that one of them has a documented reader somewhere on the web, and it is the older one.

What 762 home pages published when both files were asked for

The method was deliberately narrow, because the interesting comparison needs both numbers to come from the same population on the same day. Every hostname in the committed industry corpus got one HTTPS request for its home page, sent as LantadBot/1.0 with the user agent documented on our bot page, redirects followed, a fifteen second timeout, one attempt, no JavaScript executed and from one network location. 762 answered with a 2xx status and an HTML content type. Each of those 762 then got one request for /llms.txt at the same origin.

Feed declarations were counted from link elements in the returned bytes, not from guessing at paths. A declaration counted when a link element carried a rel value including alternate and a type of application/rss+xml, application/atom+xml or application/feed+json, which is the rel attribute doing the job it was defined for: telling a machine that an alternate representation of this document exists and where. Counting declarations rather than probing /feed and /rss is the stricter choice and it is the right one here, because an undeclared feed at a guessable path is not a discovery surface. It is a file that happens to exist.

86 of the 762 declared at least one. Between them they declared 141 distinct feed URLs, because 34 of the 86 declared more than one, and the most on any single page was eight, at lamoncloa.gob.es. On the other side, an /llms.txt counted when the response was a 2xx, carried a text/plain or text/markdown content type, and did not open with an HTML document. 159 qualified, and 152 of those 159 open with a markdown heading, which is the shape the specification asks for. 473 hosts answered 404. Another 73 answered with a 2xx and an HTML page, which is a site telling a crawler that something exists at that path when nothing does, a failure mode this blog has seen before in 31 real llms.txt files and one whose 75 links all 404. One host, westpac.com.au, answered with a text/plain content type and compressed bytes this client could not read as text, and is excluded.

What is missing from both numbers is the 265 hostnames that never reached the comparison. 184 of them answered the home page request with a 403, 52 with a 503 and six with a 429. Sites most defensive about unfamiliar bot traffic are therefore over represented among the absent, which is its own finding and one we have measured directly: 79 of 115 sites refused a crawler their own robots.txt allows. Every rate below is a rate among sites that answer this client, and the scan methodology says the same thing about every figure this scanner produces.

The two files split the web by sector, not by evidence

The headline totals hide the real result. Break the 762 by the corpus sector each host is filed under and the two formats barely compete: they are adopted by different halves of the web, and almost nobody runs both.

News is the feed sector. 28 of the 63 news hostnames that answered declared a feed, including 404media.co, propublica.org, techcrunch.com, scientificamerican.com, variety.com and texastribune.org. Two of the 63 served an llms.txt. Software and infrastructure runs the opposite way: 73 of 122 software hostnames served an llms.txt, among them cloudflare.com, atlassian.com, databricks.com, circleci.com and fastly.com, while 13 declared a feed. Retail is the starkest of all. 23 of 74 ecommerce hostnames served an llms.txt and exactly one, bergfreunde.de, declared a feed.

Nothing in that pattern looks like a judgement about what AI systems read, because there is no published evidence for either side to have judged. It looks like what each sector's publishing stack ships by default. A newsroom's content management system has emitted a feed since before any of this started. A documentation platform added llms.txt generation in the last two years and switched it on. The eleven sites carrying both are mostly the places where those two stacks meet: foxnews.com, gitlab.com, datadoghq.com, clickhouse.com, netlify.com, mistral.ai and collibra.com among them.

That matters for anyone reading a generative engine optimization checklist, because the checklists have a strong opinion and the deployment data does not support it having one. We have published the evidence against our own product surface here before: what the evidence actually says about llms.txt reports that Ahrefs measured 137,210 domains in May 2026 and found 97 percent of the valid llms.txt files on them received zero traffic that month. We have also measured that the file usually stands alone, since 19 of the 23 sites publishing an llms.txt said nothing about AI in their robots.txt. Add today's number and the picture is consistent: the file is spreading through defaults rather than through results.

Fifteen of the 141 declared feeds did not answer, and 121 carried a date

A declaration is a promise about a URL, so each of the 141 declared feed URLs was then requested once, the same way. 126 answered HTTP 200. Eight answered 404, five answered 403 and two answered 410, and the failures are not obscure sites: berlin.de declares two feeds and both answer 410 Gone, cbsnews.com declares a latest news feed that answers 404, and washington.edu and jhu.edu between them declare four that answer 404 or 403. Counting by host, 14 of the 86 have no declared feed that this client could read back with a single item in it.

Of the 126 that answered, 124 opened with an rss, feed or RDF root element, so the content type promise held. 108 carried at least one item or entry, and across the parsed feeds there were 5,168 items in total. The number that matters most for an answer engine is the last one: 121 of the 126 carried a date element, a pubDate, an updated, a published or a lastBuildDate. That is the structural difference between the two files, and it is not a matter of maturity or adoption. RFC 4287, the Atom Syndication Format, published in December 2005, makes an updated element mandatory on every entry. The llms.txt format has no date field at all. It is a heading, a summary, and sections of links, and a reader cannot tell from it whether anything changed this morning or in 2024.

That gap lines up with the one real weakness this blog keeps measuring in the machine readable surfaces sites actually publish. A sitemap can carry a lastmod and frequently gets it wrong, which is why a sitemap tells a crawler where pages are, not that they changed. At the HTTP layer the situation is worse, since 250 of 647 home pages sent no signal that anything had changed when asked a second time with a conditional request. A dated feed is the one widely deployed artefact on a normal site that answers the freshness question directly, and the sector data above says it is the artefact the non news web has stopped publishing.

None of which makes a feed a retrieval strategy. 5,168 dated items are 5,168 dated items whether or not anything fetches them, and we did not observe anything fetching them.

Which file should you publish, and what this measurement cannot tell you

The fair answer is that these two files do different jobs and the choice follows from which job you have, not from which one an AI vendor is rumoured to prefer. A feed is a dated, ordered list of things that changed, and it is the better artefact if your site has a stream: articles, releases, status notices, anything where recency is the point. An llms.txt is a curated overview of what a site is and where its important pages sit, and for a documentation estate or a product site with no stream it genuinely says something a feed cannot. If you run both kinds of content, the eleven sites in this sample that publish both are not being redundant.

There is a sharper reason to keep a feed alive than either format's AI prospects, and it is the one finding here a site owner can act on today without believing anything about crawlers. Fifteen declared feed URLs did not answer, and on 14 hosts every declared feed failed. Those pages are telling every client that an alternate representation exists at a URL that returns 410 Gone. That costs nothing to check and it is checkable from outside, unlike most of what gets written about AI visibility.

Now the limits, stated plainly because they are large. This is a measurement of what sites publish on one day, not of what any crawler does with it. No access log was read. No request from GPTBot, ClaudeBot, PerplexityBot or any other named crawler was observed, so nothing here supports a sentence beginning with the words AI crawlers prefer. The corpus is an editorial sampling frame of large organisations stratified by sector rather than a random draw from the web, so the 11.3 percent and the 20.9 percent describe these 762 hostnames and nothing wider. And 265 hostnames never entered the comparison at all.

There is also a limit that belongs to this scanner rather than to the web. Lantad does not check your feed. The scan reads robots.txt per crawler, fetches the page as a crawler would and grades what is extractable, which you can see for yourself with what GPTBot sees or the robots.txt tester, and it will report on your llms.txt because we ship a generator for one. It reports nothing at all about whether a feed you declare resolves, which on today's numbers is a gap worth closing: the file we check is the one with no documented reader, and the file we ignore is the one with a documented reader and a mandatory date. That is not a comfortable sentence to publish on a page about AI visibility, and the version of the specification the generator follows is the one described in llms.txt v2 adds two link relations.