PerplexityBot robots.txt: 102 of 1,013 files named it, and 20 of the 45 blocking it left the user fetcher in

What does a PerplexityBot robots.txt rule control, and what does it not?

Perplexity documents its crawling on one page, and the page is explicit that the two agents are separate settings. It states that each setting works independently and that it may take up to 24 hours for Perplexity's systems to reflect changes. PerplexityBot is described as designed to surface and link websites in search results on Perplexity, with the sentence that decides whether a publisher should care: it is not used to crawl content for AI foundation models. The page then recommends allowing PerplexityBot in robots.txt and permitting requests from the published IP ranges, so that a site appears in search results. Blocking PerplexityBot is therefore not a training opt out on Perplexity's account of it. It is an opt out of being cited.

Perplexity-User is described as supporting user actions, visiting a web page to help answer a question and linking that page in the response. The documented robots.txt position is one sentence: since a user requested the fetch, this fetcher generally ignores robots.txt rules. Both agents carry a published address list, at perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json, written as plain text here because this site does not link hosts it has not registered. An address list is the only check a server can run on a claimed identity, and this blog has measured how thin those lists can be: none of the 2,344 addresses OpenAI publishes answered a reverse DNS lookup, which is the practical form of the rule that a user agent is a claim rather than an identity.

Perplexity is not alone in reserving the user-triggered case, and saying so is the difference between reporting and campaigning. OpenAI's bots documentation, read on 16 October 2026, says of ChatGPT-User that because these actions are initiated by a user, robots.txt rules may not apply, and directs owners to OAI-SearchBot for search opt outs instead. Anthropic's crawler page, dated 7 April 2026, goes the other way for its equivalent agent: it states that disabling Claude-User prevents Anthropic's system from retrieving content in response to a user query, while noting this may reduce a site's visibility in user-directed web search. Three vendors, three positions, and only one of them promises the rule works. The documentation layer of this post was covered in more depth when this blog read all three API surfaces in the Perplexity citation guide of 3 September 2026; what is new here is what the files actually say.

What was measured on 16 October 2026, and what was not

The population is the 1,419 hostnames in the two committed corpus seed files in this repository: worker/seeds/corpus-seeds-platform.json with 392 hostnames across ten platform strata, and worker/seeds/corpus-seeds-industry.json with 1,027 across eight industry strata. Each hostname was asked once for https://hostname/robots.txt, as LantadBot/1.0, from one network location, with redirects followed and a fifteen second timeout. The user agent is the one described on the bot page, and it was never altered to impersonate a crawler belonging to anybody else.

The funnel decides every proportion below, so it is worth stating in full. 1,098 hostnames answered HTTP 200 and 6 answered 202, giving 1,104 in the 2xx range. 41 of those 1,104 served an HTML page at the robots.txt address, a defect this blog has counted across the same corpus before, and they were excluded. 50 more parsed into no user-agent group at all, usually a file holding nothing but a Sitemap line or a comment, and a file with no group cannot name a token, so they were excluded too. That leaves 1,013, which is the denominator for every file count in this post. Of the rest, 178 hostnames answered 403, 49 answered 404, 35 answered 503, 3 answered 401, 4 answered 429, and 43 requests failed before any status arrived, 21 of them by timeout. A 403 on the robots file is a stronger refusal than anything the file could contain, which is the same asymmetry behind 92 of 1,056 sites refusing GPTBot the file.

Parsing and evaluation used parseRobotsTxt and evaluateRobots from core/src/robots.ts, the code that ships in the product rather than a script written for this post. A file counts as naming a token when a user-agent line lowercases to that exact string. A root verdict is the answer evaluateRobots returns for the path slash, which is the strongest version of the question, since a file can leave the root open and still close most of what matters. The approach is the one set out on the methodology page and used for the crawlability study.

Three limits apply to all of it. No Perplexity crawler was observed doing anything: there are no server logs here, so every statement about Perplexity's behaviour is a statement about Perplexity's documentation. One URL per hostname was requested, at the apex, so nothing here covers subdomains, where 73 of 277 gave an AI crawler a different answer from their apex. And these 1,419 front doors come from two committed sampling frames, not from the web, so no proportion here generalises beyond them.

102 files named PerplexityBot, and 45 closed the root to it

102 of the 1,013 files named PerplexityBot and 51 named Perplexity-User. 104 named at least one of the two, 49 named both, 53 named only the crawler and 2 named only the user fetcher. Of the 102 naming PerplexityBot, 39 disallowed it at the root and 63 left it allowed; six more files closed the root to it through a wildcard group without naming it, giving 45 across the whole set. For scale on the same files, 172 named GPTBot with 83 root blocks, 160 named ClaudeBot with 75, and 139 named CCBot with 80. Perplexity is named in a tenth of these files and refused in under five percent of them, which puts it below the two training crawlers this blog counted most recently, 130 of 1,077 files naming Bytespider and 108 of 1,016 naming Amazonbot.

Where the 45 sit is more informative than the count. 29 of them are news sites: of the 65 hostnames in the news stratum whose file parsed, 29 disallowed PerplexityBot at the root and 39 disallowed GPTBot. Across the other 948 files, 16 blocked PerplexityBot. None of the 120 SaaS files did, none of the 90 finance files did, and none of the 77 travel files did. The concentration matches what this blog found when it counted blocking by category and saw that 42 of the 77 files blocking a citation crawler were news sites. A refusal of Perplexity in this corpus is close to a synonym for a news publisher.

Then there is the shape of the refusal, which is where the commercial reasoning shows. Twelve of the 45 files that close the root to PerplexityBot reopen specific paths to it, with 38 Allow rules between them, and the reopened sections are not editorial. bloomberg.com keeps /professional, /company, /latam and /faq open. washingtonpost.com keeps /creativegroup/ and /advertising/. nzz.ch keeps /promoted-content and /sponsored-content, nzherald.co.nz keeps /sponsored-stories/*, thehill.com keeps /advertise/, variety.com keeps a 2026 shopping path and hollywoodreporter.com keeps /lifestyle/shopping/. nbcnews.com and cnbc.com both keep their /select subscription areas open, and ebay.com carries eleven Allow rules under a root block. The pattern is a publisher declining to supply the answer engine with the journalism and volunteering the pages that sell. A reader checking their own file can do the same evaluation per path with the robots.txt tester, which answers per token rather than reporting the file as a whole.

20 files blocked the crawler and left the user fetcher allowed

45 files disallow PerplexityBot at the root. 25 of them also disallow Perplexity-User. The other 20 leave Perplexity-User allowed, and only one of those 20, coralvilleanimalhospital.com, names Perplexity-User anywhere in the file. The remaining 19 are not sites that considered the user fetcher and allowed it. They are sites that wrote a rule for the name they had heard of. The list is mostly well-staffed newsrooms: nature.com, nejm.org, medscape.com, aljazeera.com, bloomberg.com, cnbc.com, forbes.com, washingtonpost.com, scientificamerican.com, thehill.com, variety.com, hollywoodreporter.com, restofworld.org, aftenposten.no, nos.nl, ansa.it, eltiempo.com, stuff.co.nz and ebay.com.

By Perplexity's documentation the gap is wider than it looks, because the half they closed is the half that carries citations and the half they left open is the one the vendor says ignores the file. A publisher that writes Disallow for PerplexityBot and nothing for Perplexity-User has, on the vendor's own account, removed itself from Perplexity's search results while leaving user-initiated retrieval exactly where it was. Twenty-five sites did write both rules, and on the same documentation the second rule has no effect either, because the fetcher generally ignores robots.txt. The honest summary is that nothing a site writes in this file reliably controls Perplexity-User, and only a rule at the edge or the origin can, which is why this blog keeps separating the two layers: 79 of 115 hostnames measured earlier refused a crawler what their own file allowed, and the refusal that held was the one in front of the file.

The same structure repeats across the three vendors that document a user-triggered agent, and the ratios differ in a way worth reading. 83 of the 1,013 files disallow GPTBot at the root, and 42 of those 83 leave ChatGPT-User allowed, a figure this blog counted at 38 of the files on 28 September 2026. 75 disallow ClaudeBot and 50 of those leave Claude-User allowed, the widest gap of the three, which matters more for Anthropic than for the others because Anthropic is the vendor whose page says the rule would have worked. Two files in the corpus run the comparison the other way, disallowing Perplexity-User at the root while leaving PerplexityBot allowed: botcity.dev and theregister.com. theregister.com is the more deliberate of the pair. It writes User-agent: PerplexityBot followed by Allow: / as its own group, an explicit welcome for the agent that carries citations, and its wildcard group then disallows everything else, which is what closes the root to Perplexity-User. That is the only configuration in these 1,013 files that reads as somebody having understood both tokens and chosen differently for each. Perplexity's gap is the narrowest of the three in absolute terms, 20 of 45. Naming both agents of a vendor is the exception everywhere: of the files naming at least one OpenAI agent, 92 of 180 name both, against 49 of 104 for Perplexity and 42 of 162 for Anthropic. Anthropic is last on that measure and is the vendor whose documentation says the second rule would have worked, which is the least comfortable combination of the three.

Five spellings of one vendor's name, and two of them bind nothing

Across the 1,013 files, Perplexity's name appeared in five distinct user-agent values: PerplexityBot on 102 files, Perplexity-User on 51, a bare Perplexity on 2, PerplexityAI on 1 and Perplexity-User/1.0 on 1. Only the first two are tokens Perplexity documents, and the three others behave in three different ways. A bare Perplexity is a prefix of both documented tokens, so under the group selection rule it binds both, and on lapresse.ca that is exactly what happened: its Perplexity group is what disallows Perplexity-User at the root, a rule the site may not realise it wrote. PerplexityAI, on scientificamerican.com, is a legal product token that matches no agent Perplexity runs, so it binds nothing, though that file's PerplexityBot token is in the same group and does bind. Perplexity-User/1.0, on france24.com, cannot bind anything at all: RFC 9309 section 2.2.1 states that a product token must contain only letters, underscores and hyphens, and a slash with digits after it is not a product token.

france24.com is the clearest case of belt and braces becoming belt, braces and a decorative third strap. It writes three consecutive groups, Perplexity-User, Perplexity-User/1.0 and PerplexityBot, each with Disallow slash. Two of the three bind. theglobeandmail.com does the same thing with a different mistake, writing Perplexity, Perplexity-User and PerplexityBot as three separate groups, where the first already covers the other two. Neither site is worse off for the redundancy, which is the point worth making: a surplus token costs nothing, while a missing one costs the whole rule. The opposite failure is more common and this blog has measured it twice, in 559 of 581 pages lost to a rule that never named the crawler and in the 2,209 tokens across 1,004 files of which 120 named one Anthropic no longer documents.

The case of the letters does not matter and the shape of the token does. RFC 9309 requires case-insensitive matching to find the group, which this corpus confirmed from the other direction two days ago when 84 mis-cased crawler tokens all bound anyway. What does not survive is a version suffix, a space, or a vendor name that is not the product token. The practical check is narrow: open the file, find every line naming the vendor, and confirm each value is exactly PerplexityBot or Perplexity-User with nothing appended. Then accept that the second of the two is documented not to obey whatever follows it, and decide the Perplexity-User question somewhere other than in this file.