Brave Search crawler: 126 of 128 robots.txt files blocking an AI crawler left Googlebot allowed

How do I block the Brave Search crawler in robots.txt?

You cannot name it, and that is the answer rather than an evasion of it. RFC 9309, the Robots Exclusion Protocol, builds the whole mechanism on the product token. Section 2.2.1 says a crawler selects the group matching its own token and falls back to the group whose value is a star when no group matches it. Section 2.2.2 says a matching group with no rules leaves the URI allowed. Nothing in the specification gives a site a way to address a crawler that declines to identify itself, because the specification assumes identification.

So the honest sequence for Brave's crawler has two steps and neither of them involves writing its name. First, whatever your file says to Googlebot is what reaches it, by Brave's own statement that a page not crawlable by Googlebot will not be crawled by Brave Search's bot. Second, Brave's help page directs site owners who want a page out of the index to the robots meta tag rather than to robots.txt: "A site owner can delist a page by using the robots noindex directive." That is a page level control served in HTML, not a crawl control served in a text file, and it only works on pages a crawler is still allowed to fetch and read.

Both steps have costs this blog has already measured, and they are the reason the question is harder than it looks. Disallowing Googlebot at the root stops Google crawling the site at all, and Google's robots.txt documentation is explicit that the file governs crawling rather than indexing, so it is an expensive instruction to write and almost nobody writes it. And the noindex route depends on a tag most sites do not serve: Lantad found that 629 of 1,023 corpus home pages carried no page level robots directive at all on 2 November 2026. A site following Brave's documented advice starts from nothing on both levers. You can check what your own file currently resolves to with the robots.txt tester, which runs the same evaluator used for every figure below.

What Brave publishes about its own crawler, and what it sells

Three Brave pages, all read at source on 3 November 2026, describe the pieces. The crawler help page carries the two sentences quoted above. The Brave Search API page at brave.com/search/api/ advertises the index to AI developers, stating that the API is "trusted by the world's top AI labs" and, in the same line, "including ones we can't name". The API documentation at api-dashboard.search.brave.com carries an endpoint called LLM Context, described there as "Web search for AI agents and chatbots" offering "Advanced Web search, with pre-extracted content that's optimized for AI agents, LLM grounding, and RAG pipelines".

That last description matters for how a site should classify this crawler. Pre-extracted page content served for grounding is not a link in a results page: it is the page body, delivered to a model at answer time, and that is the surface AI visibility measures. A site that has disallowed GPTBot and ClaudeBot at the root has stated a position on exactly that transaction, and this is a second supply route into it that no rule in the file addresses. This blog has measured the shape of that problem before from the other end, reporting that a robots.txt block did not stop 12 of 18 AI chatbots, and that study is where the Brave path first appeared in our corpus work: it recorded content associated with third party search crawlers, Brave's among them, turning up in chatbot answers whose vendor operated no such crawler.

The legal claims arrived this week and they are claims, not findings. On 2 November 2026 The New York Times Company filed a 72 page complaint against Brave Software, Inc. in the United States District Court for the Southern District of New York, case number 1:26-cv-09549, available at nytco-assets.nytimes.com/2026/11/new-york-times-brave-complaint-11-2-26.pdf. Paragraph 2 alleges that Brave "has never disclosed the name of its bot" and that Brave "crawled and copied content from the over 30 billion webpages in the Brave Index without identifying itself or providing website owners with any mechanism to block its crawler". A second suit by other newspaper publishers was reported the same day, and Lantad did not open that filing. Lantad has measured none of that, Brave has not answered it in anything we have read, and nothing in the complaint is established. The documented crawler behaviour is the part a site owner can act on today, and it is published by Brave itself.

126 of 128 files that refuse an AI crawler leave Googlebot allowed

The measurement is a single comparison run over every file that held rules. For each of the 1,025, resolve the path / for each of the 15 AI crawler tokens in core/src/bots.ts, then resolve the same path for Googlebot. 128 files disallow the root to at least one AI token. 2 of those 128 also disallow it to Googlebot: wa.gov and helsinki.fi. The other 126 do not.

The per token counts sit in the hero table and they are ordinary for this corpus: GPTBot refused at the root by 97 files, CCBot by 95, Bytespider by 93, ClaudeBot by 91. Those are the same files and roughly the same proportions this blog reported when it counted that 1,004 files named 2,209 crawler tokens, and today's parse found 2,219 distinct tokens across 1,025 files. The point is not that the numbers moved. It is that 2 is the Googlebot figure in every version of this measurement, because disallowing Googlebot at the root is a decision to leave Google Search, and almost nobody takes it.

The news stratum is the cleanest case because it is where AI refusal is most deliberate. 68 news hostnames parsed into rule groups. 46 of those refuse the root to at least one AI crawler, which matches the pattern this blog found when it reported that 41 of 69 news files disallowed GPTBot and that the sites that block AI crawlers are the ones with editors. All 46 of the 46 leave Googlebot allowed at the root. Not one news site in this corpus has closed the path Brave's documentation describes, and the publishers suing Brave are drawn from exactly this category.

Recognisable names across the 126 include who.int, congress.gov, fema.gov, va.gov, illinois.gov, britishmuseum.org, unesco.org, cambridge.org, coursera.org, amsterdam.nl and broadinstitute.org. 29 of the 126 name Googlebot explicitly, in a group that allows it, so these are not files written by people unaware of the token. They are files where the author wrote a deliberate Googlebot allowance and a deliberate AI refusal, and the two instructions select the same crawler differently.

Seven files named bravebot, and six of them disallowed it

Some site owners have tried anyway. 7 of the 1,025 files name a user-agent token called bravebot, and 6 of those 7 disallow the site root to it: qcitymetro.com, congress.gov, slate.com, nzherald.co.nz, theglobeandmail.com and svt.se. The seventh, postman.com, names it inside a long allow group. None of the 6 also disallows Googlebot at the root, so on all six the rule that would bind this crawler under Brave's documented behaviour is the one left open, and the rule they wrote is the one addressed to a name Brave does not publish.

Be careful about what that does and does not show, because the temptation here is to call the token imaginary and the evidence does not support that. Brave's crawler help page says the crawler does not advertise a differentiated user agent, which is Brave's statement about Brave rather than an observation of it. Our own earlier post on the eighteen chatbot study recorded a researcher associating fetched content with a Brave crawler, so the name is in circulation and is not something six site owners invented between them. What the measurement can say is narrower and still useful: six sites wrote a rule against a token that no Brave documentation we read publishes, and the group that Brave documentation does describe is allowed on all six.

This failure mode recurs across the corpus and it is not specific to Brave. Lantad has reported that 559 of the 581 pages GPTBot lost were closed by a rule that never named it, the mirror image of the same arithmetic, and that fourteen Google agents are not stopped by a robots.txt wildcard. Token level rules only work when the token is right, which is also why this blog keeps a maintained AI crawler reference rather than a list in a post. A rule aimed at a name nobody answers to is indistinguishable, in the file and in the verdict, from no rule at all.

On 10 sites, declaring yourself is what gets a crawler blocked

One more comparison separates the two behaviours cleanly. Resolve the root for a token that appears nowhere in any file, which is what an undeclared crawler falls back to under RFC 9309 section 2.2.1. 12 of the 1,025 files disallow the root to that token through their wildcard group. 10 of the 12 then allow Googlebot explicitly at the root: coralvilleanimalhospital.com, botcity.dev, amsterdam.nl, sciencedirect.com, scielo.org, eluniversal.com.mx, theregister.com, sap.com, instacart.com and gmarket.co.kr.

On those 10 files the incentive runs backwards. A crawler that publishes a token nobody has written a rule for lands in the wildcard group and is refused. A crawler that publishes no token, and takes Googlebot's permissions instead, is allowed, because the file goes out of its way to allow Googlebot. Brave's stated reason for advertising no differentiated user agent is to "avoid discrimination from websites that allow only Google to crawl them", and on these 10 sites that is a precise description of what the file does. The design choice and the file's behaviour fit each other exactly.

It is worth stating plainly that this is not evidence of anyone acting in bad faith, and the arithmetic would be the same if the crawler were Bingbot. It is a property of an identity based protocol: the more precisely a file allows one named crawler, the more an unnamed crawler inherits. Lantad has measured the adjacent case, finding that every web agent was identifiable and robots.txt stopped one of twelve, and that 103 of 1,089 sites served an unknown bot while refusing GPTBot, where the filtering happened at the CDN rather than in the file. The direction of the error differs, and in both cases the file was not deciding what the site owner believed it was deciding.

What this measurement does not show

Four limits, and the first is the largest. Lantad did not observe Brave's crawler. Nothing here is a log study, no request in this measurement came from Brave, and we cannot confirm from our own evidence that the crawler behaves as the help page describes. Every statement about what it does is Brave quoting itself, read at source on 3 November 2026, and a site owner should treat it as a vendor claim rather than a verified behaviour. A crawler that publishes no token also cannot be verified by reverse DNS the way Cloudflare's verified bot grading assumes, which removes the usual check.

Second, this is one request per hostname for one path. The verdicts resolve the site root only, so a file that allows the root and disallows a section is counted as allowing, and a site whose protection lives in a per directory rule is misrepresented by this frame. Third, the corpus is a stratified sample of 1,419 hostnames, not a census of the web, and the news stratum at 68 parsed files is small enough that the 46 of 46 result should be read as a consistent pattern rather than a rate. The sampling frame and its strata are described on the methodology page.

Fourth, robots.txt was never the whole of this. It governs fetching, not use, and a file that arrives at all is doing better than many: in the same run we found that 34 of 1,104 sites answered a crawler with an HTML page instead of a rules file, and separately that a 404 on robots.txt allows every crawler while a 503 blocks them all. Whether the file carries legal weight is an open question this blog has covered in is robots.txt legally binding, and the complaint filed this week will test a related question about the consequences of there being no name to write a rule for. The measurable part today is smaller than any of that, and it is this: on 126 of the 128 corpus sites that have refused an AI crawler by name, the one group Brave's own documentation says its crawler obeys is allowed.