PetalBot robots.txt: 71 of 1,026 files named it, and 64 of those refused Huawei's AI answers with it

What does a PetalBot robots.txt rule actually control?

Huawei's page answers this directly, and the answer is broader than the crawler's name suggests. PetalBot is described as an automatic program of the Petal search engine whose function is to access both PC and mobile websites and establish an index database. That index then does two jobs on the same page's account: it enables users to search the content of your site in the Petal search engine, and it presents content recommendations for the user in Huawei Assistant and AI Search services. The page says both of those services are powered by Petal Search.

The robots.txt token is the same for both of the crawler's declared products. The documentation lists PetalBot for the desktop fetcher and PetalBot for the mobile fetcher, with full user agent strings that differ only in the browser preamble: the desktop string is Mozilla/5.0 (compatible;PetalBot;+https://webmaster.petalsearch.com/site/petalbot) and the mobile string wraps the same token in an Android Safari signature. One token in a user-agent line therefore reaches the whole operation. The page also states that PetalBot complies with the Internet robots protocol, which is the robots exclusion protocol standardised as RFC 9309, and gives worked examples of both a full Disallow and a conditional Allow.

What the page does not offer anywhere is a second token. There is no Petal equivalent of a training-only or assistant-only control, and the note under the blocking instructions makes the consequence explicit rather than leaving it to be inferred: banning PetalBot from accessing your site will make the pages on your site and all search engine services provided by Petal unsearchable in the Petal search engine. A site owner who wants out of the assistant and recommendation surface has one lever, and pulling it takes the search listing with it.

Two smaller details on the same page are worth recording because they affect what a site owner should expect after writing the rule. The page says it may take several months to clear page index information already established, so the rule stops future crawling long before it changes what Petal can show. And the page carries a footer copyright line reading 2019 to 2020, with no last updated date anywhere on it, which is the dating problem this blog keeps running into when it reads crawler documentation at source: the text is current because it is published, not because it is fresh.

How many robots.txt files name PetalBot?

71 of the 1,026 parsed files named it in a user-agent line. That is well below GPTBot at 186 and ClaudeBot at 177, and below CCBot, Google-Extended, Bytespider, Applebot-Extended and Amazonbot as well. On naming alone PetalBot looks like a minor token, which is roughly how it is treated in the coverage this blog reads.

The root verdicts tell a different story. Evaluating each parsed file for PetalBot at the path / returned a disallow on 74 of the 1,026, which is more than Applebot-Extended and Amazonbot at 73 each and the same figure as Google-Extended, despite those tokens being named more than twice as often. Ten of the 74 came from the wildcard group rather than a PetalBot group, meaning those sites never wrote the name and caught the crawler with a site-wide rule, which is the pattern this blog has measured before when robots.txt names the wildcard and not the AI crawler.

The remaining 64 named PetalBot and refused it. Among the files that named a given token, that is the highest refusal rate of the twelve measured: 64 of 71, against 91 of 186 for GPTBot, 89 of 156 for CCBot, 86 of 146 for Bytespider and 66 of 125 for Amazonbot. The two conventional search crawlers in the set sit at the other end, as expected: Googlebot was named by 75 files and refused at the root by none of them, and Bingbot was named by 61 and refused by 3.

So naming GPTBot is close to a coin toss about what the rule will say. Naming PetalBot is not. Seven of the 71 left it allowed at the root, and the detail of those seven is worth stating because it is small enough to state exactly: five wrote an explicit Allow: / in the PetalBot group, and on two, ub.edu and absa.co.za, the named group held rules that matched paths further down the site and nothing that matched the root. Nobody in this corpus wrote a PetalBot group in order to welcome the crawler to the whole site in preference to anything else. They wrote one to shut it out, or they wrote one as part of a longer set of path rules.

The 34 sites that refused Huawei and kept OpenAI

Cross-tabulating the two root verdicts on the same 1,026 files puts the finding in one line. 894 files allowed both PetalBot and GPTBot at the root. 58 allowed PetalBot and refused GPTBot. 40 refused both. And 34 refused PetalBot while leaving GPTBot allowed.

That last cell is the one that does not fit the story usually told about AI crawler blocking. On those 34 sites the crawler that was shut out is the one whose operator says, in its own documentation, that the index it builds supplies an assistant and an AI search surface. The crawler left in is the one whose operator says, on the page Lantad read the same day, that disallowing it indicates a site's content should not be used in training generative AI foundation models. If the decision being made in these files were about machine reading of content, it would be hard to get that ordering by accident.

A likelier reading is that the two tokens are being judged on different criteria entirely. Huawei's page has a whole section headed on how much pressure PetalBot brings to a web server, says the crawler needs to maintain a certain amount of crawling, and gives an address to write to if its behaviour is unreasonable. A vendor does not publish a load complaints channel for a crawler nobody complains about. PetalBot appears to be refused as a traffic problem, on the evidence of where its name turns up, while GPTBot is refused as a policy question by the sites that refuse it at all.

The shape of the files supports that. The 71 files naming PetalBot had a median of 8 user-agent groups, against a median of 1 across all 1,026 parsed files. These are long, maintained documents written by somebody who adds names to a list, and 44 of the 71 named at least one AI crawler token somewhere in the same file. That is not a population of owners who have never thought about AI. It is a population who thought about both and ranked the crawl cost above the answer surface, which is a defensible call only if the person making it knew the two were the same lever. Huawei's page says they are. Nothing in a robots.txt file says so, which is the gap a robots.txt tester can close for an individual site in a few seconds.

18 identical files, written by a platform and not an owner

One group inside the 71 is not a decision at all. Eighteen of the files carried exactly four user-agent groups and an identical set of six tokens: the wildcard, adsbot-google, adsbot-google-mobile, ahrefsbot, dotbot and petalbot. All eighteen refused PetalBot at the root. All eighteen sat in the corpus stratum sampled for Wix and Squarespace sites. Not one of them named an AI crawler anywhere in the file.

That token list is the tell. AhrefsBot and dotbot are commercial link-graph crawlers, adsbot-google is Google's ad landing page checker, and the list contains nothing a language model vendor operates. It is a nuisance and SEO crawler blocklist, assembled by whoever authored the platform default, and PetalBot is in it. This blog has measured the same generated file from the other direction, finding that Wix sites named no AI crawler in robots.txt at all while the single full-site block in the generated file addressed PetalBot. Reading it across the whole corpus rather than one platform stratum gives the figure above: eighteen small businesses refusing a crawler that feeds an assistant, through a rule none of them wrote.

Eighteen of the 27 files that named PetalBot without naming any AI crawler were these. The other nine were spread across local media, finance, education, healthcare, ecommerce and travel, and are the ordinary case of a short hand-written blocklist. Stratum by stratum, the Wix and Squarespace sample is where the token concentrates: 18 of its 54 parsed files named PetalBot, against 14 of 68 news files, 8 of 85 healthcare, 5 of 98 education, 3 of 84 ecommerce, 2 of 92 finance and 1 of 118 SaaS. News is the only stratum where the naming looks like editorial policy, and it is also the stratum where this blog has previously found the heaviest AI crawler blocking by industry.

The general point survives the specific platform. A robots.txt file is read as a statement of a site's position, by vendors building compliance claims and by anyone counting blocks to describe the market, and a meaningful share of the lines in it were typed by a template author years earlier. Any figure drawn from these files, including every figure in this post, is a measurement of what the files say and not of what site owners believe. Lantad reports the same distinction in its methodology, because the gap between the two is not a rounding error here: it is 18 of 71.

One token where Google and OpenAI document two

The reason the refusal rate matters is that Huawei does not let a site separate the two uses, and its larger competitors now do. This is a documentation comparison, read at source on 5 November 2026, not a measurement of what any engine does with the files.

Google's common crawlers page says Google-Extended is a standalone product token that publishers can use to manage whether content may be used for training future generations of Gemini models and for grounding in Gemini Apps and Vertex AI. The same page states that Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal, and notes that the token has no separate HTTP user agent string because it is used in a control capacity. A site can refuse the Gemini uses and keep the search listing, which this blog measured in practice when it found that 52 of 54 sites refusing Google-Extended still allowed Googlebot.

OpenAI documents the same separation across more tokens. Its crawler page says that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, that disallowing GPTBot indicates content should not be used in training generative AI foundation models, and, in the sentence that settles the question, that each setting is independent of the others. The measurement above shows site owners using that independence: OAI-SearchBot was refused by 22 of its 84 namers while GPTBot was refused by 91 of its 186, which is the gap you would expect if people are keeping the search surface and declining the training one. It is the same asymmetry this blog found between Googlebot and GPTBot on 81 of 1,008 files.

Huawei's page offers no equivalent. There is one token, the index behind it serves search and two assistant surfaces, and the blocking note says a ban makes the site unsearchable across all Petal services. So the honest advice for a site owner with a Huawei audience is narrower than the advice for a Google or OpenAI audience: decide whether you want to be in Petal at all, because that is the only question the file can answer. Anyone weighing that for a specific market can see what their own pages return to a crawler through what GPTBot sees, and the broader readability picture across this corpus sits in the crawlability study.

One procedural difference is worth crediting, with a limit attached. Huawei's page documents a verification procedure: run a reverse DNS lookup on the accessing address, check the name is in petalsearch.com, then run a forward lookup on that name and confirm it returns the original address. Lantad ran the forward half on the page's own worked example on 5 November 2026, and petalbot-114-119-128-10.petalsearch.com resolved to 114.119.128.10 as documented. The reverse lookup did not complete from our environment, so this post does not report on that step. Publishing a procedure at all puts Huawei ahead of several vendors on paper: this blog found that 0 of 2,344 addresses OpenAI publishes answered a reverse DNS lookup, and that a user agent remains a claim rather than an identity until something checks it, which is why Lantad publishes its own crawler's identity on its bot page.

What this measurement does not show

The denominators are the first limit and they are large. 1,419 hostnames were asked and 1,101 answered HTTP 200, so 318 contributed nothing: a file that 404s, times out or returns a server error is not a site that allows or refuses anything, and reading it as either would be the error this blog has written about when a 404 on robots.txt allows every crawler and a 503 blocks them all. A further 75 answered 200 with no parseable user-agent group, which is why every figure here is against 1,026 and not 1,101 or 1,419.

The second limit is that this is a reading of files, not of behaviour. Lantad has not observed a single PetalBot request, has not measured whether Petal honours any of these rules, and has not measured whether a site refused by these files appears in Huawei Assistant or in AI Search. Everything in the comparison table about what a vendor permits comes from that vendor's own page on one day. Every claim about what the refusals take away comes from Huawei's description of its own index. Both are attributed rather than measured, and neither becomes a Lantad finding by sitting next to one.

The third is sampling. The corpus is stratified by platform and by industry to study crawlability, not weighted to represent the web, so the 18 identical Wix and Squarespace files are 18 of 54 in that stratum and would be a different share of a differently drawn sample. The strata are small enough individually that a single-digit count, such as the 1 of 118 SaaS files naming PetalBot, should be read as the direction it points rather than a rate. The same caution applies to the seven files that named PetalBot and allowed it: seven is an example set, not a trend.

What the measurement does support is narrow and, we think, useful. On 5 November 2026, in this corpus, PetalBot was the token a site was most likely to refuse once it had bothered to name it, the refusal was more often inherited from a platform template than written by an owner, and the operator's own documentation says that refusal reaches an assistant and an AI search surface as well as a search listing. Anyone running an AI visibility programme that includes a Huawei market should check which of those three sentences applies to their own file before assuming the first one does not. The vocabulary for that check sits in this site's notes on generative engine optimization and answer engine optimization, and the per-engine version of the question for the engines most readers care about is in how to get cited by ChatGPT. The full token inventory this corpus yields, 2,209 tokens across 1,004 files when it was last counted, is in crawler user agent.