Bingbot robots.txt: 62 of 1,019 files named it, and 83 that blocked GPTBot left it allowed

What does a Bingbot robots.txt rule control, and what does it not?

Every other major vendor in this measurement series answers to at least two robots.txt names: a crawler that fetches for search and a separate token a publisher can refuse to stop the content being used for AI training or grounding. Google has Googlebot and Google-Extended. OpenAI has GPTBot and OAI-SearchBot. Anthropic has ClaudeBot and Claude-SearchBot. Microsoft has one name in the same position, and the control it documents for AI answers is not a robots.txt token at all.

Microsoft's September 2023 announcement of webmaster controls for Bing Chat puts the AI opt out in a meta robots tag carrying NOCACHE or NOARCHIVE, and scopes its published example with the search crawler's own name rather than a new one. This blog established that separately when it found that Microsoft's AI opt out is a meta tag, not a robots.txt token, and that none of the product tokens in this scanner's crawler registry belongs to Microsoft. The registry itself is published on the AI crawler reference.

That leaves the Bingbot line doing a job it was not minted for. A robots.txt rule decides whether a page may be fetched; a meta tag inside a page can only be read once the page has been fetched. So the two controls sit in sequence rather than in parallel, and the robots.txt layer is the one that runs first. The honest statement about the Bingbot token is therefore narrow and worth keeping narrow: it is the name that decides what Bing's crawler may request, and nothing measured here says what refusing it does to a site's presence in any Microsoft AI surface. This post did not test that, because testing it would mean watching an answer engine rather than reading a file. What Bing does name in its AI reporting is the query it grounded on, which is more than Google exposes, and this blog measured that asymmetry when it found that Bing names the grounding query and Google names none.

What was measured on 22 October 2026, and what was not

The population is the 1,419 hostnames in the two committed corpus seed files in this repository: worker/seeds/corpus-seeds-platform.json with 392 hostnames across ten platform strata, and worker/seeds/corpus-seeds-industry.json with 1,027 across eight industry strata. Each hostname was asked once for https://hostname/robots.txt, as LantadBot/1.0, from one network location, with redirects followed and a fifteen second timeout. That user agent is the one described on the bot page, and no request was ever sent under a user agent belonging to Microsoft or anybody else.

The funnel sets every proportion below, so here it is in full. 1,103 hostnames answered HTTP 200 and 8 answered 202, giving 1,111 in the 2xx range. 44 of those served an HTML document at the robots.txt address rather than a robots file, a defect this blog has counted across the same corpus before, and they were excluded. 48 more parsed into no user-agent group at all, usually a file holding nothing but a Sitemap line or a comment, and a file with no group can name no token, so those went too. That leaves 1,019, the denominator for every file count in this post. Of the remainder, 160 answered 403, 50 answered 404, 42 answered 503, 4 answered 429, 2 answered 401, and single instances of 405, 418, 498 and 500 arrived. 46 requests failed before any status: 25 timed out and 21 failed at DNS or TLS.

Parsing and evaluation used parseRobotsTxt and evaluateRobots from core/src/robots.ts, the code that ships in the product rather than a script written for this post. A file counts as naming a token when a user-agent line lowercases to that exact string, which is why amsterdam.nl writing User-agent: BingBot counts, and the reason case folding is safe to rely on was established when 84 mis-cased crawler tokens all bound. A root verdict is the answer evaluateRobots returns for the path slash, the strongest form of the question, since a file can leave the root open and still close most of what matters. The approach is the one set out on the methodology page.

Three limits apply to everything below. No Microsoft crawler was observed doing anything, because there are no server logs here, so every statement about what Bing fetches or reports is a statement about what Microsoft publishes. One URL per hostname was requested, at the apex, so nothing here covers subdomains. And these 1,419 front doors come from two committed sampling frames built for another purpose, not from the web, so no proportion here generalises past them.

62 files named Bingbot, and only 4 closed the root to it

62 of the 1,019 files named Bingbot. For scale on exactly the same files, 175 named GPTBot, 164 named ClaudeBot, 149 named Google-Extended, 144 named CCBot, 132 named Bytespider, 124 named meta-externalagent, 117 named Applebot-Extended, 115 named Amazonbot, 103 named PerplexityBot, 82 named OAI-SearchBot and 74 named Googlebot. Bingbot is named less often than all of them. It sits below 140 of 1,013 files naming CCBot and below 159 of 1,014 naming ClaudeBot on earlier reads of the same corpus, and the only token in the series it outranks is the one counted yesterday, where 63 of 1,015 files named GoogleOther.

4 of the 1,019 files denied Bingbot the site root, and the four split evenly into two quite different acts. wa.gov and helsinki.fi name no Microsoft token at all: both write a blanket Disallow under the wildcard, so Bingbot is refused as one of the unnamed rather than as itself. amsterdam.nl is the only file in the corpus that names the crawler and shuts it out on purpose, heading a group with User-agent: BingBot and following it with an exact Disallow: /. The fourth, snu.ac.kr, is the interesting one, and its verdict depends entirely on a parsing question taken up in a later section: the file heads a group with User-agent: bingbot, follows it with Crawl-Delay:30 and no rule, and then writes User-agent: Amazonbot followed by Disallow: /.

The gap between 62 names and 4 refusals is the finding. 60 of the 62 files that name Bingbot leave the root open to it, and most of those names are doing less work than they appear to. On 15 of the 62 the rules in Bingbot's group are byte for byte the rules the wildcard group already carries, so naming the crawler changed nothing a crawler would do. 5 of the 62 head a group with the name and then write no allow or disallow rule under it at all. That leaves 46 files where Bingbot genuinely receives a rule set the wildcard does not get, which is the honest count of considered positions on Microsoft's crawler in this corpus: 46 of 1,019, or about one file in twenty two. The pattern where a name is present and decides nothing is not specific to Microsoft, and it was measured directly when 43 files put the ClaudeBot line inside the wildcard's own group.

There is one more group worth separating out, because it inverts the usual direction. 12 of the 1,019 files refuse an unnamed crawler the root outright. 9 of those 12 then name Bingbot and let it back in: coralvilleanimalhospital.com, botcity.dev, sciencedirect.com, scielo.org, eluniversal.com.mx, theregister.com, sap.com, instacart.com and gmarket.co.kr all operate a default-deny file with Microsoft's crawler on the permitted list. Only 2 of the 12 extend the same courtesy to GPTBot. Among sites that have decided to enumerate who may crawl rather than who may not, Bing is four and a half times more likely to make the list than OpenAI's training crawler is.

83 files refused GPTBot at the root and left Bingbot allowed

This is the number the post exists for. 87 of the 1,019 files disallowed GPTBot at the site root on 22 October 2026. 83 of those 87 left Bingbot allowed at the root, in the same file, on the same day. The reverse set is empty: not one file in the corpus closed the root to Bingbot while leaving GPTBot open. The four files that refuse Bingbot refuse GPTBot as well, so Bingbot's refusals are a strict subset of GPTBot's.

The mechanism is group selection, and nothing is malfunctioning. RFC 9309 defines a robots.txt group as one or more user-agent lines followed by rules, requires a crawler to obey the group matching it most specifically, and falls back to the wildcard group when no name matches. A Disallow written under User-agent: GPTBot is therefore selected by exactly one token. Bingbot does not match it, is not a prefix of it, and falls through to the wildcard, which in the overwhelming majority of these files permits the root. The file says what it says, and what it says about Bingbot is usually nothing. Divergence between two tokens inside one file is routine in this corpus, as the count showing 81 of 1,008 files allowed one of Googlebot and GPTBot and refused the other established.

The news stratum is where the asymmetry is starkest, and it is the stratum whose operators read crawler documentation most carefully. 68 news hostnames produced a parsed file. 41 of those 68 disallow GPTBot at the root. Not one disallows Bingbot, and only 6 of the 68 name Bingbot at all. The publisher position that has hardened over two years is specific rather than general: it refuses the crawlers it understands to train models and keeps the crawlers it understands to send traffic, and Microsoft's single token sits on the traffic side of that line by default. The industry contrast behind it was measured directly when 41 of 69 news robots.txt files disallowed GPTBot and 2 of 97 finance files did.

The strata also show where Bingbot rules come from at all, and it is not publishing. 16 of the 85 readable ecommerce files name Bingbot, the highest count of any stratum and more than a quarter of the 62. Seven of 81 government files and seven of 118 SaaS files name it. Four strata name it in no file at all: Framer, Webflow, Wix and Squarespace, and Shopify direct to consumer, which between them account for 134 parsed files. Those are the strata where the robots.txt file is generated by a platform rather than written by the site, so the absence is a statement about defaults rather than about intent. A per crawler refusal count across the same corpus, including the user fetcher that behaves differently from the indexer, is in the PerplexityBot measurement.

Crawl-delay is the directive Bing documents and Google ignores, and it changes which group a line belongs to

Microsoft is explicit about Crawl-delay in a way no other vendor in this series is. A Bing Webmaster Blog post of 3 May 2012 states that BingBot honors the Crawl-delay directive, and describes the value not as a rate but as a window: it defines the size of a time window, from 1 to 30 seconds, during which BingBot will crawl the site only once. The same post says a delay set in robots.txt overrides the setting in Bing Webmaster Tools, and that each subdomain with its own file is managed separately. That page is fourteen years old and is quoted here with its date attached for that reason.

Google's position is the opposite and current. Google's robots.txt documentation states that other fields such as crawl-delay are not supported, and lists only user-agent, allow, disallow and sitemap as fields it reads. So one line in a robots.txt file is load bearing for Bing and inert for Google, which is the kind of divergence that decides what a file actually does rather than what it looks like it does.

153 of the 1,019 files carried a Crawl-delay line somewhere, close to the counts this blog recorded when it found 157 of 1,014 files set one. 14 set one inside a group naming Bingbot. The values were 0.1, 1, 1, 1, 2, 3, 5, 5, 5, 5, 5, 5, 10 and 30 seconds. Two of those fall outside the window Bing's own post describes: shopee.sg asks for 0.1 seconds, below the documented floor of 1, and snu.ac.kr asks for 30, exactly at the documented ceiling. A tenth of a second is not a value the 2012 page contemplates, and what Bing does with it is not something this run could observe.

The second finding here is structural and it is the reason this section exists at all. Because Crawl-delay is not an allow or a disallow, a parser that ignores the field does not treat the line as ending the group, so consecutive user-agent lines keep accumulating across it. On 8 of the 14 files the Bingbot line therefore shares its group with other tokens: on coralvilleanimalhospital.com it sits in a group of 39 user-agent strings, on thrivemarket.com alongside twelve other AI and search tokens, and on shopee.sg with Googlebot and Googlebot-Mobile. In seven of those eight cases the shared rules are harmless. In the eighth it decides the file. snu.ac.kr writes Crawl-Delay:30 under Bingbot, no rule, then User-agent: Amazonbot and Disallow: /, so under the shipped parser the group is Bingbot and Amazonbot together and the root is closed to both. Bing, which honours the field, has no documented reason to read the line as ignorable, and this post cannot say which way its parser resolves the file. What can be said is that one directive, written to slow a crawler down, is the only thing standing between that site and a total refusal, and that the file is ambiguous rather than wrong.

Why is my site not cited in Bing's AI answers, and what can the new report tell you?

The development that prompted this measurement is worth reporting accurately, which means reporting it as Microsoft's work and not as anything measured here. On 13 October 2026 Krishna Madhavan, Saral Nigam and Vivek Jain of Microsoft Bing published Topics Portfolio, a beta view inside the AI Performance report. In their description it groups the grounding queries where a site appears into broader thematic clusters, presents them in a table of Topic, Topic Size, Sites Cited and Queries, and adds a Top Opportunity Queries list, where other sites are cited and yours is not.

Three caveats come from that post rather than from any reading of it here, and they are the reason a measurement blog finds the release interesting. Topic Size, Sites Cited and Queries are approximate, and opportunity citation volumes are deliberately shown as ranges. Topics with insufficient site presence or ecosystem activity are hidden, so in Microsoft's own words smaller sites may see a shorter list or may not even see this feature. And topic classification is AI-generated and can occasionally be imperfect, so a query can land in a cluster a human would have labelled differently. Branded queries can appear as opportunities too, which means a row can describe demand a site was never going to win.

None of that is a criticism, and Microsoft deserves credit for stating all three in the announcement rather than in a footnote. It does set what the report can settle. A ranged figure supports a direction and not a delta: if two topics come back as overlapping ranges, the report has not told you which is larger, and a number marked approximate cannot be differenced against last month's to produce a trend. This is the second Microsoft AI reporting surface this blog has read against that standard, the first being the finding that Search Console's generative AI report counts impressions and nothing else, and an honest AI visibility practice treats both as evidence of presence and absence rather than as measurement.

Which brings the two halves of this post together. An opportunity query is a question Bing answered by citing somebody else. The report tells you the question existed and that you were not in the answer. It cannot tell you which link in the chain failed, and the first link is not editorial: it is whether the crawler was allowed to fetch the page, and then whether the page said anything a crawler could read. The file level check is cheap and most sites in this corpus would pass it, since 1,015 of the 1,019 files leave Bingbot the root. The ones that would not pass are worth naming precisely, because two of the four do not mention Microsoft anywhere and would not be found by searching the file for its name. The order to check in is the order the machine works in: what the robots.txt file says to Bingbot, then whether the page carries a NOCACHE or NOARCHIVE meta tag, then what prose actually arrived in the bytes, and only then what the report says about topics.