Reflectionbot: 1 of 1,068 robots.txt files named it, and 72 of the 77 blocking GPTBot let it in

What is Reflectionbot, and what does Reflection AI say it does?

Reflection AI's documentation is short and unusually specific about matching, which is worth repeating because it is the part that decides whether a rule works. The page states that Reflectionbot announces itself with a single user-agent token, Reflectionbot, and gives an example request string of the form Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Reflectionbot/1.0; +https://reflection.ai/bot) Chrome/151.0.0.0 Safari/537.36, noting that version numbers may change. It then instructs readers to match on the Reflectionbot token rather than on the full user-agent string, whether in robots.txt or in server logs.

That instruction is correct and it is the one most often got wrong. A full user-agent string carries a browser version that moves, so a rule pinned to the whole string stops matching the first time the vendor bumps a number. This is not a hypothetical failure in this corpus: we have already found 1,004 files naming 2,209 tokens of which 120 named a token Anthropic no longer documents, which is what copied blocklists look like once the vendor has moved on.

The page also offers reverse DNS verification, stating that Reflectionbot requests originate from IP addresses resolving to hostnames under reflection.ai, and publishes the ranges at reflection.ai/bot/ips.json. Reflection AI's host is not registered in this site's outbound link policy, so both URLs appear here as plain text rather than as links, and a reader who wants to confirm any of this should open them directly. When we read that file on 3 October 2026 it held two IPv4 prefixes and nothing else, which is 1,024 addresses across the two /23 blocks and no IPv6 coverage. Its creationTime field read 22 September 2026 while the Last-Modified header on the same response read 2 October 2026, a ten day gap of exactly the kind we catalogued when crawler IP range files carried two dates that disagree.

None of that is a measurement of Reflectionbot's behaviour and this post does not make one. We hold no server logs for other people's sites, we observed no Reflectionbot request, and we did not test whether it honours anything. A crawler's documentation is a claim about intent, and a user agent is a claim rather than an identity in the first place. What can be checked cheaply is whether the vendor gives you the means to verify a request, and here it does: published ranges plus reverse DNS puts Reflection AI ahead of several larger operators, though behind the two that publish a signing key directory for web bot authentication.

How many robots.txt files name Reflectionbot?

One. Of the 1,068 parseable files, a single robots.txt carried Reflectionbot on a User-agent line: slate.com, which spells it ReflectionBot with a capital B and attaches Disallow: / to the group it sits in. The spelling difference does not matter, because RFC 9309 requires crawlers to use case-insensitive matching when finding the group that matches a product token, so slate.com's rule is a real refusal rather than a near miss.

It is worth saying what kind of file that is. slate.com's robots.txt carries 427 User-agent lines, a consecutive stack of crawler names closed by a single Disallow: /, which is the shape of a maintained publisher blocklist rather than something a site owner typed once. 131 of the 1,068 files in this corpus name twenty or more tokens, and the largest, museodelprado.es, names 639. One of those long lists had Reflectionbot in it eleven days after the creationTime its IP range file carries. The others did not.

A second file, figma.com, matched a naive search for the string reflection, and it is a useful false positive to record rather than quietly drop. Every hit was a page path such as Allow: /templates/reflections-for-meetings/$, not a user agent. Searching a robots.txt for a vendor's name finds page titles as readily as crawler rules, which is why the counts in this post come from parsing User-agent lines into groups rather than from grep.

How far any of this generalises depends on the frame, and it is worth being exact rather than reassuring. These are the 1,419 hostnames this repository seeds its corpus from, grouped into eight industry strata and ten website-builder platform strata, chosen to span the web rather than drawn at random from it. How that frame is built and what it will not support is written up in the crawlability study. Read every rate here as a measurement of 1,068 named sites on one date.

Set the single naming against the files that have demonstrably thought about this subject. 212 of the 1,068 named at least one of the fifteen AI crawler tokens listed in core/src/bots.ts, this scanner's bot registry. That registry is a setting somebody chose rather than a finding, and the AI crawlers reference lists tokens rather than companies for the reason this post keeps running into: the unit a robots.txt file operates on is the token, and a vendor is not a unit at all. 175 named GPTBot, 163 ClaudeBot, 147 Google-Extended, 142 CCBot and 129 Bytespider. Those are not people who have never heard of AI crawling. They are the population that read something, formed a view and wrote it down, and exactly one of them had Reflectionbot. A naming count is a count of strings somebody typed on some past date, a caveat that GPTBot being named in 4.5 percent of files while the wildcard appeared in 77 made concrete, and it understates nothing here: one is one.

What decides access for a crawler nobody has named?

Not nothing, which is the point people miss. RFC 9309 settles this in two sentences, and they are worth reading in the order the specification puts them. If no group matches the product token, a crawler must obey the group with a User-agent line of "*", if one is present. If no group matches and there is no "*" group either, or no groups are present at all, no rules apply.

Applied to Reflectionbot across the 1,068 files, that produces three populations. On 1 file the token was named. On 1,010 the wildcard group decided, carrying a median of 9 rules and in one case, boe.es, 12,312 of them. On 57 files no group applied at all, because the file named other crawlers and never wrote a wildcard group, which means Reflectionbot gets everything those files serve. ryanmulligan.dev is the minimal example: its entire robots.txt is one group, User-agent: GPTBot with Disallow: /, so GPTBot gets nothing and every other crawler in existence gets the whole site.

Evaluated at the site root, 14 of the 1,068 files disallowed Reflectionbot and 1,054 allowed it. That 14 is low and it should be: a wildcard group carrying Disallow: / refuses Googlebot too, so almost nobody writes one. The 14 that did are coralvilleanimalhospital.com, botcity.dev, amsterdam.nl, wa.gov, helsinki.fi, sciencedirect.com, scielo.org, pennmedicine.org, eluniversal.com.mx, slate.com, theregister.com, sap.com, instacart.com and gmarket.co.kr. Thirteen of those refuse it incidentally, as a side effect of refusing everything. One, slate.com, refuses it by name.

Inheriting the wildcard group is not the same as being unrestricted, and it is not the same as being governed either. A median of 9 rules means most of these sites hand Reflectionbot a housekeeping list: a /search path, an account area, a few asset directories. Those rules were written for search engines years before Reflection AI existed, and whatever they happen to cover is now this crawler's access policy. That mechanism, a rule deciding a crawler's fate without ever naming it, is the same one behind 559 of the 581 pages GPTBot lost being closed by a rule that never named it, running here in the permissive direction instead of the restrictive one.

72 of the 77 sites that refuse GPTBot admit Reflectionbot

This is the finding, and it is a finding about mechanism rather than about anybody's carelessness. 175 of the 1,068 files name GPTBot. 77 of those disallow it at the site root, which is a deliberate act: somebody read about OpenAI's training crawler, decided against it and wrote a rule. On 72 of those same 77 files, Reflectionbot is allowed at the root. 69 of the 72 fall to a wildcard group that names neither crawler, and 3 have no applicable group at all.

mxb.dev shows the whole shape in a 25 line file. Its wildcard group disallows /build.txt and /drafts. Underneath it sit separate groups for CCBot, ChatGPT-User, GPTBot, Google-Extended, Omgilibot and FacebookBot, each carrying Disallow: /. Six named crawlers get nothing. Reflectionbot, named nowhere, inherits the wildcard group and reads the entire site except a text file and a drafts directory. Nobody made that decision. The file is doing exactly what it says and exactly what RFC 9309 specifies, and the outcome is the opposite of what the author's six explicit refusals suggest they wanted.

The pattern is not specific to GPTBot. Of the files naming each token, 68 blocked ClaudeBot at the root and 64 of those allowed Reflectionbot, 75 blocked CCBot and 70 allowed it, 68 blocked Bytespider and 63 allowed it, 56 blocked Google-Extended and 52 allowed it, and 41 blocked PerplexityBot and 38 allowed it. Across the 212 files naming any registry token, 148 gave Reflectionbot a different rule set than at least one AI crawler they had named. The asymmetry is the same one we reported when 52 of 54 sites refusing Google-Extended still allowed Googlebot and when 6 of 1,065 files named the Meta token tied to citation: per-token rules do not generalise, because robots.txt has no concept of a category.

The distribution is the part publishers should read twice. 38 of the 72 are news sites. News is the stratum that blocks AI crawlers most deliberately in this corpus, and that is precisely why it is the most exposed: a site with no AI crawler rules at all has nothing to fall out of date, while a site with a carefully maintained list of nine named trainers has a list that is wrong the moment a tenth ships. Education contributed 7 of the 72 and healthcare 7, against 3 from SaaS and 1 from the WordPress stratum. Enumeration costs the people who did the work.

Allowing Reflectionbot does not buy a citation

There is an argument this blog could make here and will not, because the evidence does not support it. The usual framing for a product that sells AI visibility is that letting a crawler in is how you get mentioned, so a site accidentally blocking one is losing something. For Reflectionbot that argument runs backwards, and saying so is more useful than selling the alarm.

OpenAI's crawler documentation, read on 3 October 2026, separates the two jobs explicitly: it describes OAI-SearchBot as used to surface websites in search results in ChatGPT's search features while disallowing GPTBot indicates that crawled content should not be used for training its generative models. Anthropic's support article does the same, describing ClaudeBot as collecting web content that could contribute to training and Claude-SearchBot as navigating the web to improve search result quality, and states that blocking Claude-SearchBot may reduce a site's visibility in user search results. Those vendors give a site owner two decisions to make, and only one of them is about being cited.

Reflection AI's page offers one token and describes it as collecting pages to help build and improve its models. It states that Reflectionbot is the only user agent Reflection AI crawls with today. So on the documentation as published on 3 October 2026 there is no search-side or citation-side token to allow, which means allowing Reflectionbot is a contribution to a training corpus and nothing else. Whether that is a trade worth making is a judgement about licensing and about what you think training access is worth, not a visibility question, and the guides for getting cited in ChatGPT and in Claude have nothing to say about it because there is nothing yet to say.

That cuts both ways, and the honest version includes the half that is inconvenient for us. A site that wants to refuse training access while keeping citation access has a real decision to make here and the files show almost nobody has made it. But a site that accidentally allows Reflectionbot has not lost a citation, because there is no citation on offer. Access is a necessary condition for being quoted and never a sufficient one, which is the distinction the GEO category collapses most often, and it is why our own scoring methodology treats crawler access as one input rather than as the answer. The practical step is small: evaluate your file once per token rather than once per vendor, which is the ordering problem AI crawler detection turns on, and decide whether your wildcard group says what you would want it to say to a crawler that does not exist yet. On 1,010 of 1,068 files, that group is already the policy.