# GPTBot is named in 4.5 percent of robots.txt files, the wildcard in 77

> The 2025 Web Almanac, written from HTTP Archive's July 2025 crawl of 16,213,084 websites and published on 15 January 2026, reports the catch-all user agent in 77.04 percent of desktop robots.txt files and gptbot in 4.5 percent. Google's advertising crawler is named more than twice as often as any AI crawler.

- Canonical page: https://lantad.co/blog/robots-txt-names-the-wildcard-not-the-ai-crawler
- This file: https://lantad.co/blog/robots-txt-names-the-wildcard-not-the-ai-crawler.md
- Last substantive update: 2026-08-16

## Key facts

- **Published:** 2026-08-16
- **Category:** Findings
- **Author:** Lantad
- **Length:** 3590 words
- **Takeaway 1:** The 2025 Web Almanac SEO chapter, published on 15 January 2026 from HTTP Archive's July 2025 crawl of 16,213,084 websites, reports the wildcard user agent appearing in 77.04 percent of desktop robots.txt files and 77.14 percent of mobile ones.
- **Takeaway 2:** In the same crawl gptbot appeared in 4.5 percent of desktop files and claudebot in 3.6 percent, against 9.8 percent for Google's advertising crawler adsbot-google and 9.3 percent for the SEO tool crawler ahrefsbot.
- **Takeaway 3:** The Web Almanac reports gptbot up from 2.9 percent of desktop files in 2024 and claudebot up from 1.9 percent, which its authors call one of the fastest expansions of robots.txt directives for named user agents in recent memory.
- **Takeaway 4:** RFC 9309 tells a crawler to obey the group matching its own product token and to fall back to the group for the wildcard only when no specific group matches, so on most sites in this crawl an AI crawler is governed by rules nobody wrote for it.
- **Takeaway 5:** Lantad has not crawled robots.txt at web scale and reports every figure here from the Web Almanac's published chapter, read on 16 August 2026.

## Summary

A robots.txt file is the one place on a site where a publisher can address a crawler by name, and the useful question is how often anybody does. Almost every piece of advice written about AI crawlers in the last two years assumes the answer is often: open the file, add a group for GPTBot, add another for ClaudeBot, decide each one on its merits. The web that actually exists answers differently, and somebody has counted it at a scale no single scanner reaches.

HTTP Archive publishes the Web Almanac, an annual report written from its own crawl of the web rather than from a survey, a panel or a customer base. The 2025 SEO chapter, published on 15 January 2026 and written by Amaka Chulwuma, Chris Green and Sophie Brannon, carries a section on which user agents appear inside robots.txt files. Its central figure is that the catch-all wildcard appears in 77.04 percent of desktop files and 77.14 percent of mobile ones, while gptbot appears in 4.5 percent of desktop files and claudebot in 3.6 percent. The chapter is published at almanac.httparchive.org/en/2025/seo and every number below is read from it.

Lantad did not run that crawl and has measured nothing about robots.txt at web scale. What follows reports the Almanac's figures with attribution, and then covers the part this site can speak to, which is what a file shaped like that means for whether [an AI crawler](https://lantad.co/glossary/ai-crawler) is allowed to read your pages, and what is worth looking at in your own file as a result.

## What the Web Almanac counted, and on how many sites

The methodology matters more than usual here, because a percentage of the web means nothing until you know which web. The Web Almanac's 2025 methodology page states that there are 16,213,084 websites in the dataset, of which 15,426,398 are mobile websites and 12,155,374 are desktop websites, and that unless a chapter says otherwise all metrics were sourced from the July 2025 crawl. Desktop and mobile are separate runs against the same site list rather than two views of one result, which is why every figure in this post comes as a pair.

The runs are real page loads, not header probes. The methodology describes desktop testing on a Linux virtual machine using desktop Chrome at a cable network speed, and mobile testing on an emulated Moto G4 in mobile Chrome at 4G speeds with eight times CPU throttling, both from Google Cloud Platform locations in the USA. That is a heavier and more honest instrument than a fetch of one file, and it is also a US-based one, which is worth holding on to before reading any of these numbers as a global rate.

One thing the chapter does not settle, and it changes how a reader should hold the figures, is the denominator. The chart descriptions in the user agent section say a token appears in a given percentage of desktop sites and mobile sites, while the prose about the wildcard says it appeared in 77 percent of files. Those are not the same population: the Almanac's own status code section reports that around 13 percent of requests for robots.txt returned a 404, so a share of sites and a share of files differ by roughly that much. The chapter does not reconcile the two descriptions and neither will this post. Read the numbers as the shape of a distribution rather than as a rate accurate to the decimal place, which is the same discipline that applies to [the crawlability study published here](https://lantad.co/research/crawlability-study) and to every sample-based figure in [how this scanner scores a page](https://lantad.co/methodology).

The other limit is age. This is a July 2025 crawl reported in January 2026 and read in August 2026, so it describes the web thirteen months ago. In a field where a vendor can rename a crawler token in an afternoon, that gap is not a footnote. It is the reason the year-over-year comparisons in the chapter are more interesting than the absolute levels.

## 85 percent of robots.txt requests returned 200, and 13 percent returned 404

Before any directive can matter, the file has to answer. The chapter reports that a 200 status code was returned for 84.9 percent of both desktop and mobile sites, a 404 for 13.3 percent of desktop and 13.2 percent of mobile, a timeout for 1.0 and 1.1 percent, a 403 for 0.5 percent of both, and a 500 for 0.1 percent of both. In prose the authors round that to 85 percent of requests returning a valid 200, up from 84 percent in 2024, and a 404 rate that declined to 13 percent from 14 percent.

Those five outcomes are not five degrees of the same thing, and the difference between the last two is the largest single gap in the list. A 404 leaves a crawler free to fetch anything; a 5xx tells a well behaved crawler to treat the whole site as closed. We have written up [why a 404 and a 503 on robots.txt are opposites](https://lantad.co/blog/robots-txt-404-and-503-are-opposites) and will not repeat the mechanism here. What the Almanac adds is scale: on this crawl, roughly one site in eight was publishing no robots.txt at all, and under the Robots Exclusion Protocol that absence is a permission rather than a neutral state.

The chapter also reports that 1.8 percent of desktop sites and 1.7 percent of mobile sites served a completely empty robots.txt, slightly up from 2024, and notes that while most major crawlers treat an empty file as permissive, the standard does not define that behaviour explicitly, which leaves room for inconsistent handling by lesser known bots. That is an accurate reading of the specification and a good example of a case where the safest thing to publish is either a valid file or a clean 404.

On size, the chapter states that only 0.1 percent of sites exceeded the 500 KB parsing cutoff that Google enforces, and concludes that oversized files are an edge case. The finer size buckets are worth skipping: the chart description gives them in bytes while the surrounding prose gives them in kilobytes, so the units in that one figure cannot be pinned from the published text, and a number you cannot pin is a number this blog does not quote. The parsing cutoff itself, and what happens to the rules that fall past it, is covered in [what Google documents about crawl size limits](https://lantad.co/blog/googlebot-reads-the-first-two-megabytes). Worth remembering alongside all of this is that a file you fixed today is not a file a crawler has read today, for the reasons in [how long a robots.txt edit takes to reach a crawler](https://lantad.co/blog/when-a-robots-txt-edit-reaches-a-crawler).

## The wildcard is in 77 percent of files, and it is what an AI crawler reads

The catch-all group is the single most common thing in a robots.txt file. The chapter reports the wildcard user agent in 77.04 percent of desktop files and 77.14 percent of mobile ones, describes that as up slightly from 2024 and up from the mid seventies in 2022, and reads it as site owners preferring broad universal rules over complex bot specific instructions. That reading is fair, and it is also the whole problem.

[RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), the Standards Track specification for the Robots Exclusion Protocol, tells a crawler to find the group whose product token matches its own, using case-insensitive matching, and to obey that group. Where several groups could apply, the most specific match wins. Where none matches, the crawler obeys the group whose user-agent line carries the wildcard, if there is one. Nothing in that algorithm asks what the group was written for. A group written in 2019 to keep a search engine out of a faceted navigation is, in 2025, the group that governs a model training crawler that did not exist when the line was typed.

That mechanism is not new to this blog. We covered it from the Google side in [what happens when crawl budget advice reaches an AI crawler](https://lantad.co/blog/crawl-budget-advice-lands-on-ai-crawlers), and from the opposite direction in [the fourteen Google agents a wildcard does not stop](https://lantad.co/blog/google-fetchers-ignore-the-robots-txt-wildcard). The Almanac supplies the missing quantity. It is one thing to know that an unnamed crawler falls back to the wildcard group and another to know that on roughly three quarters of files a wildcard group is present and on more than nine in ten no group names any given AI crawler at all.

The chapter adds one caution that cuts the other way, and it is a good one. Not every crawler treats the wildcard consistently: the authors note that Google's AdsBot ignores it, and that Applebot falls back to Googlebot rules before applying the wildcard, which makes explicit targeting necessary in certain cases. [Google's own robots.txt documentation](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) is the primary source for the Google half of that. So the wildcard is neither a universal control nor a dead letter. It is a default that most crawlers honour, that a few named exceptions route around, and that most site owners have never revisited since the last time they thought about search engines. None of which changes the layer that actually answers a request, which is the point of [the two layers that decide whether AI can read your site](https://lantad.co/blog/two-layers-decide-if-ai-can-read-your-site).

## The crawlers publishers name most are an ad crawler and an SEO tool

Where a file does name somebody, the names are not the ones the current conversation is about. The chapter reports adsbot-google, Google's advertising crawler, in 9.82 percent of desktop files and 9.51 percent of mobile, up from 9.1 and 8.9 percent in 2024. Next is ahrefsbot at 9.29 percent desktop and 9.50 percent mobile, with ahrefssiteaudit a separate entry at 4.57 and 4.27 percent. Then mj12bot from Majestic at 7.31 and 7.28 percent, googlebot itself at 6.22 and 6.66 percent, and nutch at 5.03 and 4.81 percent. Further down are dotbot, adsbot-google-mobile, pinterest and semrushbot, the last at 3.0 percent desktop and 3.1 percent mobile.

The composition of that list says something plain about why people edit robots.txt. Two of the top four entries are SEO tool crawlers, and the chapter reads that as reflecting the ongoing importance of controlling access by SEO tools, which can generate significant crawl activity. That is a server cost problem, not a content policy problem. The most named crawler on the web is being named because it hits pages hard, not because anyone has a view about what it does with them.

The most striking entry is one that is barely there. The chapter reports that bingbot ranks 22nd among named user agents and appears in less than 3 percent of robots.txt files, and argues that low appearance rates suggest benign neglect rather than approval, since naming a bot at all means a site manager cared enough to control its behaviour. It draws the contrast itself: despite Microsoft's investment in AI and its integration of ChatGPT into Bing, the crawler has not become more prominent in these files. A crawler that feeds a major AI answer product is named less often than a backlink tool.

Read against [the crawlers this scanner tracks](https://lantad.co/tools/ai-crawlers), the pattern is that robots.txt is still mostly a document about crawl load and about search engines, edited by people solving an infrastructure problem. It is being asked to carry a content licensing decision it was not built for, by an audience that mostly has not shown up to make one. The one population that clearly has shown up is publishers, which is consistent with the finding that [the sites blocking AI crawlers are the ones with editors](https://lantad.co/blog/who-blocks-ai-crawlers-splits-by-credibility).

## Every AI crawler token in the data sits below 5 percent

The AI section of the chapter lists ten tokens and not one of them clears 5 percent. On desktop: gptbot at 4.5 percent, petalbot at 4.0, claudebot at 3.6, ccbot at 3.5, google-extended at 3.4, amazonbot at 3.3, facebookbot at 2.9, and perplexitybot, chatgpt-user and meta-externalagent all at 2.8. The mobile figures sit a few tenths lower in every case except petalbot, which is higher at 4.4 percent.

The growth is the part the authors emphasise, and it is real. Gptbot is up from 2.9 percent desktop and 2.7 percent mobile in 2024, an increase they put at roughly 55 percent. Claudebot nearly doubled from 1.9 and 1.6 percent. Ccbot went from 2.7 and 2.4 percent. The chapter calls this one of the fastest expansions of robots.txt directives for named user agents in recent memory, moving from a marginal presence in 2023 to multi-percent adoption by 2025. It also flags a comparability wrinkle worth carrying: 2024's data included broader categories such as anthropic-ai, at 2.0 percent desktop and 1.7 percent mobile, so part of the year-over-year movement is publishers writing more specific tokens rather than more of them.

Which tokens exist to be named is itself a moving target, and it is the reason a percentage here decays faster than most. [OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots) currently documents four separate tokens for four separate jobs, of which the Almanac counts two. We have covered how few vendors publish a full token list in [six of nine vendors publish one crawler token](https://lantad.co/blog/six-vendors-publish-one-crawler-token), how Meta's set grew in [Meta documents five crawler tokens](https://lantad.co/blog/meta-documents-five-crawler-tokens), and why google-extended can be named in a file and never appear in a log in [the crawler tokens that never appear in your logs](https://lantad.co/blog/the-crawler-tokens-that-never-appear-in-your-logs). A file naming three tokens out of a vendor's four is not a policy. It is a policy with holes in it, and the holes are wherever the file is older than the fleet.

None of this says a low number is a wrong number. Plenty of sites want every crawler to read everything, and for them an empty AI section is the correct configuration and the fastest route to [being cited by a generative engine](https://lantad.co/glossary/geo). The finding is narrower: on this crawl, a decision about AI access had been written down explicitly on somewhere between three and five sites in a hundred, and everywhere else the answer is whatever the wildcard group happens to say.

## What to check in your own robots.txt after reading this

The Almanac describes a population. Your file is one row in it, and four checks turn the finding into something local.

Read the whole file and list the groups it contains. Most people know what their file blocks and fewer know which user-agent lines it opens with, which is the part that decides who each rule reaches. If the only group is the wildcard, then whatever that group says is your complete published position on every AI crawler in existence and every one announced next month. That may be exactly what you want. It should be a thing you know. [The robots.txt tester here](https://lantad.co/tools/robots-txt-tester) evaluates a file against named crawler tokens rather than reading it as one blob, which is the difference between checking what you wrote and checking what a specific crawler resolves.

Check the spelling of any token you rely on against the vendor's own documentation, and check it again after a vendor announcement. A token that has been renamed leaves a group matching nothing at all, with no error anywhere, which is the failure written up in [a renamed crawler token leaves your robots.txt matching nothing](https://lantad.co/blog/renamed-crawler-tokens-leave-stale-robots-rules). Matching is case-insensitive under RFC 9309, so capitalisation is not the risk; a stale product name is.

Then confirm the file is not the only thing you are relying on. A robots.txt group is a request, and the response an unauthenticated fetch actually receives is set by your CDN, your firewall and your bot rules. Those two can disagree in both directions, and they frequently do: on one published sample, [sites banning GPTBot in robots.txt served it a 200 anyway](https://lantad.co/blog/gptbot-bans-served-a-200-anyway). Fetching one of your own pages with a crawler's user agent and reading the response is the only way to see which layer won, which is what [a check of what GPTBot sees](https://lantad.co/tools/what-gptbot-sees) does and what any decent measurement of [AI visibility](https://lantad.co/glossary/ai-visibility) has to rest on.

Finally, resist the temptation to fix this by adding a file. The pattern where a new root file gains adoption faster than any crawler operator commits to reading it is well established by now, and the evidence for the best known example is in [what the evidence actually says about llms.txt](https://lantad.co/blog/what-the-evidence-says-about-llms-txt). The Almanac puts valid llms.txt files at 2.13 percent of desktop sites and 2.10 percent of mobile in the same crawl, and notes that 39.6 percent of them carry the marks of being generated by All in One SEO and 3.6 percent by Yoast SEO, which is to say most of them were emitted by a plugin rather than curated by anybody. A wildcard group nobody has read since 2019 and a generated index file nobody fetches are the same failure wearing different clothes.

## Questions and answers

**What share of robots.txt files name GPTBot?**

The 2025 Web Almanac SEO chapter, written from HTTP Archive's July 2025 crawl of 16,213,084 websites and published on 15 January 2026, reports gptbot appearing in 4.5 percent of desktop robots.txt files and 4.2 percent of mobile ones, up from 2.9 and 2.7 percent in 2024. Every other AI crawler token in the chapter sits below that, with claudebot at 3.6 percent desktop and perplexitybot at 2.8 percent.

**If my robots.txt does not name an AI crawler, what governs it?**

The group whose user-agent line carries the wildcard, if the file has one. RFC 9309 tells a crawler to obey the most specific group matching its own product token and to fall back to the wildcard group when no group matches. The Web Almanac reports a wildcard group in 77.04 percent of desktop files, so on most sites an AI crawler is following rules written with something else in mind.

**Which crawler is named most often in robots.txt files?**

Google's advertising crawler. The 2025 Web Almanac reports adsbot-google in 9.82 percent of desktop files and 9.51 percent of mobile, ahead of the SEO crawler ahrefsbot at 9.29 and 9.50 percent, and well ahead of googlebot itself at 6.22 and 6.66 percent. The chapter also notes that bingbot ranks 22nd and appears in under 3 percent of files.

**Has Lantad measured any of this?**

No. Lantad has not crawled robots.txt at web scale and could not produce a figure like this from its own scans. Every number in this post is read from the Web Almanac's published 2025 SEO chapter on 16 August 2026. What this scanner measures is narrower and per site: what a named crawler receives when it requests one of your pages, and how that compares with what a browser renders on the same URL.

---

Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering
crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup.

Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot