BlogFindings
WordPress AI crawlers: 16 of 24 sites named none in robots.txt
Lantad requested the home page and the robots.txt of 120 hostnames on 8 September 2026. 91 home pages answered HTTP 200, and 24 of those carried a WordPress signal. All 24 served a parseable robots.txt, and 16 of the 24 name none of the 15 AI crawler tokens this scanner evaluates.
On 8 September 2026 we requested two things from each of 120 hostnames: the home page, and /robots.txt. Both requests were sent with the scanner's own user agent, redirects followed, one attempt each. 91 of the home pages answered HTTP 200. 24 of those 91 carried at least one WordPress signal in the response, and every one of those 24 served a robots.txt our parser could read. 16 of the 24 name none of the 15 AI crawler tokens the crawler directory on this site lists. The sample was picked before any platform was known and the detection sorted it afterwards, which matters because it means nobody chose these sites for being WordPress sites.
In short
- WordPress AI crawlers are governed by a file WordPress will write for you, and on 8 September 2026 Lantad found 16 of 24 WordPress sites naming none of the 15 AI crawler tokens in this scanner's registry.
- The do_robots() reference on developer.wordpress.org, last modified 20 August 2026, shows a default body of three lines: a wildcard user-agent, a disallow of the admin path, and an allow of admin-ajax.php. The function was introduced in WordPress 2.1.0 and names no crawler at all.
- 13 of the 24 WordPress sites measured on 8 September 2026 still carried the Disallow: /wp-admin/ line and 10 of those also carried Allow: /wp-admin/admin-ajax.php, so the default text was still visible in more than half the sample.
- Four WordPress ecosystem sites in the sample, being yoast.com, wpbeginner.com, kinsta.com and wpengine.com, named zero AI crawler tokens on 8 September 2026. yoast.com declared 14 user-agent groups and every one of the 13 named crawlers is an SEO crawler, a scraper or a security scanner.
- Running WordPress was not itself the disadvantage: 8 of the 24 WordPress sites named at least one AI crawler token, against 14 of the 66 non-WordPress hosts that served a parseable file on the same day.
| Step | Hosts | Note |
|---|---|---|
| Hostnames requested | 120 | Home page and robots.txt, one attempt each |
| Home page answered HTTP 200 | 91 | 29 did not: 24 returned 403, 2 returned 503, 2 returned 429, 1 returned 404 |
| Carried a WordPress signal | 24 | A floor rather than a count, for the reasons in the last section |
| Served a parseable robots.txt | 24 | All 24 answered HTTP 200 with a readable file |
| Named at least one AI crawler token | 8 | Out of the 15 tokens in this scanner's registry |
| Named none | 16 | Every AI crawler falls through to the wildcard group |
How do I control WordPress AI crawlers in robots.txt?
The first thing to know is that there may not be a file to edit. WordPress core carries a function called do_robots(), and its reference page on developer.wordpress.org, published on 25 April 2014 and last modified on 20 August 2026, shows the body it composes. It is three lines: a user-agent line set to the wildcard, a disallow built from the site's admin path, and an allow for admin-ajax.php underneath it. The page records the function as introduced in WordPress 2.1.0. A second core method, WP_Sitemaps::add_robots(), introduced in WordPress 5.5.0, appends a Sitemap line pointing at the sitemap index when the site is publicly accessible. That is the whole of what core writes, and the URL is developer.wordpress.org/reference/functions/do_robots/ if you want to read the source yourself.
Read that text against what a crawler actually does with it. Under RFC 9309, a crawler looks for a group headed by its own product token and falls through to the wildcard group only when no group names it. A file whose only group is the wildcard therefore gives GPTBot, ClaudeBot and PerplexityBot precisely the same answer it gives a feed reader, and that answer is governed by rules aimed at an admin directory. Nothing in it refuses an AI crawler and nothing in it invites one. It is a file about wp-admin.
That is not a WordPress failing so much as a dating problem, and the same problem shows up wherever a platform composes the file for you. We found it on Shopify storefronts when we measured six real sites whose robots.txt named no AI crawler at all, and the robots.txt tester on this site exists because the gap between what a file says and what a person believes it says is where most of the surprises live. What a generated default cannot do is change when the crawler population changes, which is the argument for the opposite approach, where a platform regenerates your token list on its own schedule.
One limit belongs here rather than at the end, because it changes how you should read the rest of this post. The reference page describes what do_robots() outputs. It does not state what happens when a physical robots.txt file exists at the document root, and we did not test that, so nothing below claims which one wins on any particular install. What we measured is the file each hostname actually served, whatever produced it.
Flow: GET /robots.txt to do_robots() composes 3 lines; do_robots() composes 3 lines to add_robots() appends Sitemap; add_robots() appends Sitemap to One wildcard group; One wildcard group to GPTBot looks for its own group; GPTBot looks for its own group (no match) to No named group, so wildcard applies.
How many WordPress sites name an AI crawler in robots.txt?
16 of 24 name none. The eight that do name at least one are techcrunch.com, variety.com, rollingstone.com, fortune.com, aljazeera.com, amnesty.org, css-tricks.com and sitepoint.com, and the depth varies a great deal between them. variety.com and rollingstone.com each name 11 of the 15 tokens across 26 user-agent groups. css-tricks.com names 8 across 14 groups. fortune.com names 2, being Google-Extended and CCBot, and amnesty.org names 1, being CCBot alone. Six of the eight are publishers, which is the group with the most direct commercial reason to have had the argument internally.
The 16 that name none are not uniformly thin files. They run from 127 bytes at problogger.com to 2,020 bytes at searchenginejournal.com, and searchenginejournal.com spends those bytes on 12 user-agent groups without any of them being an AI crawler. 11 of the 24 sites declare exactly one user-agent group, and all 11 of those name none, which is the expected result: a single wildcard group is what a file looks like when nobody has written rules about anybody.
The frequency order across the eight is worth recording because it does not match the order the vendors would predict. CCBot appears on 7 of the 24 sites, GPTBot and Google-Extended on 6 each, ChatGPT-User, ClaudeBot, anthropic-ai and Bytespider on 5 each, PerplexityBot on 4, Applebot-Extended on 3, and OAI-SearchBot and Meta-ExternalAgent on 2 each. CCBot leading is a dating signal rather than a preference: Common Crawl predates the current generation of AI crawlers, so it is the token most likely to be in a file somebody copied years ago. Four of the 15 tokens, being Claude-SearchBot, Claude-User, Perplexity-User and Amazonbot, appear on none of the 24 sites.
Two of those absences matter more than the raw count. Naming ClaudeBot without naming Claude-SearchBot and Claude-User is a decision about training that leaves the two retrieval bots governed by the wildcard, and the same split applies to OpenAI, where GPTBot appears six times and OAI-SearchBot twice. OpenAI's crawler documentation treats those as separate bots doing separate jobs, which is also why the guide to what to allow for ChatGPT specifically has to be written as more than one decision. We saw the same shape when we measured whether the answer changes with the agent you ask about.
| Site | Bytes | User-agent groups | AI tokens named |
|---|---|---|---|
| variety.com | 1,283 | 26 | 11 |
| rollingstone.com | 1,264 | 26 | 11 |
| techcrunch.com | 710 | 14 | 8 |
| css-tricks.com | 1,197 | 14 | 8 |
| aljazeera.com | 1,834 | 10 | 6 |
| sitepoint.com | 2,853 | 7 | 3 |
| fortune.com | 1,774 | 8 | 2 |
| amnesty.org | 431 | 3 | 1 |
The WordPress default is still visible in half the files
The two lines core composes leave a fingerprint, and counting it tells you how much of each file is a decision rather than an inheritance. 13 of the 24 sites carry Disallow: /wp-admin/ and 10 of those 13 also carry Allow: /wp-admin/admin-ajax.php, which is the pair do_robots() emits together. 21 of the 24 carry a Sitemap line, which is what add_robots() appends. So on 10 of 24 sites the exact default pair survives verbatim, on 3 more the disallow survives without its allow, and on 11 there is no admin line at all.
docker.com is the clearest single case, because its file opens with Allow: /wp-admin/admin-ajax.php and Disallow: /wp-admin/ and then runs to 1,033 bytes of hand-written rules about thank-you pages, pricing paths and a style guide. Somebody worked carefully on that file. The work went into keeping marketing dead ends out of a search index, and the AI crawler question was never on the page they were editing. wpengine.com is the same shape at 1,805 bytes: a crawl-delay, the admin pair, and a long list of calendar and tag paths, with a comment on one rule reading that the rationale is unknown.
None of this makes the files wrong. A file that says nothing about AI crawlers is a permissive file, and permissive is a legitimate position and probably the right one for most sites that want to be read. The point is narrower and worth stating exactly: on these 10 sites the permission was not granted, it was inherited from a default written for a different problem. That distinction is invisible in any tool that reports allowed or disallowed, which is one reason how this scanner reaches a verdict is written down rather than implied, and why a trailing wildcard changing no verdict across 168 comparisons was worth publishing as a finding in its own right.
There is a second file in this conversation that behaves the same way. When we requested /llms.txt and /robots.txt from 200 hostnames we found that 19 of the 23 sites publishing an llms.txt said nothing about AI in their robots.txt, which is the same gap in a different direction: effort spent on the new file, none on the old one. The llms.txt glossary entry sets out what that file is and is not for.
Yoast, WP Engine and Kinsta name no AI crawler either
Four sites in the sample sell WordPress expertise: yoast.com, which publishes the most installed WordPress SEO plugin, wpengine.com and kinsta.com, which are managed WordPress hosts, and wpbeginner.com, which is a WordPress tutorial publisher. All four served a parseable robots.txt on 8 September 2026 and all four named zero of the 15 AI crawler tokens.
yoast.com is the one worth reading closely, because it disproves the easy explanation. Its file is 927 bytes and declares 14 user-agent groups, so this is not a site that never got round to writing rules. The 13 named groups are BLEXBot, Barkrowler, CazoodleBot, Gigabot, Go-http-client, Node/simplecrawler, Nuclei, PetalBot, Riddler, WikiDo, Zoominfobot, dotbot/1.0 and magpie-crawler. Every one is an SEO crawler, a scraper, a generic HTTP client or a security scanner. The file also opens with a comment linking a post explaining why it looks the way it does, and it carries a Schemamap field alongside its Sitemap line, so it is a maintained and deliberately unusual document. It is aimed at the traffic somebody decided was worth refusing, and AI crawlers were not on that list.
The three others are shorter and land in the same place. wpbeginner.com is 601 bytes in a single group, kinsta.com is 523 bytes in a single group, and both carry the core admin pair intact. wpengine.com carries the pair too. None of the four refuses an AI crawler and none of the four names one, so every AI crawler reaching any of them is admitted by a wildcard rule that was not written about it.
Now the finding that cuts the other way, because leaving it out would make this post a worse piece of evidence. WordPress was not the disadvantage here. 8 of the 24 WordPress sites named at least one AI crawler token, which is a third of them, against 14 of the 66 non-WordPress hosts that served a parseable file on the same day, which is closer to a fifth. On this sample WordPress sites were more likely to have had the conversation, not less. What the platform contributes is a floor rather than a ceiling: a file that exists and says nothing, on sites where otherwise there might have been no file to serve. Whether the crawlers honour any of it is a separate question, and a controlled study found six of ten assistants never requesting robots.txt at all.
| Site | Bytes | Groups | AI tokens | Core admin pair |
|---|---|---|---|---|
| yoast.com | 927 | 14 | 0 | Absent |
| wpengine.com | 1,805 | 1 | 0 | Present |
| wpbeginner.com | 601 | 1 | 0 | Present |
| kinsta.com | 523 | 1 | 0 | Present |
What this measurement does not show
The detection is a floor and should be read as one. WordPress can be identified from outside only by what the response leaks, and the leaks are removable. Of the 24 sites here, 24 exposed a /wp-content/ or /wp-includes/ path in the HTML, 16 carried a wp-json reference, and only 5 published a generator meta tag naming WordPress with a version, those being WordPress 6.9.7 at techcrunch.com, variety.com and rollingstone.com, and WordPress 7.0.4 at docker.com and copyblogger.com. A site that strips those markers, which security guidance routinely recommends, is a WordPress site this count records as not WordPress. 24 is the number we could prove, not the number that exist.
29 hostnames dropped out before they could be classified at all, and how they dropped out is itself a result. 24 of the 29 returned HTTP 403 to a request carrying a declared bot user agent, among them bbc.co.uk, theguardian.com, reuters.com, bloomberg.com and wired.com. Two returned 503, two returned 429 and one returned 404. That says something about how a self-identifying crawler is treated at the edge and nothing whatever about how GPTBot is treated, since we did not send that user agent and would not: the user agent we send and the addresses we send from are published for exactly this reason, and a user agent is a claim rather than an identity whoever sends it.
Three further limits. We read the file each hostname served and did not test whether a physical robots.txt or a plugin produced it, so no claim here attributes any specific file to WordPress core rather than to a plugin or a hand edit. We requested one hostname per site, and a previous measurement found three of six sites serving a different file on a second hostname, so a www or a subdomain may hold something else. And token matching is literal: a file naming a crawler with an unusual character can fail to match anything, which is what happened when we found a token spelled with a non-breaking hyphen that no crawler can match.
Finally, permission is not readability and this post measures only permission. A file that admits GPTBot says nothing about whether the page returns text before any script runs, which is a separate measurement entirely, and nothing about whether a crawler that is allowed in ever arrives. Our standing measurements keep finding the expensive failures further down the stack than the robots layer, and this sample gives no reason to think WordPress is an exception. Naming a token in a file only decides the argument you are having on purpose.
-
What each hostname servedEstablished The bytes returned for /robots.txt on 8 September 2026, and which of 15 AI crawler tokens appear in them. -
WordPress detectionFloor only 24 sites leaked a wp path, 16 a wp-json reference, 5 a generator tag. Stripped markers read as not WordPress. -
29 hosts unclassifiedExcluded 24 returned 403 to a declared bot agent, 2 returned 503, 2 returned 429, 1 returned 404. -
Which component wrote the fileNot established Core, a plugin or a physical file are indistinguishable from outside. We did not test precedence. -
Crawler behaviourNot established Permission is a statement by the site. Obedience appears only in the site's own access log.
Lantad
Published .
Most advice about WordPress AI crawlers starts by telling you to edit your robots.txt. That skips the part that decides the outcome on a large number of sites, which is that WordPress has an opinion about your robots.txt before you do. The platform ships a function that composes the file, it has done so since 2007, and the text it composes was written for search engine crawlers a decade before any of the AI crawlers in this scanner's registry existed.
Common questions
Does WordPress block AI crawlers by default?
No. The default body that do_robots() composes is a wildcard user-agent group with a disallow of the admin path and an allow for admin-ajax.php, and WordPress 5.5.0 added a Sitemap line. None of that names an AI crawler, so every AI crawler falls through to the wildcard group and is admitted anywhere the admin rules do not reach. Of the 24 WordPress sites measured on 8 September 2026, 16 named no AI crawler token at all.
How do I control WordPress AI crawlers without a plugin?
By naming the crawler token in a group of its own, because a token with no group of its own is governed by the wildcard. RFC 9309 sends a crawler to the wildcard group only when no group matches its product token. Which tokens to name is a policy question this post does not answer, and the tokens themselves are published by each vendor rather than by WordPress.
Is a WordPress site worse for AI visibility than a site on another platform?
Not on this sample. 8 of 24 WordPress sites named at least one AI crawler token on 8 September 2026, against 14 of the 66 non-WordPress hosts that served a parseable file the same day, so the WordPress share was higher. The platform supplies a file that exists and says nothing, which is a floor rather than a handicap.
Why did 29 of the 120 hostnames not produce a result?
Their home pages did not return HTTP 200 to a request carrying a declared bot user agent on 8 September 2026. 24 returned 403, 2 returned 503, 2 returned 429 and 1 returned 404. That is a fact about how those edges treat a self-identifying crawler, and it implies nothing about how they treat any AI crawler, whose user agents we did not send.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
Generative Engine Optimization (GEO)
Generative Engine Optimization (GEO) is the practice of structuring and publishing content so that generative AI engines can fetch it, read it in full, and cite it when they answer a question.
robots.txt testen: six files, and not one AI crawler named
One GET per hostname on 5 September 2026 for the robots.txt of the six real sites this scanner captured on 15 July 2026. All six answered HTTP 200. Between them the files declare 48 user-agent groups, and not one of those groups names any of the 15 AI crawler tokens this scanner evaluates.
GPTBot detection: 6 of 15 crawler tokens publish a User-Agent to match
A robots.txt product token and an HTTP User-Agent header are two different strings, and only the second one is what your access log holds. Read at nine vendors' own documentation on 27 August 2026, 6 of the 15 tokens in Lantad's crawler registry carry a complete User-Agent string that can be matched literally.
Check for robots.txt: three of six sites served a different file
Fetched on 2 September 2026 across the 14 hostnames belonging to the six real sites this scanner captured on 15 July 2026, one request per hostname returned eight distinct files. On three of the six sites the second hostname did not serve the file that governs the captured page, and on one of those it returned nothing at all.
