BlogFindings
Wildcard robots.txt: the trailing star changed nothing in 168 comparisons
Run against Lantad's shipped robots.txt matcher on 27 August 2026, appending a trailing asterisk to six rule patterns changed the verdict on none of 168 path comparisons. Appending a trailing dollar sign to four patterns changed 36 of 112.
So it was tested. On 27 August 2026 the robots.txt matcher that ships inside this scanner, the same code behind the robots.txt tester on this site, was run over a corpus of 28 distinct paths and a set of rule patterns built to isolate one character at a time. The matcher is not a crawler and this run does not observe one: it is an implementation of the standard, and what it produces is a statement about pattern semantics rather than about the behaviour of any AI crawler that shows up at your origin. That limit is real and the last section is about it. Within it, the numbers are unambiguous, they are reproducible from the repository, and two of them are worth knowing before you next edit a Disallow line.
In short
- Wildcard robots.txt patterns were run through Lantad's shipped matcher on 27 August 2026, and appending a trailing asterisk to six rule patterns changed the verdict on none of 168 path comparisons, because a robots.txt pattern already matches from the first character forward and a trailing star adds nothing to a prefix.
- The dollar sign is the character that changes an answer. Across four pattern pairs and the same 28 paths on 27 August 2026, adding a trailing dollar changed 36 of 112 verdicts, and the pair Disallow: / against Disallow: /$ accounted for 27 of those 36 on its own.
- A query string walks straight past an anchored file extension. Lantad's matcher returned a match for the pattern /*.pdf$ against /a/b/c/d.pdf and no match against /a/b/c/d.pdf?x=1, which is the behaviour Google's robots.txt documentation, Last updated 2026-07-08 UTC, records when it lists /filename.php?parameters under Doesn't match for the pattern /*.php$.
- RFC 9309 section 2.2.2 settles a conflict by length and not by the order the lines are written in: the most specific match found MUST be used, the most specific match is the match that has the most octets, and where an allow rule and a disallow rule are equivalent the allow rule SHOULD be used.
- Lantad's matcher reproduced all 44 match and non-match assertions in the six worked pattern rows of Google's robots.txt documentation on 27 August 2026, with zero disagreements, which is agreement with a published document and not evidence about what any crawler does when it arrives at your site.
| Pattern pair | Character added | Paths compared | Verdicts changed |
|---|---|---|---|
| Six pairs, /fish against /fish* and five more | trailing * | 168 | 0 |
| / against /$ | trailing $ | 28 | 27 |
| /fish against /fish$ | trailing $ | 28 | 5 |
| /*.php against /*.php$ | trailing $ | 28 | 3 |
| /blog/ against /blog/$ | trailing $ | 28 | 1 |
Wildcard robots.txt: what * and $ actually match
There are exactly two pattern characters, and the standard is short about both. RFC 9309, the Robots Exclusion Protocol, published September 2022 on the Standards Track, says in section 2.2.3 that crawlers MUST support the following special characters, and then gives a three row table. The hash designates a line comment. The dollar Designates the end of the match pattern. The asterisk Designates 0 or more instances of any character. That is the whole vocabulary. There is no character class, no optional marker, no alternation and no way to say one or more.
Google's own robots.txt documentation, carrying Last updated 2026-07-08 UTC, states the same pair in its own words: the asterisk designates 0 or more instances of any valid character and the dollar designates the end of the URL. The two descriptions of the dollar are not quite the same sentence, and the difference is worth holding on to, because the standard anchors the end of the pattern and Google anchors the end of the URL. In practice both produce the same behaviour, which the run below confirms against Google's worked examples, but only one of those phrasings tells you that the query string is part of what gets matched.
The piece that surprises people is the rule underneath the wildcards rather than the wildcards themselves. RFC 9309 section 2.2.2 says the matching MUST start with the first octet of the path. A robots.txt pattern is anchored at the start of the path whether or not you write anything to anchor it. It is not a substring search and it is not a regular expression. Disallow: /private already covers /private-beta/roadmap.html, without a wildcard, because the pattern is a prefix and the path begins with it. That single fact is what makes one of the two wildcards redundant almost everywhere it is written.
None of this is about which crawler is well behaved. The pattern grammar is the same for every agent that reads the file, including the ones this site has shown are decided somewhere else entirely: fourteen named Google agents are not governed by the group you head with an asterisk, which is a user agent question rather than a path question. This post is only about the path half. Our own crawler's rules are published at the LantadBot page and follow the same grammar.
-
*MUST support Designates 0 or more instances of any character. Section 2.2.3 lists it as a character crawlers MUST support. -
$MUST support Designates the end of the match pattern. Section 2.2.3, same table, same requirement level. -
Start of pathMUST The matching MUST start with the first octet of the path, so every pattern is anchored at the front whether or not you write it. -
Letter caseSHOULD The matching SHOULD be case sensitive, which is a recommendation rather than a requirement, and Google states the path value is case-sensitive. -
A literal * in a URLSHOULD encode Section 2.2.3 says crawlers SHOULD use percent encoding to match a special character verbatim, giving /path/file-with-a-%2A.html as the example.
Why does a trailing asterisk change nothing?
Take six pairs of patterns where the only difference is an asterisk on the end: /fish against /fish*, / against /*, /blog against /blog*, /*.php against /*.php*, /private against /private*, and /a against /a*. Run each pair over the same 28 distinct paths and count how many times the two members of a pair disagree about whether a path matches. On 27 August 2026 that produced 168 comparisons and zero differences. Not a small number of differences. None.
The reason is the anchoring rule from the previous section. A pattern matches when it matches a prefix of the path, so /fish has already matched /fishheads/yummy.html by the time the pattern runs out of characters. Appending an asterisk asks the matcher to also accept zero or more further characters at a point where it was going to accept anything anyway. The asterisk is doing real work in the middle of a pattern, where /a/*/draft skips a segment you cannot name in advance. On the end it is a no-op, and it has been a no-op in every published description of the format.
Google's documentation demonstrates this without commenting on it. Its example table gives /fish and /fish* as two separate rows, and the two rows carry an identical list of six matching URLs and an identical list of four non-matching URLs. Anyone reading the table closely is being shown that the trailing star buys nothing, which is a strange thing to have to infer from a duplicated row rather than read in a sentence.
This matters more than a style note because of what the character costs elsewhere. The pattern length is what decides a conflict, a point the next section covers, so a trailing asterisk makes a rule one character more specific without making it match anything more. Write Disallow: /admin* alongside Allow: /admin and you have created a precedence difference out of a character that has no matching effect at all. The wider habit is visible in the field: the 2025 Web Almanac count this site has written up shows the catch-all user agent in 77.04 percent of desktop robots.txt files, so wildcards get typed by reflex, and reflexes are where redundant characters come from. The related failure of a rule that has quietly stopped matching anything at all is the subject of the post on renamed crawler tokens leaving stale robots rules. Both are cases of a file that looks precise and is not, which is the class of problem our methodology is built to surface.
The dollar anchor is the one that changes the answer
The same test on the other character gives the opposite result. Four pairs differing by a trailing dollar, over the same 28 paths, produced 112 comparisons and 36 differences on 27 August 2026. The distribution is as informative as the total. The pair / against /$ accounted for 27 of the 36 by itself, which is the whole corpus minus the root: Disallow: / blocks a site and Disallow: /$ blocks one URL. The pair /blog/ against /blog/$ differed on exactly one path, because anchoring a directory pattern removes its contents and leaves the directory. The remaining two pairs sit in between, at five and three.
Inside the /*.php pair is the case worth carrying away. Lantad's matcher returned a match for the pattern /*.pdf$ against the path /a/b/c/d.pdf, and no match for the same pattern against /a/b/c/d.pdf?x=1. Adding a query string to a URL takes it out of the rule. This is not an implementation quirk and it is not a disagreement with anybody: Google's robots.txt documentation, Last updated 2026-07-08 UTC, lists /filename.php?parameters explicitly under Doesn't match for the pattern /*.php$, alongside /filename.php/ and /filename.php5. The documented behaviour and the shipped behaviour agree, and both of them mean the same thing for a site owner.
That is worth stating plainly, because the anchored extension pattern is written for exactly one purpose. Somebody who does not want their PDF reports pulled into a training corpus writes Disallow: /*.pdf$ and considers the job done. If any of those documents is ever reached through a link carrying a tracking parameter, a version marker or a download flag, the rule that was supposed to cover them does not apply to that URL. The fix is not subtle once you know: drop the dollar, or add a second rule for the query form. Knowing is the entire difficulty, and it is the kind of thing that only shows up when somebody actually runs the pattern rather than reading it.
Two neighbouring limits are worth knowing at the same time, because they also decide what a rule reaches. Both RFC 9309 section 2.5 and Google's page put the parsing floor at 500 kibibytes, so a very long file can have its tail ignored, and Google's separate documentation on how much of a page is read is the subject of the post on the first two megabytes. The habit of writing broad Disallow lines for crawl efficiency reasons, and what that does when AI crawlers read the same file, is covered in the post on crawl budget advice landing on AI crawlers. If you want to see which of your own URLs a given agent can reach, what GPTBot sees resolves it against the live file.
Disallow: /*.pdf$ covers this
- /a/b/c/d.pdf
- Matcher verdict: match, so the rule applies
- The path ends at .pdf, so the anchor is satisfied
- This is the URL the rule was written for
The same rule does not cover this
- /a/b/c/d.pdf?x=1
- Matcher verdict: no match, so the rule does not apply
- The path no longer ends at .pdf, it ends at 1
- Google lists /filename.php?parameters under Doesn't match
Which rule wins when two patterns match the same URL?
Wildcards make overlapping rules easy to write, so the tie break is part of knowing what a pattern does. It is not the order of the lines. RFC 9309 section 2.2.2 states that the most specific match found MUST be used, that the most specific match is the match that has the most octets, and that if an allow rule and a disallow rule are equivalent then the allow rule SHOULD be used. Google's page describes the identical procedure in operational language: crawlers use the most specific rule based on the length of the rule path, and in case of conflicting rules, including those with wildcards, Google uses the least restrictive rule.
Length, then allow. Running that through the shipped matcher on 27 August 2026 against a small file containing Disallow: /blog, Allow: /blog/, Disallow: /search and Allow: /search/help gives results that look inconsistent until you count characters. The path /blog came back blocked, decided by disallow /blog at 5 characters, because the pattern /blog/ does not match a path with no trailing slash. The path /blog/post.html came back allowed, decided by allow /blog/ at 6 characters, which outranks the 5 character disallow that also matched it. One character of difference in the rule flips the verdict for everything underneath.
The /search pair shows the same arithmetic with a wider gap and one further consequence. The path /search was blocked by disallow /search at 7 characters. The path /search/help was allowed by allow /search/help at 12 characters. So was /search/help/advanced, decided by the same 12 character allow rule, because an Allow is a prefix pattern like any other and opens the whole subtree beneath it rather than the single URL named. An Allow line written to make one page reachable makes every page below it reachable too.
Two things follow that are easy to miss. Because specificity is measured in characters, a longer path is a stronger rule regardless of intent, which is why the redundant trailing asterisk from the second section is not merely untidy. And because no matching rule means allowed, the absence of a rule and the presence of a matching Allow produce the same outcome by different routes, which matters when you are reading somebody else's file. RFC 9309 puts it directly: if no match is found amongst the rules in a group for a matching user-agent, or there are no rules in the group, the URI is allowed. The same logic decides the status code cases, where a 404 on robots.txt allows every crawler and a 503 blocks them all. A file can also be perfectly well formed and still fail at its purpose, which is the finding in the post where a robots.txt that blocks every citation crawler still graded B. To check a specific rule against a specific agent rather than reasoning about it, the tester does the character counting for you.
Flow: Path plus query string to Pick the group by user agent; Pick the group by user agent to Collect every matching pattern; Collect every matching pattern (one or more) to Most octets wins; Collect every matching pattern (zero) to No rule matched: allowed; Most octets wins (same length) to Equal length: allow wins; Most octets wins (allow rule) to Allowed; Most octets wins (disallow rule) to Blocked; Equal length: allow wins to Allowed.
What this run does not tell you
The measurement here is narrow on purpose and it is worth naming the edges. What was run is the pattern matcher that ships in this scanner, over a corpus of 28 paths chosen to isolate one character at a time. Those paths are not a sample of the web and no percentage in this post describes real sites. The 44 assertions the matcher reproduced come from Google's published example table, so a perfect score is agreement with a document. It says the implementation follows the written standard. It says nothing about what Google's live parser does today, and less than nothing about the parsers at vendors who publish no example table at all.
The larger gap is between a rule and an outcome. A robots.txt file is a request, and this site keeps finding the distance between the request and what happens next. A block did not stop 12 of 18 AI chatbots from retrieving the page. Of 592 sites that banned GPTBot in robots.txt, 234 served it a 200 anyway. Reading a pattern correctly tells you what you asked for. It does not tell you what you got, and anyone using robots.txt as a control rather than as a statement of preference is relying on a courtesy.
There is also a question this post does not touch: whether the agent reading your file is the one you think. Rules are selected by product token, and a token is not the string in your access log, which is the count in the post on 6 of 15 crawler tokens publishing a User-Agent to match. Whether a matching header is honest is a separate problem again. Path patterns are the last step of a chain whose earlier links are weaker than the pattern grammar is.
What the run does support is a short list of things to check in your own file, none of which need a tool to spot. Delete trailing asterisks, because they changed nothing in 168 comparisons and they inflate a rule's precedence for free. Look at every anchored extension rule and decide whether the query string form matters to you. Count characters, not lines, when two rules overlap. And keep in mind that an Allow opens the subtree beneath it. If you would rather see the verdicts than reason about them, the crawler directory lists the agents this scanner tracks and the crawlability study is where the site publishes what it finds across real pages rather than constructed ones.
- The shipped matcher follows the documented grammar 44 of 44 assertions in the six worked rows of Google's example table were reproduced with zero disagreements.
- A trailing asterisk is redundant in this implementation 168 comparisons across six pattern pairs and 28 distinct paths produced no difference.
- An anchored extension excludes the query string form /*.pdf$ matched /a/b/c/d.pdf and did not match /a/b/c/d.pdf?x=1, matching what Google's table records for /*.php$.
- What real robots.txt files on the web contain The 28 paths were constructed to isolate single characters. Nothing here is a sample of live sites.
- What any vendor's live parser does Agreement is with published documentation read on 27 August 2026, not with observed crawler behaviour.
- Whether a crawler obeys the rule at all Compliance is a separate question and this site has published cases where a block did not prevent retrieval.
Lantad
Published .
Almost every robots.txt file that tries to be precise reaches for a wildcard, and the two characters available do very different amounts of work. One of them is written constantly and, in the test below, never changed a single verdict. The other is written rarely and changes verdicts often, including in one case that quietly undoes the rule people most often write it for. The distinction is not a matter of opinion or of crawler politics. It falls out of what a robots.txt pattern is, and it is testable in a second.
Common questions
Does a wildcard in robots.txt do anything at the end of a rule?
Not in this test. Appending a trailing asterisk to six rule patterns changed the verdict on none of 168 path comparisons when run through Lantad's shipped matcher on 27 August 2026. A robots.txt pattern is already anchored at the start of the path and matches a prefix, so the end of the pattern accepts any remaining characters with or without a star. An asterisk in the middle of a pattern, such as /a/*/draft, does real work.
Why does Disallow: /*.pdf$ not block a PDF with a query string?
Because the dollar anchors the end of the match and the URL no longer ends in .pdf once a query string is appended. Lantad's matcher returned a match for /*.pdf$ against /a/b/c/d.pdf and no match against /a/b/c/d.pdf?x=1 on 27 August 2026. Google's robots.txt documentation, Last updated 2026-07-08 UTC, records the same thing for the equivalent pattern by listing /filename.php?parameters under Doesn't match for /*.php$.
If an Allow and a Disallow both match a URL, which one wins?
The longer pattern, counted in characters, and where the two are the same length the Allow wins. RFC 9309 section 2.2.2 states that the most specific match found MUST be used, that the most specific match is the match that has the most octets, and that an allow rule SHOULD be used where an allow and a disallow are equivalent. The order the lines appear in the file has no effect.
Is a robots.txt path case sensitive?
RFC 9309 section 2.2.2 says the matching SHOULD be case sensitive, which is a recommendation rather than a requirement, and Google's documentation states that the path value is case-sensitive. Lantad's matcher treats paths case sensitively: on 27 August 2026 the pattern /private did not match /Private and the pattern /*.php did not match /x.PHP. Six of the 44 URLs in Google's own example table exist to make this point.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
Generative Engine Optimization (GEO)
Generative Engine Optimization (GEO) is the practice of structuring and publishing content so that generative AI engines can fetch it, read it in full, and cite it when they answer a question.
Fourteen Google agents that a robots.txt wildcard does not stop
Google's crawling documentation sorts everything it sends at a website into three classes. Nine user-triggered fetchers generally ignore robots.txt, and five special-case crawlers ignore the global user agent group, so fourteen named Google agents are not decided by the group you head with an asterisk.
A 404 on robots.txt allows every crawler, a 503 blocks them all
RFC 9309 gives a missing robots.txt and a broken robots.txt opposite meanings: a 4xx lets a crawler access any resource on the server, a 5xx obliges it to assume complete disallow. Google's documentation carves 429 out of that rule. Lantad's code does not.
GPTBot is named in 4.5 percent of robots.txt files, the wildcard in 77
The 2025 Web Almanac, written from HTTP Archive's July 2025 crawl of 16,213,084 websites and published on 15 January 2026, reports the catch-all user agent in 77.04 percent of desktop robots.txt files and gptbot in 4.5 percent. Google's advertising crawler is named more than twice as often as any AI crawler.
A robots.txt that blocks every citation crawler still grades B
Lantad's stored fixture for a site that disallows the six AI crawlers capable of citing it scores 78.8 out of 100 and grades B, because access is weighted at 25 points and only 20 of them are lost. An audit of our own scale, read out of our own repository.
