BlogFindings
AI crawler 405 error: 13 of 1,099 sites answered a HEAD request with 405 while GET returned 200
Lantad sent a HEAD request and then a GET to the home page of all 1,419 hostnames in this repository's two committed corpus seed files on 6 November 2026, as LantadBot/1.0 with redirects followed and no JavaScript executed. 1,099 answered the GET with HTTP 200. 1,054 of those answered the HEAD with 200 as well, 45 answered it with something else, and 13 answered it with 405 Method Not Allowed, which RFC 9110 section 9.1 forbids by requiring every general-purpose server to support GET and HEAD. 16 hosts refused HEAD with a 405 in all, 15 of the 16 reproduced it on a second read, and 14 of the 16 sent the same 405 to a desktop browser user agent, so this is server configuration rather than bot defence.
That asymmetry is what makes the error hard to find from the inside. Lantad sent a HEAD request and then a GET to the home page of all 1,419 hostnames in this repository's two committed corpus seed files on 6 November 2026, every request sent as LantadBot with redirects followed and no JavaScript executed. The HEAD went first in every case, deliberately, so that a refusal could not be blamed on a GET that preceded it. 1,099 hosts answered the GET with HTTP 200, and 45 of those 1,099 answered the HEAD with something other than 200. 13 answered it with 405, and that is the number this post is about, because unlike a 403 or a 429 it is the one response the specification does not permit.
In short
- An AI crawler 405 error on a page that loads normally in a browser is almost always a server that answers GET and refuses HEAD. Lantad measured 13 of the 1,099 corpus sites that returned HTTP 200 to a GET on 6 November 2026 answering a HEAD request to the same URL with 405 Method Not Allowed.
- RFC 9110 section 9.1 states that all general-purpose servers MUST support the methods GET and HEAD, and that all other methods are OPTIONAL. A 405 on HEAD is therefore not a policy choice the specification leaves open, which is what separates it from a 403 or a 429.
- The 405 is configuration, not bot defence. 14 of the 16 hosts that refused LantadBot's HEAD request on 6 November 2026 returned 405 to a desktop Chrome user agent string as well, and 15 of the 16 reproduced the 405 on a second read the same day.
- 9 of those 16 hosts sent no Allow header at all, which RFC 9110 section 15.5.6 requires an origin server to generate in a 405 response. None of the 7 Allow headers that did arrive named HEAD. amazon.com sent allow: GET, POST, PUT, DELETE, OPTIONS on 6 November 2026, listing five methods and omitting the one the request asked for.
- All 45 divergences sat in strata that build their own stacks. Healthcare accounted for 11 of 93 sites, finance 8 of 91 and ecommerce 7 of 62, while the Wix and Squarespace, Shopify, WordPress, static docs, SaaS marketing and local media strata returned 0 of 228 between them.
| What the HEAD request returned | Sites | Share of 1,099 | What the specification says about it |
|---|---|---|---|
| 200, same as the GET | 1,054 | 95.9 percent | The required behaviour |
| 405 Method Not Allowed | 13 | 1.2 percent | Forbidden: section 9.1 requires HEAD support |
| 403 Forbidden | 10 | 0.9 percent | Permitted, and a refusal the server chose |
| No response at all | 10 | 0.9 percent | Nothing arrived before the 20 second timeout |
| 404, 400, 401, 429, 500, 503 or 555 | 12 | 1.1 percent | Permitted, though none describes a page that loads |
What RFC 9110 requires of a server that receives a HEAD request
Three requirements in RFC 9110, the HTTP Semantics specification published in June 2022, decide whether a 405 on HEAD is a defensible choice or a defect. They are worth quoting rather than summarising, because the difference between them is the difference between a server that is strict and a server that is wrong.
Section 9.1 is the one that settles it: all general-purpose servers MUST support the methods GET and HEAD, and all other methods are OPTIONAL. That sentence is why a 405 on HEAD sits in a different category from the other refusals in the table above. A server may decide to answer a crawler with 403, and this blog has measured plenty that do, reporting that 103 of 1,089 sites served an unknown bot and refused GPTBot. A 403 is a decision about who may read the page. A 405 on HEAD is a statement that the server does not implement a method it is required to implement, addressed to everyone.
Section 9.3.2 defines what the method means: the HEAD method is identical to GET except that the server MUST NOT send content in the response, and it is used to obtain metadata about the selected representation without transferring its representation data, often for the sake of testing hypertext links or finding recent modifications. The same section says the server SHOULD send the same header fields in response to a HEAD request as it would have sent if the request method had been GET, while allowing it to omit fields that are only determined while generating the body. So the specification expects a HEAD response to be a GET response with the body removed, not a different answer.
Section 15.5.6 adds the requirement that most of the hosts measured here missed. The 405 status definition states that the code indicates the method is known by the origin server but not supported by the target resource, and that the origin server MUST generate an Allow header field in a 405 response containing a list of the target resource's currently supported methods. A 405 is meant to be self-documenting: it tells the client what to try instead. A 405 with no Allow header tells the client only that something it asked for is unavailable, and leaves it to guess what.
None of this is specific to AI. It is the baseline every HTTP client is written against, which is exactly why it matters for AI visibility: the tools reading your site were not written for your server in particular.
Flow: Client sends HEAD to Server supports HEAD?; Server supports HEAD? (yes) to Same headers as GET, no body; Server supports HEAD? (no) to Server answers 405; Server supports HEAD? to Section 9.1: it MUST; Same headers as GET, no body to Client has metadata it asked for; Server answers 405 to Section 15.5.6: MUST send Allow; Section 15.5.6: MUST send Allow to Client learns nothing about the page.
How many sites answered a HEAD request with 405?
13 of the 1,099 that answered a GET with 200, and 16 across the whole corpus. The three extra hosts are ones whose GET did not return 200 either: scoutforschools.com answered the GET with 404, wizzair.com answered both the HEAD and the GET with 405, and ozon.ru answered the HEAD with 405 and never completed the GET. Those three are excluded from the headline figure because the claim being tested is narrower and more useful than a raw count: a page that a browser loads normally, refused to a client that asked for its headers.
The 13 are madebynothing.com, nanny.network, zeroentropy.dev, paris.fr, seoul.go.kr, clevelandclinic.org, healthdirect.gov.au, icmr.gov.in, semafor.com, thejakartapost.com, amazon.com, wayfair.com and zooplus.de. They are not obscure. Four are government or public health bodies, three are large retailers, two are news publishers, and one is amazon.com.
A second read of all 48 hosts that diverged on the first pass was taken the same day. 12 of the 13 reproduced the pattern exactly, returning 405 to the HEAD and 200 to the GET on both reads. The exception is seoul.go.kr, whose GET did not complete the second time, so the HEAD 405 held and the comparison did not. Of the 32 divergences that were not a 405, 25 reproduced, which is the expected shape: a 403 or a 429 is frequently a rate limit that moves between reads, and a 405 is a route that does not exist.
Scale is worth stating plainly because it is small. 1.2 percent of the sites that served a crawler a page refused a HEAD request for it with a 405, and 4.1 percent gave the HEAD any answer other than 200. This is not a widespread failure in the way that missing prose parity is, and the posts on this blog that measured 25 of 1,093 home pages passing 2MB or 125 of 1,094 sites answering a nonexistent URL with HTTP 200 describe larger populations. What makes the 405 worth its own measurement is that it is unambiguous. There is no reading of the specification under which it is correct, and no site owner who would choose it if they knew it was happening.
The distribution says where to look. All 45 divergences fell in strata that run their own infrastructure. Healthcare contributed 11 of 93 sites, finance 8 of 91, ecommerce 7 of 62, government 6 of 88 and education 5 of 101. The strata built on hosted platforms contributed almost nothing: the Wix and Squarespace, Shopify, WordPress, static docs, SaaS marketing and local media groups returned zero divergences across 228 sites between them, and the industry SaaS stratum returned zero across 114. One Framer site, one Webflow site and one Bubble site diverged. If your pages are served by a platform, this is not your bug. If your pages are served by a stack your team assembled, with a gateway or an edge rule in front of them, it might be.
Is a 405 on a HEAD request bot defence or server configuration?
This is the question that decides whether the finding means anything, and it has a clean test. If the 405 is a bot defence aimed at an unfamiliar user agent, then changing the user agent should change the answer. If it is server configuration, the answer should not move.
All 16 hosts that returned 405 to LantadBot's HEAD request were asked again with a desktop Chrome user agent string, on the same day, through the same path. 14 of the 16 returned 405 to that request as well. The two that behaved differently are wayfair.com, which answered the browser string with 429 rather than 405, and wizzair.com, which answered it with 200. Everything else, amazon.com and paris.fr and clevelandclinic.org and the rest, refused the browser exactly as it refused the crawler.
That is the finding behind the finding. These servers are not profiling the client. They have a route that handles GET and no route that handles HEAD, and the 405 is what sits underneath when nothing matches. It is the default behaviour of a good many application frameworks and API gateways when a method is not explicitly registered, and it is invisible in testing because nothing a developer does in a browser sends a HEAD request to a page.
It also means the fix is not a bot allowlist. This blog has repeatedly found that the opposite case, where the refusal really is about who is asking, needs a different repair: 79 of 115 sites refused a crawler their own robots.txt allowed, and the reason a user agent string cannot settle the question at all is that a user agent is a claim rather than an identity. A 405 on HEAD is the rarer and simpler problem: the server is not discriminating, it is incomplete.
One consequence is worth drawing out. Because the server is not varying its answer by client, it is also not declaring that it varies. Only 32 of 1,072 home pages declared Vary: User-Agent in an earlier measurement, and a 405 that applies to every client needs no such declaration to be honest. The response is consistent. It is just consistently wrong.
HEAD as LantadBot/1.0
- 16 of 16 answered 405 Method Not Allowed
- 15 of the 16 reproduced the 405 on a second read
- 9 of the 16 sent no Allow header
- amazon.com: allow: GET, POST, PUT, DELETE, OPTIONS
- clevelandclinic.org: allow: GET
HEAD as a desktop Chrome string
- 14 of 16 answered 405 Method Not Allowed
- wayfair.com answered 429 Too Many Requests
- wizzair.com answered 200
- No host became more permissive except wizzair.com
- So the 405 is a missing route, not a bot rule
The Allow header a 405 must carry, and what these 16 sent
RFC 9110 section 15.5.6 gives a server exactly one obligation when it sends a 405: generate an Allow header listing the methods the target resource does support. It is a small requirement and it is the difference between an error a client can act on and an error it can only log.
9 of the 16 hosts sent no Allow header at all. paris.fr, semafor.com, wayfair.com, zooplus.de, madebynothing.com, nanny.network, thejakartapost.com, scoutforschools.com and wizzair.com each returned a bare 405. A client receiving that has been told the method is wrong and given nothing to retry with, which in practice means the URL is treated as unavailable.
The 7 that did send one are more interesting, because every single one of them omitted HEAD. zeroentropy.dev, clevelandclinic.org and healthdirect.gov.au each sent allow: GET. seoul.go.kr and icmr.gov.in each sent allow: GET, POST. ozon.ru sent allow: POST, on a home page. And amazon.com sent allow: GET, POST, PUT, DELETE, OPTIONS, read directly from the response on 6 November 2026: five methods named, including three that write, and not the one the request asked for.
That list is the clearest statement of the defect available, because the server is describing itself. A resource that supports GET supports HEAD under section 9.1, and a server advertising GET without HEAD is advertising a configuration the specification does not allow. It is also a reasonable guess at the cause: these look like method lists assembled by hand or generated from a route table, in both of which HEAD is easy to forget precisely because it is normally handled for free by the layer underneath.
For a client, the practical result is that the cheap question has no cheap answer. Checking whether a page has changed is the job 391 of 1,101 home pages already made impossible by offering nothing to revalidate against, and a 405 on HEAD closes the other route to the same information. Anything that wanted the content type or the length before committing to a download has to download it instead. Across the 45 divergent hosts the GET bodies totalled 15,933,891 bytes, and semafor.com alone served 1,557,854 of them.
| Host | Stratum | GET to the same URL | Allow header sent with the 405 |
|---|---|---|---|
| amazon.com | Ecommerce | 200 | GET, POST, PUT, DELETE, OPTIONS |
| clevelandclinic.org | Healthcare | 200 | GET |
| healthdirect.gov.au | Healthcare | 200 | GET |
| icmr.gov.in | Healthcare | 200 | GET, POST |
| seoul.go.kr | Government | 200 | GET, POST |
| zeroentropy.dev | SPA startups | 200 | GET |
| ozon.ru | Ecommerce | No response | POST |
| paris.fr | Government | 200 | None sent |
| semafor.com | News | 200 | None sent |
| thejakartapost.com | News | 200 | None sent |
| wayfair.com | Ecommerce | 200 | None sent |
| zooplus.de | Ecommerce | 200 | None sent |
| madebynothing.com | Framer | 200 | None sent |
| nanny.network | Bubble and no-code | 200 | None sent |
| scoutforschools.com | SPA startups | 404 | None sent |
| wizzair.com | Travel | 405 | None sent |
What the AI crawler vendors document about HTTP methods
Almost nothing, and that is a finding rather than an absence of one. If you want to know whether GPTBot sends a HEAD request before it sends a GET, the vendor documentation will not tell you.
OpenAI's crawler documentation, read at source on 6 November 2026, names four tokens and never names an HTTP method. It describes crawler behaviour in terms of visiting, crawling and making requests, and says nothing about GET, HEAD or POST. Google's Googlebot documentation, read the same day, names no request method either; it discusses the user agent request header, protocol support and caching, and leaves the method unstated.
The closest thing to a statement from Google is old. A Google Search Central post of 1 November 2011 announcing POST support says we generally advise to use GET for fetching resources a page needs, and this is by far our preferred method of crawling, and advises site owners to prefer GET for fetching resources unless there is a specific reason to use POST. The post does not mention HEAD at all. That is fifteen years old and it is about Googlebot rather than any AI crawler, so it supports one narrow claim only: Google's published preference has long been GET, and HEAD has never been part of the documented conversation.
So Lantad cannot tell you from any vendor's documentation how often an AI crawler sends a HEAD request, and this post does not claim to know. What it measured is the site side, which is the side a site owner controls, and the site side is where the specification applies regardless. A server that refuses HEAD is refusing it to every client: link checkers, uptime monitors, CDNs revalidating a cached copy, the fetchers behind an AI crawler and anything else that would rather ask than download. The unknown is how many of those actually ask. The known is that the answer they get is wrong.
This is the same gap between documentation and behaviour that this blog has recorded in other places. No crawler vendor documents how it responds to the Retry-After header, which 0 of 63 sites that pushed back actually sent, and the handling of redirects had to be measured rather than read, which is why the finding that 878 of 1,322 home pages answered with one came from a scan rather than a specification. Where the documentation is silent, measuring the site is what remains.
-
RFC 9110 section 9.1Requires GET and HEAD All general-purpose servers MUST support the methods GET and HEAD, all other methods are OPTIONAL -
RFC 9110 section 15.5.6Requires Allow on a 405 The origin server MUST generate an Allow header field listing the supported methods -
OpenAI crawler pageNames no method Four tokens documented, no mention of GET, HEAD or POST anywhere on the page -
Google Googlebot pageNames no method Covers the user agent header, protocols and caching, and leaves the request method unstated -
Google post, 1 Nov 2011Prefers GET GET is by far our preferred method of crawling, and the post never mentions HEAD
How do you fix an AI crawler 405 error on your own site?
The diagnosis takes one command, and it is the one command that will not occur to you, because every other way you look at your own site uses GET. Send a HEAD request to a URL you know works and read the status. If it returns 200, there is nothing here to fix. If it returns 405, the route handling that URL does not register HEAD, and the repair is at the layer that decides which methods reach your application.
Four places are worth checking in order, because this measurement suggests the problem lives in front of the application rather than inside it. The first is the edge: a CDN rule, a web application firewall or an API gateway that forwards an explicit list of methods will drop HEAD if HEAD is not on the list. The second is the router in the framework, where a handler registered for GET alone will sometimes answer HEAD with 405 rather than running the GET handler and discarding the body. The third is a reverse proxy or load balancer with a method allowlist. The fourth is the application itself, which is the least likely of the four and the one people check first.
Two things are worth not doing. Do not fix it by making the server answer HEAD with a body, because section 9.3.2 says the server MUST NOT send content in a HEAD response. And do not treat it as a crawler access problem and reach for robots.txt, which governs who may fetch a URL and has nothing to say about methods; the file has enough failure modes of its own without adding one, as the 232 of 1,059 files carrying a defect and the 34 of 1,104 sites that answered a crawler with an HTML page instead of a robots file both show.
What Lantad measured here and what it did not is worth stating exactly, because the scope is narrow. This run tested one URL per host, the home page, with one request per method, on one day, from one network. It did not test interior pages, where a route table is more likely to be uneven than on a home page. It did not test whether any AI crawler sends HEAD requests in practice, because no vendor documents it and this run could not observe it. It did not compare the headers in the HEAD response with those in the GET response for the 1,054 hosts that answered both with 200, so the section 9.3.2 expectation that they should match is untested here. And the scanner itself does not currently send a HEAD request during a scan, so this measurement was made with a direct probe rather than through the pipeline described in the methodology. The AI Visibility Score does not include a HEAD check, and this finding is not an argument that it should until there is evidence that an AI crawler depends on one.
If you want to see what a crawler receives from your pages without the method question attached, what GPTBot sees fetches a URL the way a crawler does and shows you the result, and the robots.txt tester resolves a path for a named token. Neither sends a HEAD request. For that, one line in a terminal is still the whole tool you need, and on the evidence here it is a line that 13 of 1,099 sites would fail today. A page that a browser renders perfectly can still be a page your infrastructure refuses to describe, and the gap between those two facts is exactly the kind of thing nobody finds until somebody asks the cheap question first. The 404 and 503 responses to robots.txt are opposites that most people assume are equivalent; a 405 to a HEAD request is the same class of surprise, arriving one layer down.
| Layer to check | Why it is on the list | Measured in this run? |
|---|---|---|
| CDN, WAF or API gateway method list | An explicit forwarded-methods list that omits HEAD stops the request before the application sees it | No, inferred |
| Framework route registration | A handler registered for GET only, in a router that does not synthesise HEAD from it | No, inferred |
| Reverse proxy or load balancer allowlist | A method allowlist written for GET and POST in nginx, Envoy or an ingress controller | No, inferred |
| Application level method check | The least likely of the four on this evidence, and the one usually checked first | No, inferred |
| Allow header on the 405 itself | Required by RFC 9110 section 15.5.6, so it is worth fixing even while the 405 stands | Yes: 9 of 16 sent none |
| Whether the client is being profiled | A bot rule would answer a browser differently from a crawler, and these servers did not | Yes: 14 of 16 refused Chrome too |
Lantad
Published .
A HEAD request asks a server for a page's headers and nothing else: the status, the content type, the length, the cache validators, with no body attached. It is the cheapest question any client can ask about a URL, and it is the one a tool asks when it wants to know whether a page exists, how big it is or whether it has changed since last time, without paying to download it. If your server answers that question with 405 Method Not Allowed, every client that asks it that way is told the page is not available by that route, while a browser loading the same URL sees nothing wrong at all.
Common questions
What causes an AI crawler 405 error on a page that works in a browser?
A server that answers GET and does not register HEAD. A browser never sends a HEAD request for a page, so the fault is invisible in normal use, and on the 16 corpus hosts Lantad measured on 6 November 2026 the 405 came from the routing layer rather than from bot defence: 14 of the 16 returned the same 405 to a desktop Chrome user agent string.
Is returning 405 to a HEAD request allowed by the HTTP specification?
No. RFC 9110 section 9.1 states that all general-purpose servers MUST support the methods GET and HEAD, and that all other methods are OPTIONAL. That is what separates a 405 on HEAD from a 403 or a 429, both of which the specification permits as decisions a server is entitled to make.
How do I check whether my site refuses HEAD requests?
Send a HEAD request to a URL you know returns 200 to a GET and read the status line. A 200 means there is nothing to fix. A 405 means the route does not register HEAD, and the response should also carry an Allow header listing what it does support, which 9 of the 16 hosts measured on 6 November 2026 did not send.
Do AI crawlers actually send HEAD requests?
Lantad does not know and this post does not claim to. OpenAI's crawler documentation and Google's Googlebot documentation, both read at source on 6 November 2026, name no HTTP request method at all, and a 2011 Google Search Central post states GET is by far its preferred method of crawling without mentioning HEAD. What is measurable is the site side: a server refusing HEAD refuses it to every client that asks.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
AI visibility
AI visibility is whether AI systems can access, read, and faithfully represent your web pages, the prerequisite for being cited in an AI answer.
Do AI crawlers cache your pages? 391 of 1,101 home pages offered nothing to revalidate against
Lantad requested the home page of all 1,419 hostnames in this repository's two committed corpus seed files twice on 6 October 2026 as LantadBot, then asked a third time with a conditional request carrying the validator the first response had supplied. 1,101 answered HTTP 200 with HTML. 391 of those sent neither an ETag nor a Last-Modified header, so a returning crawler had nothing to ask against, and 514 of the 710 that did offer one answered HTTP 304 Not Modified.
Soft 404: 125 of 1,094 sites answered a URL that does not exist with HTTP 200
Lantad asked all 1,419 hostnames in this repository's committed corpus for a path that cannot exist on 20 September 2026. 1,094 answered both that request and their own home page. 936 returned 404, 125 returned HTTP 200, and on a second nonexistent path of a different shape 111 of those 125 returned 200 again.
CDN blocking AI crawlers: 103 of 1,089 sites served an unknown bot and refused GPTBot
Lantad requested the home page of 1,419 hostnames three times each on 26 September 2026, changing nothing between the requests but the user agent string. 1,089 answered a browser with HTTP 200. 103 of those served an unknown bot user agent as well and then refused the GPTBot string, and on 76 of the 103 the site's own robots.txt permits GPTBot at the root. Sorting the refusals by the infrastructure fingerprint on the response puts 32 of 87 Akamai fronted sites in the refused column against 37 of 344 on Cloudflare.
A 404 on robots.txt allows every crawler, a 503 blocks them all
RFC 9309 gives a missing robots.txt and a broken robots.txt opposite meanings: a 4xx lets a crawler access any resource on the server, a 5xx obliges it to assume complete disallow. Google's documentation carves 429 out of that rule. Lantad's code does not.
