AI crawler SSL error: 16 of 1,419 sites served a certificate that did not validate, and 4 left no route in

What is an AI crawler SSL error, and why does it close the whole site?

A crawler fetching a page over HTTPS does four things in order, and only the fourth is the part anyone writes about. It completes a TLS handshake, it validates the certificate the server presented against its own set of trusted roots, it requests robots.txt to find out what it is allowed to read, and then it requests the page. Every check this site normally reports, from access rules through to schema, lives in that fourth step.

Validation failing at step two has a consequence that is easy to understand and easy to miss. The crawler is not refused the page; it is refused the connection, and the connection is what robots.txt arrives over too. Lantad confirmed this directly rather than assuming it: all 16 hosts were asked for https://host/robots.txt on 7 November 2026 and all 16 failed with the same validation error they gave for the home page.

That matters because of what the standard says about a robots.txt that cannot be fetched. RFC 9309, the Robots Exclusion Protocol of September 2022, states in section 2.3.1.4 that if the robots.txt file is unreachable due to server or network errors, this means the robots.txt file is undefined and the crawler MUST assume complete disallow. A site in that state is not merely hard to read. Under the rule its crawlers are written to, it is closed.

One honest qualification belongs here, because this is the load-bearing claim of the post. RFC 9309 never uses the words TLS, SSL or certificate anywhere in its prose; Lantad read the document at source on 7 November 2026 to check. The section names server errors in the 500 to 599 range as its example and otherwise says network errors. Reading a failed certificate validation as a network error is an interpretation of that phrase, and a defensible one, since the fetch returns no status code at all. It is not the specification naming certificates, and this post does not present it as one. The companion case, where robots.txt does return a status code, is the subject of an earlier measurement on 404 and 503 being opposites.

What was measured, and against which trust store

The run asked for https://host/ once for each of the 1,419 hostnames held in worker/seeds/corpus-seeds-platform.json, which carries 392 hostnames in ten platform strata, and worker/seeds/corpus-seeds-industry.json, which carries 1,027 in eight industry strata. Every request was sent as LantadBot/1.0, the user agent documented on the bot page, with redirects followed to a limit of five, a twenty five second timeout and no JavaScript executed. Of the 1,419, 1,092 answered HTTP 200, 202 answered HTTP 403, and 16 failed certificate validation.

Validation is only meaningful against a named set of roots, so here is the set. The client used the trust store at /etc/ssl/certs/ca-certificates.crt, holding 122 root certificates, from the ca-certificates package version 20260816ubuntu0.24.04.1. Every one of the 16 failures was then re-requested at least twice more on the same day and reproduced each time, and the chain each host sent was read directly with a TLS client so that the cause could be attributed to a certificate rather than guessed from an error string. That distinction did real work: two hosts produced different error strings for the same underlying cause, and one produced a certificate-shaped error for a cause this run could not isolate.

Two classes of transport failure were deliberately excluded rather than folded in, and naming them is the difference between a measurement and a headline. 24 hostnames failed with an HTTP/2 framing error and 28 with a connection reset or a protocol alert. Those are not certificate validation failures, several of them returned HTTP 200 when asked again minutes later, and some are attributable to the network this run was made from rather than to the site. None of them is counted among the 16. The same care applies to the 202 hosts that answered 403, which is a refusal of a request that was successfully made, and the subject of a separate measurement on CDN blocking.

The sixteen are not spread evenly. Seven sit in the education stratum, 7 of its 130 sites, three in government, 3 of 129, three in healthcare, 3 of 123, one in finance, 1 of 127, and two in the no-code platform stratum, 2 of 42. That concentration is the most interesting thing in the dataset, and the next two sections are about why.

Six sites sent one certificate and no issuer

The largest group is also the one most likely to surprise the people running the sites, because nothing is wrong with their certificates. Six hosts sent exactly one certificate, the leaf, and no intermediate. Each of those leaves was issued by a widely trusted commercial authority, and in every case the root that authority chains to was already sitting in the trust store. The missing piece is the certificate in the middle that connects them.

A client cannot invent that link. Given a leaf issued by DigiCert Global G2 TLS RSA SHA256 2020 CA1 and a store containing the DigiCert root, it still needs the intermediate itself to build a path, and a server that does not send one has not finished configuring TLS. The reason this survives in production for months is that browsers frequently paper over it: many will fetch a missing intermediate from the address printed in the leaf, or reuse one they cached from another site on the same authority. A strict client does neither, and nor does any client that has never visited a sibling site.

That is what makes this class worth separating from a bad certificate. The six are caltech.edu and nyu.edu, both with leaves from InCommon RSA OV SSL CA 3, gob.mx from GlobalSign GCC R46 OV TLS CA 2025, seattlechildrens.org and ssa.gov both from DigiCert Global G2 TLS RSA SHA256 2020 CA1, and uonbi.ac.ke from Sectigo Public Server Authentication CA OV R36. Every one of those six certificates names its own host in the subject alternative name, and every one was still inside its validity window on the day of measurement. A site owner checking the expiry date, which is the thing people check, would find nothing at all.

This is the same category of fault as the ones behind a robots.txt that answers with a web page and a HEAD request answered with 405: the content is fine and the plumbing in front of it is not. It is also the cheapest of the sixteen to fix, since the intermediate is published by the issuing authority and adding it to the server's chain file changes nothing a visitor sees.

Four chains ended in a root the trust store does not hold

The second group explains why education is the worst represented stratum in this measurement. Four hosts sent a complete, well formed chain that still did not validate, because the root at the end of it is not in the store. All four leaves were issued by an Internet2 InCommon intermediate, stanford.edu by InCommon Intermediate CA DVG2C and jhu.edu, pennmedicine.org and whoi.edu by InCommon Intermediate CA OVG2C. In each case that intermediate was itself signed by emSign Root TLS CA G1, which in turn chains to emSign Root CA G1, both operated by eMudhra Technologies Limited. Neither emSign root appears anywhere in the 122 certificates of the store used here.

This is a different failure from the previous one and it needs saying plainly: these four sites did nothing careless. They hold certificates from the authority that serves American research and education, they sent the full path, and the path is internally valid. What they cannot control is whether the root at the top of the particular path they send is in the store of whatever client arrives. Lantad has not established why those roots are absent from this store and is not going to speculate about it in a post that cannot cite the reason, so the finding is stated as what it is: a chain terminating in a root this client does not trust.

The practical consequence does not depend on the reason. A client whose store holds an emSign root validates these four without complaint, and a client whose store does not holds no path to them, which is why the same site can be fine in a browser and unreachable to a crawler on the same afternoon. Where an authority publishes more than one cross-signed path, which one a server sends is a server configuration choice, and it is the choice that decides how many clients can build a path at all.

It is worth being precise about the error strings too, because they disagreed while the cause did not. stanford.edu, pennmedicine.org and whoi.edu each sent four certificates including the self-signed root and reported a self-signed certificate in the chain. jhu.edu sent three, stopping before that root, and reported that the local issuer certificate could not be found. Two messages, one cause, which is exactly why this run read chains instead of counting error text. Anyone wanting to see what their own site sends can start from what GPTBot sees and the method notes.

An expiry, two names that did not match, and two private authorities

The remaining six are each their own story, and together they cover every other way this fails. One certificate had simply expired. minsal.cl, the Chilean health ministry, served a wildcard certificate for minsal.cl whose notAfter date was 9 November 2025, putting it 362 days past expiry on the day it was read. That is the failure everybody pictures when this subject comes up, and it was the rarest in the corpus: one site in 1,419.

Two hosts presented a certificate that was entirely valid for a name other than the one asked for. bankofengland.co.uk sent an Entrust certificate good until 29 March 2027 whose subject alternative name lists only www.bankofengland.co.uk, so a request to the bare domain fails on the name while the same server answers the www host perfectly. nushift.ai sent a Sectigo certificate naming ingress-earth.ewp.live and its wildcard, which is the ingress domain of the platform hosting it rather than anything belonging to the site. Both are reminders that a certificate is a statement about a name, and that the apex and the www host are two names. The broader habit of answering at more than one address is something this corpus has measured before.

Two more were issued by authorities no public client has any reason to trust. ibge.gov.br, the Brazilian national statistics institute, served a certificate issued by a CA named CA-IBGE under its own domain, with a leaf expiring in 2069. saashook.com served a self-signed certificate whose only name is ip-172-31-86-210.ec2.internal, the internal hostname of the cloud instance behind it, issued in 2021 and valid until 2031. The second of those is a default that was never replaced, and it is the clearest single example in the dataset of a site that has never been fetched over HTTPS by anything that checks.

The sixteenth is the one this run could not close. unam.mx sent four certificates, a leaf from GlobalSign GCC R3 EV TLS CA 2025 and a tail reaching GlobalSign Root CA, and validation still failed for want of a local issuer. Both GlobalSign roots involved are the sort a store normally carries, so the likeliest explanation is a gap in the middle of the chain rather than an untrusted root, but Lantad did not isolate it and is recording it as unresolved rather than filing it under a cause it did not verify.

Four of the sixteen left a crawler no route in

A failure at the apex over HTTPS is not automatically a site a crawler cannot reach, because most sites also answer plain HTTP and redirect. So the run followed that path too, asking each of the 16 for http://host/ and following where it led. The answers split the sixteen into two very different situations.

For six of them the redirect rescues the crawler. bankofengland.co.uk, caltech.edu, gob.mx, minsal.cl, seattlechildrens.org and stanford.edu all redirect plain HTTP to their www host, and that host validated and answered HTTP 200 on 7 November 2026. A crawler that arrives at the apex over HTTPS fails; one that arrives over HTTP, or that follows a link to the www host, reads the site normally. The damage is confined to one spelling of the address, which still matters, because the apex is the spelling people write in links and sitemaps.

For four of them the redirect leads back into the failure. pennmedicine.org, saashook.com, ssa.gov and whoi.edu each answer plain HTTP with a redirect to an HTTPS address on the same host, which is the address whose certificate does not validate. There is no third option to try. On the evidence of this run, taken on one day from one network, those four sites had no route in that a validating client could use, which puts them in a stricter position than a site that deliberately refuses crawlers in its rules.

Three others, jhu.edu, nyu.edu and unam.mx, redirect to a www host that validates and then answers HTTP 403, so a crawler is refused at a second gate for reasons that have nothing to do with certificates and everything to do with the subject of whether a site is blocking AI crawlers. The last three, ibge.gov.br, nushift.ai and uonbi.ac.ke, sent no redirect to HTTPS at all: uonbi.ac.ke served its home page over plain HTTP with HTTP 200, and the other two answered HTTP 403 and HTTP 404 there.

None of this tells you what any particular crawler does, and the post does not claim otherwise. Lantad read the published crawler documentation of two vendors at source on 7 November 2026 to see whether either addresses the question. OpenAI's bots documentation names OAI-SearchBot, OAI-AdsBot, GPTBot and ChatGPT-User and says nothing about TLS, certificates or what happens when validation fails. Anthropic's crawler support article names ClaudeBot, Claude-User and Claude-SearchBot and is silent on the same subject. Absence of a statement is recorded here as absence, not as evidence of behaviour, which is the same standard applied to the vendor silence in this corpus measurement of who cleared every gate.

What this measurement does not show

The figure to carry away is 16 of 1,419, and the things it does not establish are worth as much as the things it does. This run observed what servers presented to one client on one day. It did not observe a single AI crawler, so every sentence about consequence rests on the published standard and on what each vendor has documented, which in the two cases checked is nothing.

The trust store is the clearest boundary. Validation is a relation between a chain and a set of roots, and a different set gives different answers, which is most of the story for the four chains ending in an emSign root. A crawler whose store carries those roots sees no problem at those four sites. The six leaf-only failures are the sturdier half of the finding, because their roots were present and the gap was in what the server sent, and the four cases of expiry, name and private authority are sturdier still, since those hold against any store.

Revocation was not checked at all. A certificate that validates here may have been revoked, and a client that checks revocation would reject it. That is a real gap rather than a technicality, and the only honest thing to do with it is name it rather than imply that validating means current. One URL per host was tested and it was the home page, so a different certificate on an interior host or a separate service was not seen, and one reading on one day from one network cannot distinguish a standing misconfiguration from one that was fixed the following morning.

Finally, the corpus is a stratified sampling frame built to study crawlability, not a census of the web, so 7 of 130 education sites indicates direction and not a rate for universities generally. Nothing here changes the AI Visibility Score either: the scanner does not currently report a certificate validation failure as its own finding, so this was taken with a direct probe rather than through the published pipeline, and the post does not argue the score should change until there is evidence an AI crawler is losing pages to it. Readers who want the surrounding argument can start from the two layers that decide whether AI can read a site, the crawler reference, or the research index.