Absolute vs relative URLs: 47,233 of 150,585 internal links were absolute, and 102 pinned plain HTTP

Absolute vs relative URLs: what actually changes for a crawler?

RFC 3986, the January 2005 standards track document that defines URI syntax, separates the two cases cleanly. Section 4.2 describes a relative reference as one that takes advantage of the hierarchical syntax, and names a reference that does not begin with a slash a relative-path reference. Section 5.1 then lists where the base to resolve it against comes from, in order of precedence: a base embedded in the content, a base from the encapsulating entity, a base from the retrieval URI, and finally a default. The innermost definition wins.

That ordering is the whole subject of this post. For almost every page on the web there is no base embedded in the content, so the base is the retrieval URI: the address the document was actually fetched from, after redirects. A relative link therefore inherits its scheme and its host from wherever the crawler happened to arrive, and inherits them correctly by construction. It cannot point at the wrong host, because it names no host.

An absolute link makes the opposite trade. It names the scheme and the host outright, so it is immune to the base, and it is immune in both directions: right when the author typed the canonical address, wrong forever when they did not. Nothing about serving the page will correct it. That is not a theoretical risk, and the two sections after the measurement are both instances of it.

For a crawler the practical consequence is a request count. A link resolving to an address that answers 200 is one request. A link resolving to an address that redirects is two, and the crawler has to decide whether the destination is a page it already holds. Redirects are already the commonest thing a crawler meets on the way into a corpus home page, and on about half the pages in an earlier run a redirect took most of the wait. Adding an avoidable hop to every internal link multiplies that across the whole site rather than across one entry point.

One case is worse than a redirect, because it does not resolve into anything. Google's crawlable links page states that Google can only crawl a link if it is an anchor element, that it cannot reliably extract URLs from anchors without an href, and asks authors to make sure the URL in the anchor resolves into an actual web address. The links counted here all had an href that parsed. Whether the address it built was live is a separate question, and one this run went and asked.

What 1,084 home pages actually wrote

The method was deliberately narrow, because a link census goes wrong in the counting rather than in the fetching. One request per hostname, the home page only, redirects followed, and the final URL after redirects used as the base for everything that followed. An anchor counted only if it carried an href whose value was not empty and did not begin with a fragment or with a scheme a crawler would not follow, which removed mailto, tel, sms, javascript and about. 178,431 anchors survived that filter.

Each surviving href was then classified by its written form, not by where it ended up. A value beginning with a scheme and two slashes is absolute. A value beginning with two slashes alone is protocol-relative. A value beginning with one slash is root-relative. Anything else is path-relative. The link was called internal when its host matched the host the page was served at, matched that host's www or apex counterpart, or when it named no host at all. 150,585 did; 27,845 pointed somewhere else.

The headline split is that relative references won, and they won on volume rather than on pages. 97,044 of the 150,585 internal links were root-relative, which is 64.4 percent, and they appeared on 869 of the 1,084 pages. Absolute internal links came to 47,233, which is 31.4 percent, spread across 757 pages. So most pages use both forms, and the two forms are not competing conventions on the same site so much as different parts of the same template disagreeing.

The per-page picture is more lopsided than the totals suggest, and it is the number worth carrying away. Across the 1,012 pages with at least one internal link, the median page wrote 6.9 percent of them as absolute URLs. The mean is dragged up by a minority that writes nothing else: 89 pages wrote every internal link as an absolute URL, and 255 wrote every one as a relative reference. A corpus average of 31.4 percent is therefore describing two populations rather than one habit, which is a distinction the methodology treats as mandatory whenever a percentage covers a mixed frame.

Two forms barely registered and are worth naming so their absence is on the record. Path-relative references, the kind that actually depend on the directory the page sits in, numbered 5,534 on 204 pages, which is 3.7 percent of internal links. Protocol-relative links, the //host/path form that inherits only the scheme, came to 1,105 in total across 34 pages, of which 774 pointed back into the site and 331 off it. That form was common practice while sites served both http and https and is now residue. It does no harm on a page fetched over HTTPS, because the scheme it inherits is the right one.

The platform decides the link form more than the author does

The corpus is stratified two ways: one seed file samples by the platform a site is built on, the other by what the site's pages are for. Splitting the link forms by stratum turns a flat average into something closer to a cause, because an author writing a page in a content editor does not usually choose how the template serialises a link.

The spread is wide enough that it is not a sampling artefact. The WordPress small business stratum wrote 2,932 of its 3,483 internal links as absolute URLs, which is 84.2 percent, and the local media stratum wrote 2,748 of 3,321, which is 82.7 percent. At the other end the single page application stratum wrote 27 absolute internal links out of 2,134, which is 1.3 percent, and the static documentation stratum wrote 13 out of 683, which is 1.9 percent. Webflow sites came in at 4.7 percent and Shopify direct to consumer sites at 5.6 percent.

That ordering has an obvious reading and it is worth stating carefully, because the measurement supports the pattern and not the mechanism. Systems that store content as editable documents tend to store the full address with it, since the address has to survive being moved between a draft, a preview and a live page. Systems that build routes from a path at request time have nothing to gain from writing the host down. This run observed the output of those systems, not their internals, so the correlation is measured and the explanation is inference. The stack guides exist for exactly this kind of platform-specific default, and the Shopify guide is the other end of the same range.

The practical point for anyone acting on this is that the link form is usually not a per-link decision to review. On a WordPress site writing 84 percent of its links absolute, the question is what the editor and the theme put in the database, not whether one page got it wrong. On a React or Astro site writing 99 percent of them relative, the absolute links that do appear are nearly always hand typed into prose or into a configuration file, which makes them both rarer and likelier to be stale.

None of this changes what the scanner reports, and it is not meant to. The AI visibility grade does not score the reference form of a link, because the form is not a defect. What follows is about the cases where the form carried a value the site no longer honours, which is a defect, and which is measurable without any judgement about style.

102 internal links pinned plain HTTP, and the probe says what that costs

An absolute link records the scheme its author typed. 102 of the 47,233 absolute internal links, spread across 38 of the 1,084 pages, named the page's own host over plain http rather than https, on pages that Lantad itself had just reached over HTTPS. A further 37 did the same thing while also naming the sibling host.

That is a small count and a precise one, so rather than assert a consequence this run went and measured it. 67 of those http links, across all 38 hosts, were requested on 8 November 2026 as LantadBot/1.0 with redirects deliberately not followed, so that the first response was recorded rather than the eventual destination. 53 of the 67 returned a redirect, and 46 of those were a straight upgrade to https on the same host and the same path. That is the benign case: one extra round trip per link, paid by every client that follows it, including every crawler that trusts the href rather than guessing.

12 of the 67 did not redirect at all. One returned 404. Eleven returned 403. Because a 403 to a declared bot is usually a bot rule rather than a scheme rule, the obvious alternative explanation had to be eliminated, so each of those 12 addresses was requested again over https with the same user agent, in the same run, from the same network. Nine returned 200, one returned 301 and one returned 404. Only one returned 403 both ways, and that link pointed at marriott.com from a page that had itself redirected onto marriott.com, so the refusal there is about the client and not the scheme.

So for nine of those twelve links the plain http spelling was the only thing standing between a crawler and a page the same server serves happily. The hosts were bbc.com, bild.de, lefigaro.fr, sueddeutsche.de, wyborcza.pl, asahi.com and nyulangone.org. A person clicking the same link is usually unaffected, because browsers upgrade known hosts before the request leaves, which is precisely why this class of defect survives on live pages for years: the people who would notice it are not the ones reading it.

Two links returned 200 over plain http, both on imss.gob.mx, which means that host still serves content without TLS. That is the opposite problem and it belongs in a different post, but it is the honest reading of those two rows. Readers checking their own pages can see the raw response a crawler receives through what GPTBot sees, and the user agent this run used is documented on the bot page.

600 links named the other host, and 107 of 146 probed redirected

The second way an absolute link goes stale is the host rather than the scheme. 600 internal links, on 77 of the 1,084 pages, named the www or apex counterpart of the host the page was actually served at. A page served at the bare domain linking www, or a page served at www linking the bare domain, is a page sending every one of those links through whichever redirect reconciles the two.

146 of those 600, covering all 77 hosts, were probed on 8 November 2026 without following redirects. 107 returned a redirect, split 92 permanent, nine temporary and six permanent of the 308 kind. Five failed to connect at all. The remaining 34 returned 200, which is the result worth stopping on: on those sites the sibling host is not a redirect to the canonical address, it is a second live address serving the same content.

That is a different problem from an extra hop, and it is one this blog has measured from the other direction. An earlier run found 305 of 313 home pages answering at more than one live URL, and another found 163 of 1,069 home pages declaring no canonical tag at all, with 63 more naming a URL the page was not served at. An internal link pointing at a live sibling host is how a crawler is handed the duplicate in the first place. The canonical tag is then the only thing telling it which of the two to keep, and on the pages with no canonical tag there is nothing.

The largest single case was snu.ac.kr, with 203 internal links on its home page naming the sibling host. The next largest were mirrormate.com with 60, iitb.ac.in with 24, and uba.ar and fudan.edu.cn with 22 each. Education sites dominate that list, which fits a pattern the corpus has shown before on robots.txt redirects crossing hosts: long lived institutional sites accumulate host spellings faster than they retire them.

It is worth being clear about what this is not. None of these links is broken, and a crawler following them reaches the content. The cost is paid in requests, in duplicate detection, and in whatever the site's own canonical signals say about which address to keep. A site whose internal links all pointed at one host would not need those signals to adjudicate anything, which is the quiet argument for the relative form: it cannot name the wrong host because it names none.

One base element in 68 moved a relative link, and it moved 119 of them

RFC 3986 puts a base embedded in the content above the retrieval URI, and in HTML that embedded base is the base element. It is the one mechanism that can make a relative link point somewhere other than where its own address implies, so a run that counts relative links owes a count of it.

70 of the 1,084 pages carried a base element. Two of those, boe.es and taobao.com, set only a target attribute and no href, which leaves the document base alone. Of the 68 that did set an href, 39 set it to the empty string, 22 set a relative path and 7 set an absolute URL on the page's own host. One page, bloomandwild.com, carried two base elements; the MDN reference for the element, last modified on 24 April 2026, states that there can be only one base element in a document and that where several appear only the first href and first target are obeyed, so its first value is the one that counts.

On 67 of those 68 pages the base moved no link, and there are two separate reasons for that rather than one. On 60 of them the declared base resolved to the same directory as the page's own address, so there was nothing for it to change: resolving an empty value against the page address returns the page address, and resolving / against a home page served at the root does the same. The other eight declared a base that genuinely pointed somewhere else, and on seven of those eight no reference moved anyway, because a link beginning with a slash resolves identically against any base on the same origin and those seven pages had no path-relative link at all. A base element only reaches the one form that was 3.7 percent of the corpus.

The first of those reasons is partly an artefact of measuring home pages, and the limitation deserves saying plainly. A base of / is inert on a page served at / and would relocate every path-relative link on a page served three directories down. This run did not test interior pages, so it cannot say how often that happens.

The exception is the case that proves the mechanism, and it is a national statistics office. destatis.de serves its home page at https://www.destatis.de/DE/Home/_inhalt.html and declares a base of https://www.destatis.de/. 137 of its 147 usable anchors are path-relative, written as paths like DE/Presse/_inhalt.html. Resolved against the declared base they produce /DE/Presse/_inhalt.html. Resolved against the page address, which is what a client ignoring the base element would do, they produce /DE/Home/DE/Presse/_inhalt.html. 119 of the 120 unique relative references on the page resolve to different addresses depending on which base is used.

Both resolutions were then requested, which turns the argument into a measurement. 106 of the differing targets were distinct after normalising fragments, and 25 of those were sampled and fetched both ways on 8 November 2026. All 25 returned 404 resolved against the page address and 200 resolved against the declared base. So on that one page, a crawler that does not implement the base element finds no pages at all through the navigation, while one that does finds the site. That is not a grading question, it is the difference between a crawl path and no crawl path, and it rests on one element in the head.

What this measurement does not show

No AI crawler was observed. Everything here was taken with Lantad's own client, and the consequences argued from it rest on RFC 3986 and on vendor documentation rather than on a crawler being watched losing a page. No vendor among the ones the crawler reference tracks publishes whether its fetcher implements the base element, so the destatis case establishes what the markup does and not which crawlers get it wrong. That gap is the honest shape of the finding and not a hedge around it.

One page per host, and that page the home page. A home page is the least representative page on most sites for exactly the property measured here, because it sits at the root, where a base of / is inert and where path-relative references have the shortest distance to travel. A run over interior pages would almost certainly find more base elements doing something, and this one cannot say how many.

The classification is of the written form, not of the destination. A link counted as absolute and same host was compared against the host the page was served at after redirects, so a site that redirects its apex onto www has its www links counted as same host and its apex links counted as the sibling. That is the right frame for asking what a crawler arriving at the final address pays, and the wrong frame for asking what the author intended. HTML entities in href values were not decoded before classification, which does not affect a scheme or a host but does mean a small number of query strings were probed with an ampersand entity still in them; the http and https probes used the identical string, so the comparison between them holds. One known misclassification is worth recording rather than rounding away: the filter that removes unfollowable schemes did not recognise the unsafe:javascript: prefix one framework emits when it sanitises a script link, so a single href on handelsblatt.com was counted as path-relative when it is not a link at all. That is one anchor in 178,431 and it moves no figure in this post.

Only 213 of the 702 links flagged as plain http or sibling host were probed, sampled at up to three per page so that one large site could not dominate the result. The probe results are therefore a reading of those 213 and not a rate for all 702. Status codes were read once, on one day, from one network, which cannot separate a standing misconfiguration from a transient one.

Finally the corpus is a stratified sampling frame built to study crawlability, not a census, so a stratum figure such as 84.2 percent for WordPress small business sites shows direction rather than a rate for WordPress generally. Nothing in this run changes what the scanner reports: the reference form of a link is not scored, and this post does not argue it should be, because a relative link and a correct absolute link are equally readable. What is scored, and what the research index collects, is whether the address a crawler builds answers. Readers who want the surrounding argument can start from the crawlability study, the underscore finding on the same corpus links, or the sitemap and link comparison.