Do AI crawlers respect nofollow? 1,176 of 158,327 links carried it, and 2 of 9 vendors document honouring it

Do AI crawlers respect nofollow?

Two of the nine vendors say so in writing, and the other seven leave it open. The Common Crawl FAQ, read on 25 September 2026, asks the question under its own heading and answers it: "We currently honor the nofollow attribute as it applies to links embedded on your site." It then adds the caveat that makes the commitment legible, which is that if it ever did stop honouring the attribute it would use those links only for discovery and would never create an association between the discovered link and the source document. Amazon's Amazonbot page, read the same day, is one sentence: when these user agents access web pages they respect the link-level rel=nofollow directive, and the page-level robots meta tags of noarchive, noindex and none.

The seven others say nothing you can act on. OpenAI's crawler documentation does not contain the string nofollow anywhere in its prose. Neither does the Perplexity crawler page at docs.perplexity.ai/docs/resources/perplexity-crawlers, nor the ByteDance page at zhanzhang.toutiao.com/docs/intro/26899. Anthropic's crawler article, last updated 7 April 2026, carries sixteen occurrences of the string in its page source and zero in its text: fourteen of the sixteen are rel attributes on the support site's own navigation links, which is a small joke at the expense of anyone grepping a page instead of reading it.

Apple's Applebot page at support.apple.com/en-us/119829 defines nofollow, but as a robots meta tag directive rather than as a link attribute: "nofollow: Applebot won't follow any links on the page." That is a page-level control, and the token in this registry is Applebot-Extended, which is a robots.txt token. Google documents nofollow extensively for Google Search and not at all for Google-Extended, which is likewise a robots.txt token with no link-level vocabulary of its own. Meta's crawler documentation returned HTTP 400 to this scanner on 25 September 2026 and was not read, so Meta is unknown here rather than silent, which is the same gap that showed up when this corpus went looking for key directories for web bot authentication.

Two of nine is also what came back the last time these nine pages were searched for a different directive: on 9 September 2026 two of the nine operator pages named the noindex tag, and they were Apple's and Amazon's. The overlap between that pair and this one is a single vendor. Amazon's page is the only one of the nine that takes a written position on both the link attribute and the page-level tag, and it does it in a single sentence that fits on one line. Four of the nine, OpenAI, Anthropic, Perplexity and ByteDance, name neither, so for those four a publisher who writes any markup-level control is guessing at what the crawler does with it.

The HTML standard defines nofollow and never mentions crawling

It is worth reading the definition before assuming what it instructs. The HTML Living Standard's link types section, at html.spec.whatwg.org/multipage/links.html on 25 September 2026, lists 26 link types under their own headings, and its entry for nofollow reads: "The nofollow keyword indicates that the link is not endorsed by the original author or publisher of the page, or that the link to the referenced document was included primarily because of a commercial relationship between people affiliated with the two pages."

That is an editorial statement about endorsement. It says nothing about fetching, crawling, indexing or following, and a crawler that reads the standard alone has been told what the author thinks of the destination, not what to do about it. The two vendors that have committed to honouring it are therefore going further than the specification asks, and the seven that say nothing are not in breach of anything.

The search engine reading is a separate layer on top. Google's page on qualifying outbound links, carrying Last updated 2025-12-10 UTC, defines the attribute in its own terms: "Use the nofollow value when other values don't apply, and you'd rather Google not associate your site with, or crawl the linked page from, your site." The same page introduces two more values, rel="sponsored" and rel="ugc", neither of which appears among the 26 link types the HTML standard defines. So the vocabulary an SEO audit hands a publisher is partly a web standard and partly one company's extension of it, and the AI crawler documentation in the table above covers neither half consistently.

The corpus reflects that split precisely. Of the 158,327 links this run resolved, 11,527 carried a rel attribute of any kind, and the attribute is used overwhelmingly for browser behaviour rather than for crawler instruction: 8,107 declared noopener and 6,231 noreferrer. Beyond nofollow itself, Google's newer vocabulary is close to absent from this surface. Fifteen links across two sites, france24.com and handelsblatt.com, declared rel="sponsored". Not one link in the corpus declared rel="ugc", which is the expected result on a home page and would not be on a comment thread.

How many links carry rel=nofollow, and who is using it

1,176 of 158,327 is 0.74 percent, and the aggregate hides the real shape, which is that almost nobody uses this attribute and a handful of sites use it heavily. 942 of the 1,069 home pages carried no nofollow link at all. Among the 127 that carried at least one, the median count was four and 30 sites carried exactly one. The two largest users between them account for 299 of the 1,176, which is a quarter of every nofollow link in the corpus.

The category breakdown is flatter than the headline sites suggest, and the differences it does show are small enough that a corpus of this shape cannot carry much weight on them. The platform seeds, which are the smaller sites built on Framer, Webflow, Wix, Squarespace, Shopify and WordPress, carried 307 nofollow links over 28,716 links from 374 sites, a rate of 1.07 percent. Healthcare was the densest of the industry categories at 1.39 percent, almost entirely because of one hospital site. Education was the sparsest at 0.20 percent across 101 sites and 17,887 links.

Where the external nofollow links point is the least surprising and most instructive part of the whole count. Ranked by how many distinct sites nofollowed at least one link to them, the destinations are youtube.com on 32 sites, facebook.com on 31, instagram.com on 26, linkedin.com on 24, twitter.com on 13, tiktok.com on 8 and x.com on 7. These are social profiles in a footer, and a site that nofollows its own social links is applying an attribute about endorsement to the accounts it owns. It is also, on the same pages, the exact set of URLs that the sameAs property points a resolver at: this corpus published 310 sites naming a social profile in their identity markup on the same day. One part of the page says these accounts are us. Another part says do not follow them. Neither instruction is wrong on its own terms, and a generative engine optimization audit that reads only one of the two will report a site as consistent when its markup is not.

51 sites told crawlers not to follow links to their own pages

499 of the 1,176 nofollow links in this corpus point at the same host that served the page. That is the finding worth acting on, because the endorsement reading of the attribute cannot explain it: a site does not need to withhold endorsement from itself. What it can do, if a crawler takes the attribute as an instruction, is remove its own pages from the set a crawler discovers from the home page, which is the one page an AI crawler is most likely to fetch first.

nyulangone.org is the clearest case. On 25 September 2026 its home page carried 226 links to its own host and 134 of them declared rel=nofollow. The targets are not utility pages: 36 sit under /conditions/areas-of-expertise, 21 under /locations/directory, 12 under /locations/hassenfeld-childrens-hospital, and the rest run through /care-services, /cancer, /heart and /womens-health. A hospital's clinical directory is exactly the material an answer engine would want when somebody asks where to be treated for something, and the home page is telling any crawler that honours the attribute to leave it alone.

www.oaklandish.com, a Shopify store in the platform seeds, carried 271 links to its own host and nofollowed 152 of them, including every top-level collection: /collections/men, /collections/women, /collections/kid, /collections/outerwear, /collections/bottoms and /collections/headwear, plus /pages/about-us, /pages/contact-us and its own blog. The pattern is familiar from theme markup rather than from an editorial decision, and it is the kind of default that a Shopify visibility fix has to look for. Both figures reproduced on three consecutive requests the same day.

What this run did not do is test whether any crawler acts on those attributes, and nothing here should be read as saying it does. Measuring that needs server logs from a site that nofollows a page no other path reaches, which is a different study from a one-request-per-host sweep. What the sweep can say is what the links on a home page offer a crawler, and on 51 of these sites part of that offer carries a flag asking two documented crawlers not to take it.

A page-level nofollow on a home page is usually a block page, not a decision

The link attribute has a page-level sibling. A robots meta tag carrying nofollow asks a crawler not to follow any link on the page, and Google's robots meta tag documentation, carrying Last updated 2026-03-24 UTC, defines it in four words: do not follow the links on this page. Twelve of the 1,069 home pages in this run carried one.

Eight of the twelve were not decisions. boston.gov, statssa.gov.za, nus.edu.sg, pennmedicine.org, heb.com, gtbank.com, regions.com and zurich.com each returned HTTP 200 with a body of between 212 and 1,167 bytes containing a reference to /_Incapsula_Resource, which is Imperva's bot mitigation product, and zero links of any kind. The noindex, nofollow meta tag in those responses belongs to the block page, not to the site. A ninth, hu-berlin.de, returned 7,751 bytes titled "Making sure you're not a bot!". This corpus has hit the same shape before from the other direction, when every X-Robots-Tag noindex header it found turned out to be a captcha, and it is the single most common way an automated reading of this kind gets a false positive.

That leaves three real pages. wiley.com returned 529,309 bytes titled "Trusted Research & AI Insights | Wiley" with a meta robots value of noindex, nofollow and 167 links in the markup. hku.hk returned 98,684 bytes with the same directive and 164 links. sendbird.com returned 104,815 bytes with a server-rendered noindex, nofollow, which this site has recorded on that host once before. Whether any of the three is deliberate is not ours to say, and the tag is what a crawler reads. It is worth noting that the same directive also carries noindex, so the more consequential half of it is the half nobody is asking about.

Distinguishing the eight from the three took a second request and a look at the body, which is the general lesson rather than a detail of this run. A crawler-shaped request that gets a 200 has not necessarily reached the site, and a robots.txt tester or a view of what GPTBot sees is more useful than a status code for telling the two apart.

What to check on your own site

The practical question is narrow, and it is not whether to remove every nofollow on the page. It is whether any link on a page a crawler fetches first is the only route to a page you want quoted. On 51 of these 1,069 sites the answer is at least plausibly yes, and on two of them it is a large part of the site's own catalogue.

The check is four steps and needs no tooling beyond a fetch and a text editor. Request the page with no JavaScript running, because that is what a non-rendering crawler receives and it is where this whole measurement was taken. Read the rel attribute on every anchor, not just the ones in the main navigation, since the nyulangone.org and oaklandish.com cases both live in body content and menus rather than in a footer. Read the anchor text at the same time, because an earlier capture found 85 links carrying no anchor text at all, and a link a crawler cannot name is a weaker signal than one it can follow but will not. Separate the links pointing at your own host from the ones pointing elsewhere, because those two groups raise different questions. Then, for each internal one that carries the attribute, ask whether the destination is reachable by a path that does not.

If it is reachable another way, a nofollow costs nothing measurable and the attribute is doing what a template author meant it to do. If it is not, you have made the destination depend on a crawler ignoring an attribute that two vendors have said in writing they honour. That is a different risk from a robots.txt rule, which is explicit and which this corpus has shown is usually decided by a wildcard group nobody wrote for AI, and a different risk again from a page that offers a crawler no internal path at all.

Two things this post does not establish are worth restating plainly. It does not show that any AI crawler behaves differently because of a nofollow, only what the vendors publish and what the pages ship. And it is one request per host from one network location on one day, so it describes this corpus rather than the web, which is the standing limit on every sweep of this kind and is set out in the scanner methodology. The absence of a vendor statement is also not a claim that the vendor ignores the attribute. It is a claim that a publisher cannot find out, which is the ordinary condition of AI visibility work and the reason a count like this is worth taking at all. The other seven vendors could each close it with one sentence, and the two that already have show exactly how short that sentence needs to be.