Blog
The Lantad blog
What we find when we measure whether AI crawlers can actually read a site, what to do about it, and what we are changing in the product. Definitions and per-stack fixes live in the glossary and the fix guides; this is the dated, argued half.
Featured
-
Can AI read my pricing page? 17 of 93 SaaS pricing pages put a price in the markup
Lantad requested the home page of the 130 SaaS hostnames in this repository's committed corpus on 22 September 2026, followed each site's own link to its pricing page and read the raw bytes with no JavaScript executed. 93 pricing pages answered with HTML. 77 of them showed a price in visible text and 17 declared one inside JSON-LD, and four of those 17 declared a price of zero on a page whose text listed paid plans.
-
Do AI crawlers read iframes? 834 on 433 home pages, and 17 held 50 words
Lantad requested the home page of all 1,419 hostnames in this repository's committed corpus on 21 September 2026 and read the raw bytes with no JavaScript executed. 1,076 answered HTTP 200 with HTML and 433 of those carried an iframe, 834 elements in total. Every address they pointed at was then requested on its own: 354 of the 688 resolvable ones point at a tag manager, all 636 distinct iframe documents held 9,098 words between them, and two frames on two sites held 6,072 of those words.
Latest
-
Do cookie banners block AI crawlers? 269 of 1,093 home pages ran a consent platform, and it cost 0.44 percent of the words
Lantad requested the home page of all 1,419 hostnames in this repository's committed corpus on 21 September 2026 and read the raw bytes with no JavaScript executed. 1,093 answered HTTP 200 with HTML. 269 of them loaded one of 25 named consent platforms and 257 carried consent wording in the delivered HTML, but that wording came to 6,143 words against 1,381,167 visible words in total, and no page in the corpus gave a cookie notice more than half its text. -
Internal links and AI crawlers: 84 of 1,091 home pages offered a crawler no path at all
Lantad requested the home page of all 1,419 hostnames in this repository's committed corpus on 21 September 2026 and read the raw bytes with no JavaScript executed. 1,091 answered HTTP 200 with HTML. 84 of those carried no internal link a crawler could follow, and 55 of the 84 contained no anchor element at all. On the 1,007 that did offer a path, the first one was requested too. -
Schema markup validator: 117 of 612 home pages with JSON-LD carried a defect, and on 7 not one block parsed
Lantad requested the home page of all 1,419 hostnames in this repository's committed corpus on 21 September 2026 and read the raw bytes with no JavaScript executed. 1,072 answered HTTP 200 with HTML, and 612 of those carried at least one JSON-LD block, 1,143 blocks between them. 117 of the 612 carried a defect, 16 blocks never parsed as JSON, and on 7 sites not one block on the page did. -
Client side rendering: JavaScript supplied 7.6 percent of the prose, and all of it on 11 of 271 pages
Lantad loaded the home page of 392 hostnames twice on 20 September 2026, from the same browser at the same moment, changing one thing between the two loads: whether JavaScript was allowed to run. On the 271 pages that answered HTTP 200 both times, the unrendered document held 173,138 words and the rendered one held 187,422. The average is undramatic and the distribution is not: 100 pages were word for word identical, and 11 returned nothing readable at all until the bundle ran. -
Crawl delay robots txt: 157 of 1,014 files set one, and 11 named an AI crawler
Crawl-delay appears in no version of the robots exclusion standard, and it is still the most common non-standard field in this corpus. Lantad requested /robots.txt once from each of 1,419 hostnames on 20 September 2026. Of the 1,014 files that parsed into at least one group, 157 carried a Crawl-delay line, and 11 of those 157 put one inside a group naming one of the 15 AI crawler tokens this scanner evaluates. -
Soft 404: 125 of 1,094 sites answered a URL that does not exist with HTTP 200
Lantad asked all 1,419 hostnames in this repository's committed corpus for a path that cannot exist on 20 September 2026. 1,094 answered both that request and their own home page. 936 returned 404, 125 returned HTTP 200, and on a second nonexistent path of a different shape 111 of those 125 returned 200 again.
Findings
Measured results from real scans and published studies
173 more
-
Agentic SEO: 1 of 1,419 sites published an agent card, and 55 of the 69 OAuth documents came from a platform default
The only agent discovery path any standards registry lists is /.well-known/agent-card.json. Lantad asked all 1,419 hostnames in this repository's committed corpus for it on 19 September 2026, and one returned a usable card. 69 of the same hosts served an OAuth protected resource document at the path an MCP client reads, and 55 of those 69 carry the signature of a Shopify or WordPress default rather than of a file written for that site. -
Applebot robots.txt: 64 files hand Apple's crawler to their Googlebot group, and 44 of those answer differently
Apple documents that when a robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot instructions. Lantad requested /robots.txt once from each of 1,419 hostnames on 19 September 2026. 1,023 parsed into at least one group, 22 named Applebot, and 64 named Googlebot without naming Applebot. On 44 of those 64 the Googlebot group and the wildcard group give different answers, and on five the wildcard refuses the whole site while the Googlebot group lets Apple in. -
Do AI crawlers read RSS feeds? 86 of 762 sites declared one, and 159 served an llms.txt
Lantad requested the home page of 1,027 hostnames on 19 September 2026 and asked each of the 762 that answered for /llms.txt as well. 86 declared an RSS or Atom feed in a link element and 159 served an llms.txt, so the file no crawler vendor documents reading is on nearly twice as many of these sites as the format that has been standardised since 2005. -
robots.txt validator: 232 of 1,059 files carried a defect, and 33 sites answered with a web page
Lantad asked 1,419 hostnames for /robots.txt on 19 September 2026 and checked every answer against RFC 9309 and Google's published parsing rules. 1,104 returned HTTP 200. 33 of those returned HTML rather than robots.txt text, 12 returned nothing at all, and of the 1,059 that held real directives, 232 carried at least one defect a validator can name. -
Can GPTBot read your robots.txt? 92 of 1,056 sites refused it the file
Lantad asked 1,419 hostnames for /robots.txt three times on 18 September 2026, changing nothing between the requests but the user agent string. 1,056 returned a robots.txt to a desktop browser string. 92 of those returned a non-200 to OpenAI's example GPTBot string, and 63 of the 92 had served the same file to this scanner's own token seconds earlier. -
Content negotiation for markdown: 34 of 718 home pages answered, and six of those sent HTML
Lantad requested the home page of 1,027 hostnames twice on 18 September 2026, once normally and once with an Accept header naming text/markdown. 718 answered the first request with HTML. 34 answered the second with a markdown content type, six of those returned HTML bytes under that label, and four refused with 406 Not Acceptable. None of the nine AI crawler vendors behind the 15 tokens this scanner evaluates documents its crawlers asking.
Earlier
- Grok SEO: xAI documents no crawler, and 10 robots.txt files guessed 8 names 14 min
- Robots.txt subdomains: 73 of 277 gave an AI crawler a different answer from the apex 19 min
- Blocked by robots.txt: 559 of the 581 pages GPTBot lost were closed by a rule that never named it 17 min
- Crawler user agent: 1,004 robots.txt files named 2,209 tokens, and 120 name one Anthropic no longer documents 15 min
- Drupal AI crawlers: 79 of 86 robots.txt files named none, and 68 are Drupal's own file 15 min
- Robots.txt sitemap: 839 of 1,016 files declare one, and a site-wide block covers it on 92 of 93 17 min
- Does blocking AI training affect search rankings: 52 of 54 sites refusing Google-Extended still allow Googlebot 14 min
- Hreflang and AI crawlers: 27 of 719 home pages changed language on a header alone 20 min
- Web bot authentication: 2 of 9 AI crawler vendors publish a key directory 15 min
- AI structured data: 103 of 385 pages with JSON-LD named no organization 15 min
- Does page speed affect AI search: on 184 of 379 home pages a redirect took half the wait 19 min
- JSON-LD vs microdata: 22 of 385 home pages carried microdata and one carried RDFa 14 min
- LocalBusiness schema: 2 of 172 home pages carried opening hours a machine could read 16 min
- Allow AI crawlers in robots.txt: 89 of 145 files ruled only on the whole site 20 min
- Answer engine optimization: 250 of 647 home pages sent no signal that anything had changed 17 min
- Block AI training: 7 of 22 sites declaring ai-train=no blocked no crawler 16 min
- Is my site blocking AI crawlers: 79 of 115 refused one their robots.txt allows 17 min
- llms-full.txt: 32 of 44 sites served one, and the specification never names it 18 min
- Mistral AI crawler: 1 of 323 robots.txt files named one 15 min
- Most blocked AI crawler: GPTBot in 82 of 718 robots.txt files, and its search twin in 24 20 min
- Next.js AI crawlers: 28 home pages sent 15 times more payload than prose 18 min
- AI SEO: 17 of 380 home pages sent a crawler zero words 15 min
- Framer AI crawlers: 35 of 36 sites served markdown, and one robots.txt named a crawler 16 min
- Structured data: 141 of 382 home pages carried none in the raw HTML 14 min
- Webflow AI crawlers: 20 of 40 robots.txt files held no rule at all 18 min
- AI brand visibility: across 75,000 brands, the top correlate was not on the site 16 min
- robots.txt redirect: 122 of 391 sites sent one, and all 122 crossed hosts 16 min
- Squarespace AI crawlers: 61 of 66 sites named 26 and blocked none 15 min
- Wix AI crawlers: 0 of 69 robots.txt files named one, and Wix's own fix names BingAI 15 min
- Does noindex block AI crawlers: 2 of 9 operator pages name the tag 16 min
- GA4 AI Assistant channel: 2 of 6 assistant hosts answered a scan 15 min
- Shopify AI crawlers: 38 stores served 1,112 products as open JSON 14 min
- X-Robots-Tag noindex: 5 of 391 home pages sent one, and all five were a captcha 17 min
- How to get cited by Gemini: one robots.txt token covers training and grounding 16 min
- OAI-SearchBot: 16 of 140 robots.txt files named it, and 25 named GPTBot 16 min
- WordPress AI crawlers: 16 of 24 sites named none in robots.txt 14 min
- AI crawler robots.txt: ChatGPT's file disallows a token no crawler can match 21 min
- GEO engine: 23 percent of news citations came from Grok, and our registry reads none of them 15 min
- How to get cited by Claude: three named bots, and one IP list that cannot tell them apart 15 min
- llms.txt vs robots.txt: 19 of the 23 sites publishing one say nothing about AI in the other 14 min
- ChatGPT SEO: the EU has designated ChatGPT a very large online search engine 14 min
- Do AI crawlers respect robots.txt: 6 of 10 assistants never asked for the file 17 min
- How to get content cited in Google AI Overviews: 0 of 81 home pages blocked their own snippet 16 min
- Can ChatGPT read my website: on 4 of 29 sites the answer changed with the agent 14 min
- Do AI crawlers render JavaScript: 2 of 9 operators say either way 15 min
- llms.txt example: 31 real files, and one whose 75 links all 404 16 min
- robots.txt testen: six files, and not one AI crawler named 15 min
- ai.txt: 22 policy fields, and the Training default is deny 14 min
- Cloudflare pay per use: the crawl is countable and the use is not 16 min
- NLWeb: two endpoints, and a server you run rather than a file a crawler finds 13 min
- Structured data came third in a 1,100 URL citation audit 14 min
- Content signals robots.txt: one of ten Cloudflare endpoints reads it 16 min
- Microsoft Copilot SEO: Bing names the grounding query, Google names none 16 min
- ChatGPT product feed: 12 required fields, and 4 are not on your page 15 min
- Check for robots.txt: three of six sites served a different file 15 min
- Generative engine optimization: structure moved citations 7.8 points, not 17.3 14 min
- AI publisher licensing deals: 10.2 ChatGPT citations per page against 6.9 16 min
- Block AI crawlers: Cloudflare's new default depends on whether the page carries ads 16 min
- Canonical tags and AI crawlers: five of five pointed at themselves 17 min
- Crawl to refer ratio: Cloudflare says its own number may overstate 16 min
- GEO vs SEO: the crawl is the same, and four things after it are not 16 min
- LLM crawler detection: a bot declares three things and proves one 15 min
- Scan a website for all pages: 985 links found, 200,712 URLs declared 16 min
- Cloudflare will generate your robots.txt, and its token list changes without you 17 min
- How to measure GEO when the final prompt holds 35.6 percent of the vocabulary 16 min
- Schema structured data: a valid BreadcrumbList scored zero because of its syntax 18 min
- GPTBot detection: 6 of 15 crawler tokens publish a User-Agent to match 19 min
- Wildcard robots.txt: the trailing star changed nothing in 168 comparisons 16 min
- 31 disclosure widgets on five pages, and not one held an FAQ 16 min
- Deleting the main element cost 1,665 of 13,615 words 16 min
- Structured data for AI: 70 of 146 markup values were not on the page 17 min
- Title and meta rewrites gained 22 percent of retrievals in a GEO benchmark 20 min
- 1,395 of 2,729 text blocks were navigation, not main content 18 min
- Fourteen noscript elements on five pages, and not one held text 17 min
- 85 meta elements in five page heads, and two addressed a crawler 17 min
- Seven of 102 schema nodes named an author, and six were reviewers 18 min
- 85 links carried no anchor text, and 34 were named only by aria-label 19 min
- Nineteen counted brand mentions, and about two were the company 21 min
- Google deep links into a section, and 55 of 245 headings had an id 19 min
- The AI Act requires a machine-readable mark on AI text, and names no format 18 min
- Three captured pages scored full marks for schema, and one of them carried a date 15 min
- Three table elements on five real pages, and all three were navigation 19 min
- A GEO detector flagged 8.90 percent of retrieved pages, and 16.36 percent of 2026 edits 17 min
- AI usage preferences stopped inheriting, and the attachment draft is back 19 min
- Google shipped a preferred sources button, and a plain URL for sites that block third party scripts 14 min
- robots.txt allowed all 60 AI crawler probes, and 11 got a different answer 16 min
- Crawler IP range files carry two dates, and six of thirteen disagree 16 min
- Google indexes 20 document file types, and three AI crawler vendors name none 14 min
- Text in hydration JSON scored the same as no text at all, both F at 34.4 15 min
- 16 percent of the sources four AI search engines cited were AI-generated 15 min
- GEO rewriting degraded the document over five rounds 16 min
- Three of five captured pages carried JSON-LD, and one held 96 of the 102 nodes 17 min
- An h1 inside a header element counted as no h1 at all 16 min
- Deep research kept cited links above 94 percent, and no model reached 77 on facts 16 min
- JavaScript added no new crawl paths, and rendering dropped nine locale links 17 min
- 173 of 466 images carried alt text a crawler could read 14 min
- AI visibility took seven runs per prompt to settle 19 min
- GPTBot is named in 4.5 percent of robots.txt files, the wildcard in 77 16 min
- An AI usage preference cannot name a party, and two drafts propose fixes 16 min
- MCP labels every request, and the label stops at the MCP server 18 min
- Ten AI crawlers fetched markdown by URL, and none of them by content negotiation 16 min
- Google does not interact with your page, so scroll-loaded content never loads 17 min
- Retry-After names when to come back, and four crawler vendors never document it 16 min
- Search Console's generative AI report counts impressions and nothing else 15 min
- A trusted domain list moved AI citations from 12 to 21 percent 15 min
- Earmark signs an AI preference into the content, and prevents no use 18 min
- Google clears cookies between page loads, so a consent gate never opens 14 min
- Googlebot sets no Accept-Language, so your site picks the locale for it 16 min
- Meta took the crawl volume and ChatGPT kept 80 to 88 percent of the referrals 17 min
- A robots.txt that blocks every citation crawler still grades B 18 min
- Four factors decided the first citation, and formatting was not one 18 min
- IndexNow names five search engines, and a 200 only means received 17 min
- llms.txt v2 adds two link relations and drops the tool that read v1 16 min
- A brand's own domain drew 2.9 percent of AI citations 17 min
- An IETF draft makes an AI crawler's compliance claim falsifiable, not true 17 min
- Googlebot reads the first 2MB, and three AI crawler docs name no limit 16 min
- A robots.txt block did not stop 12 of 18 AI chatbots 19 min
- A stop request stopped 0 of 40 AI agents in the middle of a task 18 min
- Cloudflare shipped an agent browser that is not Chromium 15 min
- RSL writes a licence into robots.txt and says it changes no access 16 min
- A sitemap tells a crawler where pages are, not that they changed 16 min
- Blocking AI crawlers cost large publishers about 7 percent of traffic 15 min
- Google's review guideline asks for something the markup cannot record 15 min
- Meta documents five crawler tokens, and we published that it documents one 14 min
- AI Overviews cut English Wikipedia traffic by about 15 percent 17 min
- Cloudflare can unverify a crawler over a directive RFC 9309 never defined 17 min
- Cloudflare put a WebMCP interface on any site, and only a browser can find it 14 min
- Cloudflare's agent readiness score names four check groups and counts three 15 min
- AI Overviews retrieved less from sites blocking Google-Extended 17 min
- Every web agent was identifiable, and robots.txt stopped one of twelve 17 min
- Hidden text was the weakest of five attacks on AI search agents 16 min
- 17.0 percent of news outlets billed an AI crawler, and no robots.txt said so 15 min
- A renamed crawler token leaves your robots.txt group matching nothing 16 min
- An agent-ready storefront passed 89.3 percent of tasks, the baseline 49.3 19 min
- Google deleted the FAQ rich result documentation, and we still score FAQPage 16 min
- Google's paywall markup names two crawlers, and both are Google's 16 min
- Headless Chromium was blocked on 15.2 percent of the top 10,000 sites 18 min
- Three crawler documentation pages name robots.txt, and none names TDMRep 15 min
- A robots.txt edit does not take effect when you save it 16 min
- AI bot blocking stopped one of seven AI browsing agents 18 min
- Citations reached 6.8 percent of ChatGPT prompts, and the visit landed on the homepage 18 min
- The sites that block AI crawlers are the ones with editors 15 min
- 234 of 592 sites that ban GPTBot in robots.txt served it a 200 14 min
- Chrome's agentic browsing audit checks for llms.txt, Google Search ignores it 15 min
- Pay per crawl prices AI crawlers with HTTP 402, a code the spec leaves undefined 13 min
- Seven of nine AI coding agents arrived without a product name 18 min
- Common Crawl's own researchers put its persistent core near 40 percent 17 min
- Google's generative AI guide names five GEO tactics you can ignore 18 min
- More AI citations did not mean more of your page in the answer 15 min
- Six of the nine AI vendors we track publish one crawler token 16 min
- Time is selling sponsored blocks inside the markdown AI crawlers read 15 min
- A survey of 45 GEO studies puts crawling second and says few observe it 16 min
- Fourteen Google agents that a robots.txt wildcard does not stop 15 min
- Only 16 schema.org types reach 10 million domains 19 min
- Search Console added four platforms, and three of them block AI crawlers 17 min
- Cloudflare's Content-signal line asks, and the Disallow lines under it block 15 min
- Common Crawl archived 2.14 billion pages in July without running JavaScript 16 min
- How many prompts an AI visibility measurement needs 16 min
- Microsoft's AI opt out is a meta tag, not a robots.txt token 17 min
- A 404 on robots.txt allows every crawler, a 503 blocks them all 18 min
- Four OpenAI crawler IP lists, opened and counted 14 min
- The EU AI Act never writes robots.txt, and its Code of Practice does twice 16 min
- A perfect schema score and a zero, measured on the same page 23 min
- AI Overviews and the one crawler our scanner does not model 22 min
- Nearly a third of AI Overview citations are not on page one 15 min
- What the evidence actually says about llms.txt 5 min
- What a crawler actually meets on a real storefront 4 min
Guides
How to get read by AI crawlers, and what actually moves the needle
11 more
-
How do I get cited by Perplexity: two named crawlers and one API deprecation on 27 September
Perplexity documents two named crawlers with different robots.txt behaviour, and its Sonar Chat Completions endpoint is deprecated with support ending on 27 September 2026 in favour of the Agent API. -
Best AI visibility tools: two machines sold under one name
Two different products are sold as AI visibility tools: one asks models a set of prompts and counts what the answers name, the other checks whether a crawler can fetch and read the pages at all. Published plans read on 1 September 2026 show the prompt allowances are not even the same unit. -
Ten AI visibility products, and three of them fetch your pages
Ten products sold into AI visibility were read at their own published pages on 17 August 2026. Three of them examine the customer's own pages, four read a record of requests that already arrived, and six put prompts to models and read the answers back. The crawler half of the category is bundled free or sold as an annual licence, and the answer half is sold monthly by prompt volume. -
Crawl budget advice ends in a Disallow line, and AI crawlers read it too
Google's crawl budget guide, carrying Last updated 2026-07-22 UTC, tells site owners to block crawling of low value URLs with robots.txt. The guide is scoped to Googlebot and names no AI crawler. The file it tells you to edit is scoped to nobody in particular, and RFC 9309 says a crawler with no group of its own must obey the wildcard. -
What Web Bot Auth actually specifies, and what it does not
The IETF working group on crawler authentication listed ten active Internet-Drafts on 29 July 2026, two of them filed on 19 July. The core protocol draft carries intended status Informational, and every mechanism it describes sits in places an external scan of your site cannot reach. -
A user agent is a claim, not an identity
Every per-crawler number you have ever seen, including ours, counts requests that said they were a crawler. Verifying the claim needs an IP check, and most tools do not do one.
Earlier
- The AI Overviews opt out does not live on your site 23 min
- The AI preferences standard you cannot deploy yet 19 min
- Three AI crawler tokens that will never appear in your logs 5 min
- Two layers decide whether AI can read your site, and robots.txt is the weaker one 21 min
- Five structural signals that tell an AI who you are 8 min
Product
What the scanner does, and why it works that way
7 more
-
AI visibility tracking: 3 of the 8 engines we ask cannot search the web
Read out of this repository on 29 August 2026, the answer engine registry Lantad ships carries 8 engines. Five are recorded as performing live retrieval and returning the sources they grounded in. Three are recorded as having no web access and no sources at all, and one of those three is the only engine a free run asks. -
Eight named reasons our scanner refuses a URL before fetching it
A public URL box is a server-side request forgery machine unless something stops it. Lantad's guard names eight rejections, refuses 14 IPv4 ranges and applies eight IPv6 rules, re-runs in full on every redirect hop, and carries one gap we document rather than close. -
Text inside shadow DOM reaches the browser and not the extractor
Declarative shadow DOM ships component text in the HTML, inside a template element. Lantad's extraction ruleset drops template subtrees, the default DOM serialization omits shadow roots, and the shadow piercing fallback that recovers the text runs in one of our two renderers. -
A Gemini citation tool is not an AI Overview tool
What actually comes back when a tool asks Gemini a grounded question, why the cited URI can be a Google redirect rather than a publisher, and which parts of a citation table survive that. -
AI crawler detection is an ordering problem
Turning a user agent string into an AI crawler label is a matching order with named failure modes, and every one of them produces a confident wrong answer rather than an error. -
The rival names on your report were never typed in, until you could pin them
Every rival name on a Lantad report is still extracted from the answer text, through screens we built after shipping two pieces of output that should never have passed. Since 2026-08-07 you can also pin competitors to a roster, which reverses a position this post originally presented as a design choice, so the post now narrates that change instead of quietly absorbing it.
Earlier
This blog publishes what the Lantad scanner measures, dated and sourced. Every post starts from a real scan, a vendor document or a published specification, and says which. Where the evidence does not support a conclusion it says so rather than filling the gap. Definitions live in the glossary and per-stack fixes in the fix guides; this is the argued half.