Blog
The Lantad blog
What we find when we measure whether AI crawlers can actually read a site, what to do about it, and what we are changing in the product. Definitions and per-stack fixes live in the glossary and the fix guides; this is the dated, argued half.
Featured
-
AI crawler list: 201 tokens on one blocklist, and 7 of the 9 files naming 50 or more refused every citation crawler
A widely shared community blocklist, read at source on 6 November 2026, carries 204 User-agent lines resolving to 201 distinct tokens under the case-insensitive matching RFC 9309 requires, followed by a single Disallow rule for the whole site. Its own README says it holds AI related crawlers of all types, regardless of purpose, and that is literal: every one of the 15 tokens in this scanner's registry is on it, including the three whose vendors document them as the crawlers that surface and link a site in AI answers. Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 6 November 2026 and parsed the 1,031 files that resolved into at least one user-agent group. 753 named none of the 201. Nine named 50 or more, and 7 of those 9 refused all three citation crawlers at the site root.
-
GPTBot vs ClaudeBot: 22 of 1,026 robots.txt files gave the two crawlers different answers
Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 6 November 2026, as LantadBot/1.0 with redirects followed and no JavaScript executed, and resolved the site root for both tokens on the 1,026 files that parsed into at least one user-agent group. 921 allowed both, 83 refused both, 14 refused GPTBot alone and 8 refused ClaudeBot alone. 21 of those 22 split files named exactly one of the two tokens and left the other to the wildcard group, so the split is usually a consequence of what the file left unsaid.
Latest
-
AI agent traffic: Anthropic's hosted browsers identify as Claude-Agent, and 0 of 1,102 robots.txt files name it
Anthropic documents that browsers it hosts for Claude users add the token Claude-Agent to the browser's own User-Agent, and that robots settings are managed separately. Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 5 November 2026 and parsed the 1,102 that resolved into at least one user-agent group. None wrote claude-agent in a user-agent line. One file reaches it, by accident, through a bare claude token. Of the 84 files that name ClaudeBot and disallow it at the site root, 80 leave Claude-Agent allowed there. -
PetalBot robots.txt: 71 of 1,026 files named it, and 64 of those refused Huawei's AI answers with it
Huawei's own PetalBot documentation, read at source on 5 November 2026, says the index PetalBot builds both answers searches in Petal and supplies content recommendations in Huawei Assistant and AI Search, and that banning the crawler makes a site unsearchable across all Petal services. There is one robots.txt token for all of it. Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 5 November 2026. 1,101 answered HTTP 200 and 1,026 parsed into at least one user-agent group. 71 named PetalBot, and 64 of those refused it at the site root, the highest refusal rate among namers of the twelve tokens measured. 34 of the 74 sites refusing PetalBot left GPTBot allowed at that same root. -
agents.md: 46 of 1,419 sites served one at the web root, and 42 of the 46 were Shopify
The AGENTS.md format is specified for the root of a code repository, and the site that publishes it calls it a README for agents. Shopify's developer changelog of 28 May 2026 states that every store includes a default agents.md file accessible at /agents.md, and that /llms.txt and /llms-full.txt point to that content by default. Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for /agents.md on 4 November 2026. 170 answered HTTP 200, but 121 of those were hosts that answered the same way for a control path that does not exist, and 3 more returned an HTML document. 46 served a file, and 41 of the 46 carried the Shopify storefront default's own heading. -
Do AI crawlers read accordion content? 429 of 2,982 named panels were not in the HTML
Lantad requested robots.txt and then the home page of all 1,419 hostnames in this repository's two committed corpus seed files on 4 November 2026, as LantadBot/1.0 with redirects followed and no JavaScript executed. 1,109 answered HTTP 200 with HTML. 622 of them carried at least one element with aria-expanded set to false, 7,000 such elements in all. 4,061 of the 7,000 named no panel at all. The 2,939 that did named 2,982 id references, and 429 of those pointed at an element that was not in the delivered bytes. -
Fact check schema markup: Google is phasing it out, and 7 of 45 fact-check pages still declared it
Google's fact check structured data documentation, which carries a last updated date of 2026-09-08, opens by saying that support for ClaimReview markup in Google Search is being phased out while the markup remains supported by the Factcheck Explorer Tool. Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt and then the home page on 4 November 2026. 1,080 answered HTTP 200 with HTML, 622 of those carried a parseable JSON-LD block, and none declared ClaimReview. Following each home page's own fact-check links reached 46 pages on 8 hosts, 45 of which answered with HTML. 7 declared ClaimReview, every one of them on rappler.com, and all 7 carried the three properties Google documents as required. -
Brave Search crawler: 126 of 128 robots.txt files blocking an AI crawler left Googlebot allowed
Brave's own help page states that its search crawler does not advertise a differentiated user agent, and that it will not crawl a page Googlebot cannot crawl. Lantad parsed robots.txt from the 1,419 hostnames in this repository's two committed corpus seed files on 3 November 2026. 1,025 files held at least one rule group. 128 of those disallowed the site root to at least one of the 15 AI crawler tokens in core/src/bots.ts, and 126 of the 128 left the site root allowed for Googlebot, which is the only group Brave's documented behaviour follows.
Findings
Measured results from real scans and published studies
304 more
-
Claude-SearchBot: 25 of 1,024 robots.txt files refused it, and 55 of the 67 that kept it never named it
Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 3 November 2026, as LantadBot with redirects followed and no JavaScript executed, and parsed the 1,024 files that resolved into at least one user-agent group. 46 named Claude-SearchBot, the crawler Anthropic documents as the one that improves search result quality, and 25 files disallowed it at the root. 67 files refused Anthropic's training crawler and left the search crawler allowed, but only 11 of those 67 named the search crawler to allow it. In 55 of them the search crawler fell through to the wildcard group, which happened to permit the root. -
Robots.txt not working: 34 of 1,104 sites answered a crawler with an HTML page
Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 3 November 2026, as LantadBot with redirects followed and no JavaScript executed. 1,104 answered HTTP 200, and 76 of those parsed into no user-agent group at all. 34 of the 76 were HTML documents rather than robots.txt files, every one of them sent with a text/html content type, and 8 of the 34 were larger than the 500 kibibyte limit Google documents for the file. Resolving the site root for GPTBot against all 76 returns allowed, because a response with no groups in it has no rules to apply. -
AI training opt out: Google and Microsoft named one robots.txt token, and 0 of 1,024 files carry it
Google and Microsoft each published plans on 30 October 2026 to honour a single shared robots.txt signal for AI training, written as the user-agent token __usage_ai-training. Google's crawler documentation puts its support in early 2027 and Microsoft's post names 1 February 2027. Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 2 November 2026 and parsed the 1,024 files that resolved into at least one user-agent group. None named the new token, and none carried the Content-Usage line the IETF working group's own draft defines instead. 1,016 of the 1,024 already hold a wildcard group that RFC 9309 says a crawler reading the new token must obey. -
Robots meta tag: 629 of 1,023 home pages carried no page level directive at all
Lantad requested robots.txt and then the home page of all 1,419 hostnames in this repository's two committed corpus seed files on 2 November 2026, as LantadBot/1.0 with redirects followed and no JavaScript executed. 1,023 answered HTTP 200 with HTML. 629 of those carried no robots meta tag, no crawler named meta tag and no X-Robots-Tag header, so there was no page level carrier on the page for any directive at all. 380 carried a generic robots meta tag, 33 carried one addressed to a named crawler, and 12 sent the header. -
AI citation tracking: 50 to 83 percent of 37,500 citations were gone by day two, and about three quarters came back
Ahrefs published a citation stability study on 29 October 2026 that followed about 37,500 citations across five AI tools every day for 20 days. The share of day one citations missing by day two ran from 50 percent in Google AI Overviews to 83 percent in Copilot. Roughly 75 percent of the citations that dropped at least once came back inside the window and about half returned within three days, so the headline finding is churn rather than loss. Lantad measured none of this: the scanner reads sites rather than answers. What it can say is that the site side of the same question does not move on that timescale. -
How often do AI Overviews appear? One headline said over 80 percent, and the same method gave 40.4 percent in Germany
Search Engine Roundtable reported on 26 October 2026 that AI Overviews now show up for over 80 percent of Google searches, attributing the figure to recent data from Semrush, SE Ranking and seoClarity. Lantad opened all three sources on 27 October 2026. Semrush published 88 percent of 400 United States desktop commercial keywords, SE Ranking published 82.5 percent of 1,000 United States brand keywords and 40.4 percent for the same method on 1,000 German ones, and the seoClarity figure of over 62 percent is an interview remark with no study behind it. Google publishes no figure at all.
Earlier
- AI share of voice: the same 31 answers scored 33.3 percent on one surface and 36.9 on the other 17 min
- How to get cited by ChatGPT: 980 of 1,419 sites cleared every gate, and robots.txt closed 25 19 min
- Crawlable URL structure: Google clarified hyphens over underscores, and 2,203 of 80,660 corpus links used an underscore 19 min
- Verified bots: Cloudflare will grade robots.txt compliance, and 232 of 1,059 files carried a defect 18 min
- AI content tracking: a new standard names five stages, and a site can observe one of them 15 min
- AI rank tracking: 15 of 31 brand mentions carried no rank, and the answer shape decided 18 min
- Gemini UTM parameters: 171 of 1,047 sites dropped the tag during a redirect 16 min
- Address schema markup: 203 of 1,102 home pages declared one, and 19 LocalBusiness nodes omitted the property Google requires 19 min
- Google-CloudVertexBot robots.txt: 26 of 1,023 files named it, and all 26 named Google-Extended as well 20 min
- No-code SEO: 10 of 40 home pages served a crawler zero words, and our own detector could not name the builder on 20 19 min
- Robots.txt special characters: 22 rules carried a raw non-ASCII path, and our matcher bound none of them encoded 17 min
- AI brand sentiment: a word list graded 3 of 55 stored answers non-neutral, and all three were wrong 18 min
- Bingbot robots.txt: 62 of 1,019 files named it, and 83 that blocked GPTBot left it allowed 20 min
- Reduce crawl rate: Google now documents Retry-After, and 0 of 63 sites that pushed back sent it 16 min
- robots.txt Allow: 343 of 716 files wrote no line that changed a verdict, and 24 rules won on a length tie 21 min
- GoogleOther robots.txt: 63 of 1,015 files named it, and 43 that refused Google-Extended left it allowed 18 min
- Hotel schema markup: 1 of 1,093 home pages declared it, and none of the 111 Hotel nodes named a check-in time 18 min
- Multiple h1 tags: 170 of 1,059 home pages declared more than one, and 259 declared none 18 min
- Block AI training but allow AI search: the IETF vocabulary added a third category, and search now overrides the other two 17 min
- How does ChatGPT find my website? A Cloudflare pilot on about 1 percent of its network indexed hundreds of millions of pages 17 min
- Is robots.txt legally binding? An EU draft makes it evidence, and 4 of 1,419 sites served the other file it names 20 min
- Which industries block AI crawlers? 41 of 69 news files disallowed GPTBot, and 2 of 97 finance files did 19 min
- Dataset schema: 0 of 117 data pages declared it, and 2 of 1,062 home pages did 20 min
- Healthcare schema markup: 95 medical nodes, and not one declared a specialty 19 min
- Meta description for AI search: 138 of 1,068 home pages served none, and 20 served no title either 20 min
- AboutPage schema: 31 of 623 about pages declared it, and none declared the type Google documents 21 min
- Can Meta crawl my website? Its own terms say yes from 30 October, and 2 of 1,011 robots.txt files named the crawler 18 min
- ClaudeBot robots.txt: 159 of 1,014 files named it, and on 43 the name sits in the wildcard's own group 18 min
- Recipe schema: Google requires a name and an image, and 5 of 60 nodes declared an empty method or ingredient list 18 min
- AI Mode citations: five link changes on 15 October, and the only number in the post is a multiple with no base 16 min
- CCBot robots.txt: 140 of 1,013 files named it, and 15 that blocked GPTBot left it allowed 16 min
- Googlebot vs GPTBot: 81 of 1,008 robots.txt files allowed one and refused the other 18 min
- Organization schema type: 277 of 448 identity nodes declared no category, and 55 of 63 subtypes matched the industry 19 min
- Documentation for AI search: 43 of 220 docs pages gave a crawler under 100 words 18 min
- Google supplier unit: 418 of 1,419 sites gave a crawler no words to read 18 min
- PerplexityBot robots.txt: 102 of 1,013 files named it, and 20 of the 45 blocking it left the user fetcher in 15 min
- Schema markup types: 88 of the 939 schema.org defines reached 626 home pages, and the top three name the page 18 min
- Can AI crawlers read gated content? 2 of 66 password pages tripped our own login wall test 16 min
- Duplicate content and AI crawlers: 305 of 313 home pages answered at more than one live URL 15 min
- Page size limit: 25 of 1,093 home pages passed 2MB, and not one did on the wire 16 min
- Amazonbot robots.txt: 108 of 1,016 files named it, and 38 that blocked GPTBot let it in 21 min
- Applebot-Extended: 110 of 1,014 robots.txt files named Apple's training opt out, and 61 left it allowed 17 min
- Is robots.txt case sensitive? 84 mis-cased crawler tokens all bound, and 9 disallowed pages answered anyway 21 min
- Text and data mining opt out: 15 sites reserved the right in prose, and none served the machine-readable file 18 min
- AI visibility score: 972 of 1,419 corpus sites were graded, and parity took 60.7 percent of every point they lost 17 min
- Do AI crawlers read video captions? 3 of 1,078 home pages served a caption file that opened 19 min
- OAI-AdsBot: 4 of 1,014 robots.txt files named it, and one wrote a rule shaped like its job 15 min
- Robots.txt noindex: 7 of 1,057 files wrote the rule Google retired in 2019 17 min
- Bytespider robots.txt: 130 of 1,077 files named it, and 70 shut it out at the root 16 min
- Detect GPTBot: 0 of 2,344 addresses OpenAI publishes answered a reverse DNS lookup 18 min
- SoftwareApplication schema: 57 of 1,074 home pages declared one, and 4 carried every required property 17 min
- Course schema markup: 1 of 242 course pages declared it, and 124 declared nothing at all 18 min
- Do AI crawlers follow redirects? 878 of 1,322 home pages answered with one 17 min
- HowTo schema: 8 of 272 how-to pages declared it, and 109 carried their steps with no markup 18 min
- Job posting schema: 5 of 374 careers pages declared it, and 86 linked the jobs to an applicant tracking system 15 min
- Pricing page schema: 120 of 195 pricing pages showed a price and declared none in their markup 14 min
- Return policy structured data: 6 of 1,072 home pages declared it, and none of 201 policy pages did 16 min
- Robots.txt blocked resources: 78 of 925 home pages disallowed a script or stylesheet they load 20 min
- Discussion forum structured data: zero of 88 community pages declared it, and 6 shut out GPTBot 16 min
- Semantic HTML for AI: 379 of 1,066 home pages declared no main landmark 19 min
- Website schema markup: 158 of 338 sites declared both nodes and connected neither 15 min
- Do AI crawlers read web components? 3,291 of 7,124 custom elements arrived with no text 16 min
- Entity SEO: 107 of 163 schema topic statements named the site itself 18 min
- ImageObject schema: 253 home pages declared 943 nodes, and 685 carried nothing but a URL and a size 16 min
- Character encoding for AI search: 15 of 1,088 home pages declared none, and not one of them was a page 19 min
- DuckAssistBot: 63 of 1,063 robots.txt files named it, and 48 of them let it in 17 min
- Google AI Mode: 132 of 1,083 home pages sent the directive that governs it, and 3 limited anything 16 min
- Service schema markup: 20 of 616 home pages declared it, and 46 of 89 nodes carried no property the type defines 18 min
- Do AI crawlers cache your pages? 391 of 1,101 home pages offered nothing to revalidate against 20 min
- Do AI crawlers read SVG text? 80 of 37,019 inline graphics carried any 17 min
- Person schema: 111 of 1,079 sites named a human, and 162 of those names appeared nowhere else on the page 19 min
- Vary: User-Agent: 32 of 1,072 home pages declared it, and 30 of them sent both agents the same bytes 16 min
- Event schema markup: 19 of 1,419 sites declared it, and 171 of 429 start dates had passed 22 min
- noarchive: 7 of 1,077 home pages sent the tag two AI vendors read as a training opt out 19 min
- dateModified schema: 38 of 195 pages never showed the date, and 74 of 81 disagreed with their own server 16 min
- Crawl-delay in robots.txt: 152 of 1,056 sites set one, and 11 gave it to an AI crawler 17 min
- Do AI crawlers follow pagination? 116 of 466 archive pages declared a next page, and 24 offered only a button 16 min
- Google Search profile badge: 2 of 1,093 home pages referenced it, both as data Google never asked for 17 min
- Reflectionbot: 1 of 1,068 robots.txt files named it, and 72 of the 77 blocking GPTBot let it in 15 min
- GEO optimization: 76 of 225 data-nosnippet attributes sat on an element Google does not support 15 min
- ItemList schema: 17 of 605 home pages declared one, and 525 of 766 items carried no name 15 min
- Mobile first indexing and AI crawlers: 29 of 1,087 home pages answered a phone and a desktop differently 16 min
- Sitemap validator: 44 of 871 declared sitemaps delivered no file, and none of the rest broke the 50,000 URL limit 20 min
- Canonical tag: 163 of 1,069 home pages declared none, and 63 named a URL the page was not served at 20 min
- ContactPoint schema: 119 of the 191 home pages with a clickable phone number named no telephone in their markup 18 min
- Do AI crawlers read hidden text? 121,312 of 1,269,054 words sat behind a marker on 1,115 home pages 24 min
- Speakable schema: 8 of 1,114 home pages declared it, and 5 of 12 resolvable selectors pointed at no readable text 21 min
- AI crawler statistics: 77 of 706 robots.txt files blocked a citation crawler, and 42 of those were news sites 17 min
- JavaScript structured data: 27 of 404 home pages with none in the HTML gained some when a browser ran the page 17 min
- llms.txt vs sitemap.xml: a median of 23 URLs against 969 on the 239 sites publishing both 17 min
- Open Graph tags: 370 of 948 home pages left out one of the four properties the specification requires 20 min
- Sitemap lastmod: 340 of 440 pages disagreed with the Last-Modified header their own server sent 18 min
- Block ChatGPT: 38 robots.txt files refuse GPTBot at the root and let ChatGPT-User in 16 min
- No code SEO: 10 of 42 no code home pages served a crawler zero words 15 min
- Organization schema logo: 44 of 433 declared URLs returned no image, and 22 of those were 404 16 min
- SearchAction schema: 231 of 619 home pages declared a search endpoint, and robots.txt closed 57 of them 18 min
- Article schema: 27 of 165 article pages declared a headline the page's own h1 does not contain 18 min
- Do AI crawlers see lazy loaded images? 3,155 of 51,990 carried no URL in the HTML 16 min
- How to format content for AI search: 80,761 of 104,474 list items were navigation 16 min
- Review schema: 33 of 615 home pages declared a rating, and 14 rated their own business 17 min
- AI brand visibility: 412 of 1,080 home pages named their site in neither source Google reads first 18 min
- CDN blocking AI crawlers: 103 of 1,089 sites served an unknown bot and refused GPTBot 18 min
- inLanguage schema: 429 of 615 home pages carrying parseable JSON-LD named no language in it 22 min
- Structured data: 313 of 615 home pages gave no schema node an @id, and 14 reused one for two things 23 min
- Do AI crawlers respect nofollow? 1,176 of 158,327 links carried it, and 2 of 9 vendors document honouring it 17 min
- Heading structure for AI search: 311 of 704 article pages ran over 300 words with no heading 18 min
- Meta tags for AI search: 38 of 1,083 home pages sent a crawler no name at all 21 min
- sameAs schema: 85 of 317 organizations named a reference entry, and 310 named a social profile 19 min
- Do AI crawlers read PDFs? 332 on 124 home pages, and 297 carried no X-Robots-Tag 19 min
- Meta AI crawler: 6 of 1,065 robots.txt files named the token Meta ties to citation 15 min
- Do AI crawlers read images? 52,077 on 1,006 home pages, and 18,171 carried an empty alt 19 min
- Do AI crawlers read tables? 87 on 1,079 home pages, and 2 carried a caption 15 min
- FAQ schema: 83 of 1,078 sites still declare it, and 298 of 1,277 answers are not on the page 20 min
- Video schema: 21 of the 270 home pages showing a video declared one 16 min
- Author schema: 95 of 111 article pages declared an author, and 8 named a brand as a person 18 min
- Breadcrumb schema: 201 of 936 interior pages declared a trail, and 35 of those held one item 22 min
- Can AI read my pricing page? 17 of 93 SaaS pricing pages put a price in the markup 14 min
- Product schema: 42 of 50 product pages declared it, and 22 of those carried no GTIN 16 min
- Do AI crawlers read iframes? 834 on 433 home pages, and 17 held 50 words 18 min
- Do cookie banners block AI crawlers? 269 of 1,093 home pages ran a consent platform, and it cost 0.44 percent of the words 16 min
- Internal links and AI crawlers: 84 of 1,091 home pages offered a crawler no path at all 18 min
- Schema markup validator: 117 of 612 home pages with JSON-LD carried a defect, and on 7 not one block parsed 17 min
- Client side rendering: JavaScript supplied 7.6 percent of the prose, and all of it on 11 of 271 pages 17 min
- Crawl delay robots txt: 157 of 1,014 files set one, and 11 named an AI crawler 20 min
- Soft 404: 125 of 1,094 sites answered a URL that does not exist with HTTP 200 15 min
- Agentic SEO: 1 of 1,419 sites published an agent card, and 55 of the 69 OAuth documents came from a platform default 16 min
- Applebot robots.txt: 64 files hand Apple's crawler to their Googlebot group, and 44 of those answer differently 18 min
- Do AI crawlers read RSS feeds? 86 of 762 sites declared one, and 159 served an llms.txt 15 min
- robots.txt validator: 232 of 1,059 files carried a defect, and 33 sites answered with a web page 19 min
- Can GPTBot read your robots.txt? 92 of 1,056 sites refused it the file 16 min
- Content negotiation for markdown: 34 of 718 home pages answered, and six of those sent HTML 18 min
- Grok SEO: xAI documents no crawler, and 10 robots.txt files guessed 8 names 14 min
- Robots.txt subdomains: 73 of 277 gave an AI crawler a different answer from the apex 19 min
- Blocked by robots.txt: 559 of the 581 pages GPTBot lost were closed by a rule that never named it 17 min
- Crawler user agent: 1,004 robots.txt files named 2,209 tokens, and 120 name one Anthropic no longer documents 15 min
- Drupal AI crawlers: 79 of 86 robots.txt files named none, and 68 are Drupal's own file 15 min
- Robots.txt sitemap: 839 of 1,016 files declare one, and a site-wide block covers it on 92 of 93 17 min
- Does blocking AI training affect search rankings: 52 of 54 sites refusing Google-Extended still allow Googlebot 14 min
- Hreflang and AI crawlers: 27 of 719 home pages changed language on a header alone 20 min
- Web bot authentication: 2 of 9 AI crawler vendors publish a key directory 15 min
- AI structured data: 103 of 385 pages with JSON-LD named no organization 15 min
- Does page speed affect AI search: on 184 of 379 home pages a redirect took half the wait 19 min
- JSON-LD vs microdata: 22 of 385 home pages carried microdata and one carried RDFa 14 min
- LocalBusiness schema: 2 of 172 home pages carried opening hours a machine could read 16 min
- Allow AI crawlers in robots.txt: 89 of 145 files ruled only on the whole site 20 min
- Answer engine optimization: 250 of 647 home pages sent no signal that anything had changed 17 min
- Block AI training: 7 of 22 sites declaring ai-train=no blocked no crawler 16 min
- Is my site blocking AI crawlers: 79 of 115 refused one their robots.txt allows 17 min
- llms-full.txt: 32 of 44 sites served one, and the specification never names it 18 min
- Mistral AI crawler: 1 of 323 robots.txt files named one 15 min
- Most blocked AI crawler: GPTBot in 82 of 718 robots.txt files, and its search twin in 24 20 min
- Next.js AI crawlers: 28 home pages sent 15 times more payload than prose 18 min
- AI SEO: 17 of 380 home pages sent a crawler zero words 15 min
- Framer AI crawlers: 35 of 36 sites served markdown, and one robots.txt named a crawler 16 min
- Structured data: 141 of 382 home pages carried none in the raw HTML 14 min
- Webflow AI crawlers: 20 of 40 robots.txt files held no rule at all 18 min
- AI brand visibility: across 75,000 brands, the top correlate was not on the site 16 min
- robots.txt redirect: 122 of 391 sites sent one, and all 122 crossed hosts 16 min
- Squarespace AI crawlers: 61 of 66 sites named 26 and blocked none 15 min
- Wix AI crawlers: 0 of 69 robots.txt files named one, and Wix's own fix names BingAI 15 min
- Does noindex block AI crawlers: 2 of 9 operator pages name the tag 16 min
- GA4 AI Assistant channel: 2 of 6 assistant hosts answered a scan 15 min
- Shopify AI crawlers: 38 stores served 1,112 products as open JSON 14 min
- X-Robots-Tag noindex: 5 of 391 home pages sent one, and all five were a captcha 17 min
- How to get cited by Gemini: one robots.txt token covers training and grounding 16 min
- OAI-SearchBot: 16 of 140 robots.txt files named it, and 25 named GPTBot 16 min
- WordPress AI crawlers: 16 of 24 sites named none in robots.txt 14 min
- AI crawler robots.txt: ChatGPT's file disallows a token no crawler can match 21 min
- GEO engine: 23 percent of news citations came from Grok, and our registry reads none of them 15 min
- How to get cited by Claude: three named bots, and one IP list that cannot tell them apart 15 min
- llms.txt vs robots.txt: 19 of the 23 sites publishing one say nothing about AI in the other 14 min
- ChatGPT SEO: the EU has designated ChatGPT a very large online search engine 14 min
- Do AI crawlers respect robots.txt: 6 of 10 assistants never asked for the file 17 min
- How to get content cited in Google AI Overviews: 0 of 81 home pages blocked their own snippet 16 min
- Can ChatGPT read my website: on 4 of 29 sites the answer changed with the agent 14 min
- Do AI crawlers render JavaScript: 2 of 9 operators say either way 15 min
- llms.txt example: 31 real files, and one whose 75 links all 404 16 min
- robots.txt testen: six files, and not one AI crawler named 15 min
- ai.txt: 22 policy fields, and the Training default is deny 14 min
- Cloudflare pay per use: the crawl is countable and the use is not 16 min
- NLWeb: two endpoints, and a server you run rather than a file a crawler finds 13 min
- Structured data came third in a 1,100 URL citation audit 14 min
- Content signals robots.txt: one of ten Cloudflare endpoints reads it 16 min
- Microsoft Copilot SEO: Bing names the grounding query, Google names none 16 min
- ChatGPT product feed: 12 required fields, and 4 are not on your page 15 min
- Check for robots.txt: three of six sites served a different file 15 min
- Generative engine optimization: structure moved citations 7.8 points, not 17.3 14 min
- AI publisher licensing deals: 10.2 ChatGPT citations per page against 6.9 16 min
- Block AI crawlers: Cloudflare's new default depends on whether the page carries ads 16 min
- Canonical tags and AI crawlers: five of five pointed at themselves 17 min
- Crawl to refer ratio: Cloudflare says its own number may overstate 16 min
- GEO vs SEO: the crawl is the same, and four things after it are not 16 min
- LLM crawler detection: a bot declares three things and proves one 15 min
- Scan a website for all pages: 985 links found, 200,712 URLs declared 16 min
- Cloudflare will generate your robots.txt, and its token list changes without you 17 min
- How to measure GEO when the final prompt holds 35.6 percent of the vocabulary 16 min
- Schema structured data: a valid BreadcrumbList scored zero because of its syntax 18 min
- GPTBot detection: 6 of 15 crawler tokens publish a User-Agent to match 19 min
- Wildcard robots.txt: the trailing star changed nothing in 168 comparisons 16 min
- 31 disclosure widgets on five pages, and not one held an FAQ 16 min
- Deleting the main element cost 1,665 of 13,615 words 16 min
- Structured data for AI: 70 of 146 markup values were not on the page 17 min
- Title and meta rewrites gained 22 percent of retrievals in a GEO benchmark 20 min
- 1,395 of 2,729 text blocks were navigation, not main content 18 min
- Fourteen noscript elements on five pages, and not one held text 17 min
- 85 meta elements in five page heads, and two addressed a crawler 17 min
- Seven of 102 schema nodes named an author, and six were reviewers 18 min
- 85 links carried no anchor text, and 34 were named only by aria-label 19 min
- Nineteen counted brand mentions, and about two were the company 21 min
- Google deep links into a section, and 55 of 245 headings had an id 19 min
- The AI Act requires a machine-readable mark on AI text, and names no format 18 min
- Three captured pages scored full marks for schema, and one of them carried a date 15 min
- Three table elements on five real pages, and all three were navigation 19 min
- A GEO detector flagged 8.90 percent of retrieved pages, and 16.36 percent of 2026 edits 17 min
- AI usage preferences stopped inheriting, and the attachment draft is back 19 min
- Google shipped a preferred sources button, and a plain URL for sites that block third party scripts 14 min
- robots.txt allowed all 60 AI crawler probes, and 11 got a different answer 16 min
- Crawler IP range files carry two dates, and six of thirteen disagree 16 min
- Google indexes 20 document file types, and three AI crawler vendors name none 14 min
- Text in hydration JSON scored the same as no text at all, both F at 34.4 15 min
- 16 percent of the sources four AI search engines cited were AI-generated 15 min
- GEO rewriting degraded the document over five rounds 16 min
- Three of five captured pages carried JSON-LD, and one held 96 of the 102 nodes 17 min
- An h1 inside a header element counted as no h1 at all 16 min
- Deep research kept cited links above 94 percent, and no model reached 77 on facts 16 min
- JavaScript added no new crawl paths, and rendering dropped nine locale links 17 min
- 173 of 466 images carried alt text a crawler could read 14 min
- AI visibility took seven runs per prompt to settle 19 min
- GPTBot is named in 4.5 percent of robots.txt files, the wildcard in 77 16 min
- An AI usage preference cannot name a party, and two drafts propose fixes 16 min
- MCP labels every request, and the label stops at the MCP server 18 min
- Ten AI crawlers fetched markdown by URL, and none of them by content negotiation 16 min
- Google does not interact with your page, so scroll-loaded content never loads 17 min
- Retry-After names when to come back, and four crawler vendors never document it 16 min
- Search Console's generative AI report counts impressions and nothing else 15 min
- A trusted domain list moved AI citations from 12 to 21 percent 15 min
- Earmark signs an AI preference into the content, and prevents no use 18 min
- Google clears cookies between page loads, so a consent gate never opens 14 min
- Googlebot sets no Accept-Language, so your site picks the locale for it 16 min
- Meta took the crawl volume and ChatGPT kept 80 to 88 percent of the referrals 17 min
- A robots.txt that blocks every citation crawler still grades B 18 min
- Four factors decided the first citation, and formatting was not one 18 min
- IndexNow names five search engines, and a 200 only means received 17 min
- llms.txt v2 adds two link relations and drops the tool that read v1 16 min
- A brand's own domain drew 2.9 percent of AI citations 17 min
- An IETF draft makes an AI crawler's compliance claim falsifiable, not true 17 min
- Googlebot reads the first 2MB, and three AI crawler docs name no limit 16 min
- A robots.txt block did not stop 12 of 18 AI chatbots 19 min
- A stop request stopped 0 of 40 AI agents in the middle of a task 18 min
- Cloudflare shipped an agent browser that is not Chromium 15 min
- RSL writes a licence into robots.txt and says it changes no access 16 min
- A sitemap tells a crawler where pages are, not that they changed 16 min
- Blocking AI crawlers cost large publishers about 7 percent of traffic 15 min
- Google's review guideline asks for something the markup cannot record 15 min
- Meta documents five crawler tokens, and we published that it documents one 14 min
- AI Overviews cut English Wikipedia traffic by about 15 percent 17 min
- Cloudflare can unverify a crawler over a directive RFC 9309 never defined 17 min
- Cloudflare put a WebMCP interface on any site, and only a browser can find it 14 min
- Cloudflare's agent readiness score names four check groups and counts three 15 min
- AI Overviews retrieved less from sites blocking Google-Extended 17 min
- Every web agent was identifiable, and robots.txt stopped one of twelve 17 min
- Hidden text was the weakest of five attacks on AI search agents 16 min
- 17.0 percent of news outlets billed an AI crawler, and no robots.txt said so 15 min
- A renamed crawler token leaves your robots.txt group matching nothing 16 min
- An agent-ready storefront passed 89.3 percent of tasks, the baseline 49.3 19 min
- Google deleted the FAQ rich result documentation, and we still score FAQPage 16 min
- Google's paywall markup names two crawlers, and both are Google's 16 min
- Headless Chromium was blocked on 15.2 percent of the top 10,000 sites 18 min
- Three crawler documentation pages name robots.txt, and none names TDMRep 15 min
- A robots.txt edit does not take effect when you save it 16 min
- AI bot blocking stopped one of seven AI browsing agents 18 min
- Citations reached 6.8 percent of ChatGPT prompts, and the visit landed on the homepage 18 min
- The sites that block AI crawlers are the ones with editors 15 min
- 234 of 592 sites that ban GPTBot in robots.txt served it a 200 14 min
- Chrome's agentic browsing audit checks for llms.txt, Google Search ignores it 15 min
- Pay per crawl prices AI crawlers with HTTP 402, a code the spec leaves undefined 13 min
- Seven of nine AI coding agents arrived without a product name 18 min
- Common Crawl's own researchers put its persistent core near 40 percent 17 min
- Google's generative AI guide names five GEO tactics you can ignore 18 min
- More AI citations did not mean more of your page in the answer 15 min
- Six of the nine AI vendors we track publish one crawler token 16 min
- Time is selling sponsored blocks inside the markdown AI crawlers read 15 min
- A survey of 45 GEO studies puts crawling second and says few observe it 16 min
- Fourteen Google agents that a robots.txt wildcard does not stop 15 min
- Only 16 schema.org types reach 10 million domains 19 min
- Search Console added four platforms, and three of them block AI crawlers 17 min
- Cloudflare's Content-signal line asks, and the Disallow lines under it block 15 min
- Common Crawl archived 2.14 billion pages in July without running JavaScript 16 min
- How many prompts an AI visibility measurement needs 16 min
- Microsoft's AI opt out is a meta tag, not a robots.txt token 17 min
- A 404 on robots.txt allows every crawler, a 503 blocks them all 18 min
- Four OpenAI crawler IP lists, opened and counted 14 min
- The EU AI Act never writes robots.txt, and its Code of Practice does twice 16 min
- A perfect schema score and a zero, measured on the same page 23 min
- AI Overviews and the one crawler our scanner does not model 22 min
- Nearly a third of AI Overview citations are not on page one 15 min
- What the evidence actually says about llms.txt 5 min
- What a crawler actually meets on a real storefront 4 min
Guides
How to get read by AI crawlers, and what actually moves the needle
14 more
-
Best GEO tools: 5 of 7 published a price today, against 3 of 6 a month ago
Lantad requested the pricing page of the same seven GEO tools on 26 October 2026 as LantadBot, following redirects and executing no JavaScript, 32 days after reading them the first time. All seven answered HTTP 200 and five carried a numeric plan price in the bytes. Two of the seven changed verdict in that month, and one plan Lantad recorded in September has gone. -
How to rank in AI Overviews: Google is placing more links inside its AI answers, not just beneath them
On 15 October 2026 Google said it is adding more inline links to AI Overviews and AI Mode, showing them higher, and testing a top-of-answer section for official pages. Being the page an answer links is now a contest decided before the organic position you already fight for. -
Best AEO tools: three of six pricing pages carried no price a crawler could read
Lantad requested the pricing page of seven AEO tools on 24 September 2026 as LantadBot, following redirects and executing no JavaScript. One answered twice with a body under 180 bytes and no words in it. Of the six that returned a readable page, three carried a numeric plan price in the bytes and three carried none. -
How do I get cited by Perplexity: two named crawlers and one API deprecation on 27 September
Perplexity documents two named crawlers with different robots.txt behaviour, and its Sonar Chat Completions endpoint is deprecated with support ending on 27 September 2026 in favour of the Agent API. -
Best AI visibility tools: two machines sold under one name
Two different products are sold as AI visibility tools: one asks models a set of prompts and counts what the answers name, the other checks whether a crawler can fetch and read the pages at all. Published plans read on 1 September 2026 show the prompt allowances are not even the same unit. -
Ten AI visibility products, and three of them fetch your pages
Ten products sold into AI visibility were read at their own published pages on 17 August 2026. Three of them examine the customer's own pages, four read a record of requests that already arrived, and six put prompts to models and read the answers back. The crawler half of the category is bundled free or sold as an annual licence, and the answer half is sold monthly by prompt volume.
Earlier
- Crawl budget advice ends in a Disallow line, and AI crawlers read it too 17 min
- What Web Bot Auth actually specifies, and what it does not 13 min
- A user agent is a claim, not an identity 5 min
- The AI Overviews opt out does not live on your site 23 min
- The AI preferences standard you cannot deploy yet 19 min
- Three AI crawler tokens that will never appear in your logs 5 min
- Two layers decide whether AI can read your site, and robots.txt is the weaker one 21 min
- Five structural signals that tell an AI who you are 8 min
Product
What the scanner does, and why it works that way
7 more
-
AI visibility tracking: 3 of the 8 engines we ask cannot search the web
Read out of this repository on 29 August 2026, the answer engine registry Lantad ships carries 8 engines. Five are recorded as performing live retrieval and returning the sources they grounded in. Three are recorded as having no web access and no sources at all, and one of those three is the only engine a free run asks. -
Eight named reasons our scanner refuses a URL before fetching it
A public URL box is a server-side request forgery machine unless something stops it. Lantad's guard names eight rejections, refuses 14 IPv4 ranges and applies eight IPv6 rules, re-runs in full on every redirect hop, and carries one gap we document rather than close. -
Text inside shadow DOM reaches the browser and not the extractor
Declarative shadow DOM ships component text in the HTML, inside a template element. Lantad's extraction ruleset drops template subtrees, the default DOM serialization omits shadow roots, and the shadow piercing fallback that recovers the text runs in one of our two renderers. -
A Gemini citation tool is not an AI Overview tool
What actually comes back when a tool asks Gemini a grounded question, why the cited URI can be a Google redirect rather than a publisher, and which parts of a citation table survive that. -
AI crawler detection is an ordering problem
Turning a user agent string into an AI crawler label is a matching order with named failure modes, and every one of them produces a confident wrong answer rather than an error. -
The rival names on your report were never typed in, until you could pin them
Every rival name on a Lantad report is still extracted from the answer text, through screens we built after shipping two pieces of output that should never have passed. Since 2026-08-07 you can also pin competitors to a roster, which reverses a position this post originally presented as a design choice, so the post now narrates that change instead of quietly absorbing it.
Earlier
This blog publishes what the Lantad scanner measures, dated and sourced. Every post starts from a real scan, a vendor document or a published specification, and says which. Where the evidence does not support a conclusion it says so rather than filling the gap. Definitions live in the glossary and per-stack fixes in the fix guides; this is the argued half.