Every page on this site, in markdown

Paid engagements where Lantad fixes what the scan found, rather than only reporting it.

/services.md from /services

The scan is free and needs no account. Paid plans are there for what happens after: AI readability breaks quietly on a redeploy, and monitoring is what tells you the week it does.

/pricing.md from /pricing

Lantad is an AI visibility checker: it fetches your page the way a non-rendering AI crawler does, renders it the way a browser does, and reports the difference. Free, one URL, no signup.

/index.md from /

Puts two reports side by side so a change is attributable rather than remembered.

/compare.md from /compare

The full method: what is fetched, how prose parity is computed, how access is probed per crawler, and every weight in the composite.

/methodology.md from /methodology

Programmatic access to scans and reports, including the JSON a scan returns.

/developers.md from /developers

What LantadBot fetches, how it identifies itself, and how to block it. It never spoofs another crawler's user agent on the baseline fetch.

/bot.md from /bot

Tracks a set of prompts over time across several answer engines, so a change in what they say is visible rather than anecdotal.

/prompts.md from /prompts

Runs the same measurement across a set of URLs and reports them together, so a pattern across a site becomes visible.

/multiscan.md from /multiscan

Asks answer engines real questions about your brand and reports what comes back, including whether you are named at all.

/what-ai-says.md from /what-ai-says

The 15 AI crawler tokens Lantad tracks, grouped by purpose: training, search indexing, and fetches triggered by a person asking an assistant.

/tools/ai-crawlers.md from /tools/ai-crawlers

Eight checks, all free, and seven of them never ask for an email. Each one answers a single question, and each one links to the full scan when the answer turns out to be complicated.

/tools.md from /tools

Fetches your /llms.txt and reports whether it exists and how it is structured. Ships beside our own published finding that the evidence for llms.txt is weak.

/tools/llms-txt.md from /tools/llms-txt

Parses a robots.txt and reports, for each of the 15 named AI crawler tokens, whether it is allowed at the site root and which rule decided.

/tools/robots-txt-tester.md from /tools/robots-txt-tester

Fetches your page with GPTBot's published user agent and no JavaScript, then shows the text a non-rendering crawler actually receives.

/tools/what-gptbot-sees.md from /tools/what-gptbot-sees

An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.

/glossary/ai-crawler.md from /glossary/ai-crawler

AI visibility is whether AI systems can access, read, and faithfully represent your web pages, the prerequisite for being cited in an AI answer.

/glossary/ai-visibility.md from /glossary/ai-visibility

Answer Engine Optimization (AEO) is the practice of making a page readable, allowed, and structured enough that AI answer engines can use it as a source when they compose an answer.

/glossary/aeo.md from /glossary/aeo

Entity confidence is a free diagnostic on every graded Lantad report that scores, out of 100, how completely the JSON-LD in your crawler-visible HTML declares one identity and how far that identity is anchored by identifiers which resolve outside the page.

/glossary/entity-confidence.md from /glossary/entity-confidence

Generative Engine Optimization (GEO) is the practice of structuring and publishing content so that generative AI engines can fetch it, read it in full, and cite it when they answer a question.

/glossary/geo.md from /glossary/geo

llms.txt is a proposed plain-text markdown file placed at a site's root that gives AI systems a curated map of its most important content, so models can find and read what matters without parsing every page.

/glossary/llms-txt.md from /glossary/llms-txt

Prose Parity is Lantad's metric for the share of a page's rendered, human-visible text that is present in the raw HTML an AI crawler receives, where 100% means a crawler sees everything a human visitor sees.

/glossary/prose-parity.md from /glossary/prose-parity

Structured data is schema.org markup, usually written as JSON-LD, that labels a page's facts in a machine-readable format so AI systems can extract and attribute them without having to interpret the surrounding prose.

/glossary/structured-data.md from /glossary/structured-data

AI search has collected a lot of acronyms in a short time, several of which mean the same thing. These are the terms that appear on this site, with the definitions Lantad uses when it uses them.

/glossary.md from /glossary

A published Framer site is pre-rendered to static HTML and is usually readable without JavaScript, so when ChatGPT misses your content the cause is specific: a code component, an embed, or an interaction that keeps text out of the HTML.

/fix/framer.md from /fix/framer

Lovable server-renders new projects by default and serves older client-rendered projects a rendered copy to recognized crawlers, so many sites are readable. The gap is browser-fetched content and any crawler Lovable does not recognize.

/fix/lovable.md from /fix/lovable

Next.js hands AI crawlers fully rendered HTML when you use SSR, static generation, or server components, and an empty shell only when your content is rendered client-side or loaded in useEffect after the page mounts.

/fix/nextjs.md from /fix/nextjs

React is not invisible to AI, but a client-rendered single-page app ships an empty div and builds the page in the browser, so a crawler that does not run JavaScript receives a shell with almost no text.

/fix/react.md from /fix/react

Shopify themes are server-rendered Liquid, so most of your store is already readable to AI crawlers, and the real gap is the content third-party apps and widgets inject with JavaScript after the page loads.

/fix/shopify.md from /fix/shopify

Six patterns account for most of the readable content Lantad measures as missing. Each one has a specific cause in a specific stack, and a change your developer can make this week.

/fix.md from /fix

ChatGPT, Claude and Perplexity build answers from pages they could fetch and read. Five steps, worked in order, make a page one of them; nothing on the page promises a citation.

/how-to-get-cited.md from /how-to-get-cited

ChatGPT can only cite a page its crawlers are allowed to fetch and can read without JavaScript, so being permitted and readable comes first, and any citation follows from there.

/how-to-get-cited/chatgpt.md from /how-to-get-cited/chatgpt

Claude reaches your page through named Anthropic crawlers that honor robots.txt and work from your raw HTML, so being cited starts with being fetchable, allowed, and readable without JavaScript.

/how-to-get-cited/claude.md from /how-to-get-cited/claude

Google AI Overviews are built from Google's Search index, so getting cited starts with a page Googlebot can crawl, index, and read, and no site can promise the citation itself.

/how-to-get-cited/google-ai-overviews.md from /how-to-get-cited/google-ai-overviews

To be cited by Perplexity, a page first has to be readable: it must fetch cleanly, carry its real text in the raw HTML before any JavaScript runs, and stay open to PerplexityBot in robots.txt.

/how-to-get-cited/perplexity.md from /how-to-get-cited/perplexity

Ahrefs Brand Radar tracks brand mentions and citations across AI platforms at very large scale; Lantad measures whether an AI crawler could read the page those citations would point at, and names the fix when it could not.

/vs/ahrefs.md from /vs/ahrefs

Honest comparisons against named alternatives, including what each of them does better than Lantad.

/vs.md from /vs

Otterly.AI tracks what AI engines already say about your brand; Lantad measures and fixes whether AI crawlers can read your site in the first place.

/vs/otterly.md from /vs/otterly

Peec AI tracks what AI assistants already say about your brand; Lantad measures and fixes whether AI crawlers can read your site in the first place.

/vs/peec.md from /vs/peec

Profound tracks what AI assistants say about your brand across many engines; Lantad tests and fixes whether AI crawlers can read your pages in the first place.

/vs/profound.md from /vs/profound

PromptAlpha bundles AI answer tracking with done-for-you AEO articles; Lantad measures whether AI can read your pages, tracks what 8 answer engines say, and deliberately does not sell content production.

/vs/promptalpha.md from /vs/promptalpha

Semrush's AI Visibility Toolkit reports how often AI answers mention your brand; Lantad measures whether an AI crawler could read the page those answers would have to cite, and names the fix when it could not.

/vs/semrush.md from /vs/semrush

Counted on 25 August 2026 from five real pages this scanner captured and stored on 15 July 2026, the crawler view holds 16,636 visible words and 2,729 text blocks. 1,395 of those blocks sit inside nav, header, footer or aside, and between them they carry 2,920 words. More than half the blocks hold under a fifth of the text.

/blog/half-the-text-blocks-were-navigation.md from /blog/half-the-text-blocks-were-navigation

An audit posted to arXiv on 22 May 2026 put 712 queries to ChatGPT, Copilot, Gemini and Perplexity through their own interfaces, collected 26,266 cited URLs and scraped 19,154 of them. A detection classifier labelled 3,056 of those, approximately 16 percent, as AI-generated, ranging from 7.3 percent of ChatGPT's cited sources to 27.8 percent of Copilot's. The authors present the figure as a lower bound, because the same classifier missed 31.4 percent of a sample of articles already documented as AI-generated.

/blog/sixteen-percent-of-cited-sources-were-ai-generated.md from /blog/sixteen-percent-of-cited-sources-were-ai-generated

A probe of 185 media properties published on 4 August 2026 reports that 17.0 percent of the outlets giving a clean baseline answer at least one AI vendor with HTTP 402 Payment Required, that the gate is selective by vendor, and that none of it is declared in robots.txt.

/blog/publisher-tolls-are-invisible-in-robots-txt.md from /blog/publisher-tolls-are-invisible-in-robots-txt

Counted with an HTML parser over five real pages this scanner captured and stored on 15 July 2026, the rendered DOM held 466 img elements: 173 with a non-empty alt attribute, 246 with an explicitly empty one, and 47 with no alt attribute at all. Our own extractor reads none of them.

/blog/alt-text-on-466-captured-images.md from /blog/alt-text-on-466-captured-images

HasData fetched as well as read. Across a 2,096 domain enforcement subset measured in July 2026, 39.5 percent of the sites disallowing GPTBot returned it a live 200 when a request carrying that name actually knocked.

/blog/gptbot-bans-served-a-200-anyway.md from /blog/gptbot-bans-served-a-200-anyway

Counted on 26 August 2026 across the five real pages this scanner captured on 15 July 2026, the details element appears 31 times, every one collapsed by default, and not a single summary label contains a question mark.

/blog/thirty-one-disclosure-widgets-held-no-faq.md from /blog/thirty-one-disclosure-widgets-held-no-faq

Counted with an HTML parser over five real pages this scanner captured and stored on 15 July 2026, the rendered DOM holds 1,951 links with an href and 85 of them carry no text between the tags. Google's link documentation names two fallbacks for an empty link, the title attribute and a nested image's alt text. Thirty four of the 85 carry neither, and 23 carry none of the three.

/blog/textless-links-named-only-by-aria-label.md from /blog/textless-links-named-only-by-aria-label

Counted with an HTML parser on 24 August 2026 over five real pages this scanner captured and stored on 15 July 2026, the head of those documents holds 85 meta elements. Two are a robots meta tag, and both of those carry nothing but an image preview size. Five of the five tell a social network how to draw a card.

/blog/eighty-five-meta-elements-two-addressed-a-crawler.md from /blog/eighty-five-meta-elements-two-addressed-a-crawler

RFC 9309 gives a missing robots.txt and a broken robots.txt opposite meanings: a 4xx lets a crawler access any resource on the server, a 5xx obliges it to assume complete disallow. Google's documentation carves 429 out of that rule. Lantad's code does not.

/blog/robots-txt-404-and-503-are-opposites.md from /blog/robots-txt-404-and-503-are-opposites

A preprint posted to arXiv on 18 June 2026 by a single author at the AI visibility vendor Ranqo breaks 149,912 citations from five AI engines into nine source classes. The brand's own domain accounts for 4,392 of them, 2.9 percent, while 75.2 percent point at other companies in the same category.

/blog/ai-citations-mostly-point-at-other-companies.md from /blog/ai-citations-mostly-point-at-other-companies

What actually comes back when a tool asks Gemini a grounded question, why the cited URI can be a Google redirect rather than a publisher, and which parts of a citation table survive that.

/blog/a-gemini-citation-tool-is-not-an-ai-overview-tool.md from /blog/a-gemini-citation-tool-is-not-an-ai-overview-tool

A benchmark posted to arXiv on 17 August 2026 fetched 10,095 pages out of Google Search and Gemini results between 28 and 31 July 2026, flagged 898 of them as optimised for generative engines, and found that the off-the-shelf detectors it tested were partly reading who wrote a page rather than whether anyone had tuned it.

/blog/geo-detector-flagged-8-9-percent-of-retrieved-pages.md from /blog/geo-detector-flagged-8-9-percent-of-retrieved-pages

Lantad reads the same JSON-LD twice with two different traversals, and on our stored Wikipedia capture one reading returns a perfect 1.0 while the other returns 0 out of 100. Both are right, and the gap is the honest answer to what structured data does for AI.

/blog/structured-data-measured-twice.md from /blog/structured-data-measured-twice

Google replaced Google-NotebookLM with Google-GeminiNotebook on 16 July 2026 and supports the old name only until August 2026. A group headed by a name no crawler answers to falls through to the wildcard, or to no rules at all, and nothing on your site reports it.

/blog/renamed-crawler-tokens-leave-stale-robots-rules.md from /blog/renamed-crawler-tokens-leave-stale-robots-rules

A canary token study of 20 websites, posted to arXiv on 13 May 2026, mapped 22 AI chatbots to the scrapers that fed them. After the sites were taken offline and after a robots.txt file disallowed every bot, 12 of the 18 measurable systems still returned their content, and 10 of those 18 had been fed by a search engine's crawler rather than by any AI crawler of the vendor's own.

/blog/robots-txt-block-did-not-stop-twelve-chatbots.md from /blog/robots-txt-block-did-not-stop-twelve-chatbots

A controlled experiment run over 40 days on university web logs reports fewer than 40 percent of AI assistant and AI search bots re-fetching robots.txt inside a 168 hour window, and 15 of 47 bots never fetching it at all during two of its experiments. RFC 9309 asks crawlers not to use a cached copy for more than 24 hours.

/blog/when-a-robots-txt-edit-reaches-a-crawler.md from /blog/when-a-robots-txt-edit-reaches-a-crawler

Lantad's stored fixture for a site that disallows the six AI crawlers capable of citing it scores 78.8 out of 100 and grades B, because access is weighted at 25 points and only 20 of them are lost. An audit of our own scale, read out of our own repository.

/blog/a-robots-txt-that-blocks-every-citation-crawler-still-grades-b.md from /blog/a-robots-txt-that-blocks-every-citation-crawler-still-grades-b

Google's sitemap documentation, carrying Last updated 2026-07-08 UTC, says it uses the lastmod value if it is consistently and verifiably accurate, and that it ignores changefreq and priority. The crawler documentation published by OpenAI, Anthropic and Perplexity, read on 8 August 2026, does not mention sitemaps at all. The mechanism that can settle whether a page changed lives in HTTP.

/blog/sitemap-lastmod-is-an-assertion.md from /blog/sitemap-lastmod-is-an-assertion

A paper revised on arXiv on 1 August 2026 asks whether an LLM agent honours a server's request to stand down. At the access door four frontier models recused at rates from 100 percent to 55 percent. Delivered mid task to two OpenAI models it stopped nobody, 0 of 40, while a later run of the same apparatus had Gemini 2.5 Flash stopping 16 of 20.

/blog/mid-task-stop-request-stopped-zero-of-forty.md from /blog/mid-task-stop-request-stopped-zero-of-forty

A critical survey posted to arXiv on 15 July 2026 reviewed 45 generative engine optimization studies and reported that none of them shows a stable, cross-platform causal effect on organic discoverability. Its own pipeline diagram puts crawling and indexing at stage two of seven, and its caption says far fewer studies look there.

/blog/geo-survey-45-studies-crawling-stage.md from /blog/geo-survey-45-studies-crawling-stage

A pre-launch expert evaluation of Evrópuvefur, a government-funded Icelandic service answering questions about the EU, compared a curated corpus against open web search across 449 answers. Naming roughly sixty trusted domains in the system prompt raised the share of citations landing on one of them from 12 percent to 21 percent, and the country's most widely used news source was never cited across 287 web-search answers.

/blog/trusted-domain-list-moved-citations-12-to-21-percent.md from /blog/trusted-domain-list-moved-citations-12-to-21-percent

Every per-crawler number you have ever seen, including ours, counts requests that said they were a crawler. Verifying the claim needs an IP check, and most tools do not do one.

/blog/a-user-agent-is-a-claim-not-an-identity.md from /blog/a-user-agent-is-a-claim-not-an-identity

Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 18 October 2026, read every permitted home page with no JavaScript executed, then followed each page's own link to an about, company or team surface and read that too. 623 of those pages answered HTTP 200 with HTML. 31 declared schema.org AboutPage, a type that defines no properties of its own, and not one declared ProfilePage, the page type Google's structured data gallery documents for a page about a single person or organisation.

/blog/aboutpage-schema-31-of-623-about-pages.md from /blog/aboutpage-schema-31-of-623-about-pages

Lantad fetched the home page of all 1,419 hostnames in this repository's two committed corpus seed files on 23 October 2026 and parsed the JSON-LD of the 1,102 that answered. 514 sites declared an Organization or Place family node, and 283 of those 514 published no address anywhere in their markup. 203 sites declared a PostalAddress, 438 nodes in total, and 19 of the 173 LocalBusiness family nodes carried no address property at all, which is one of the two properties Google documents as required for the type.

/blog/address-schema-markup-203-of-1102-declared-one.md from /blog/address-schema-markup-203-of-1102-declared-one

The only agent discovery path any standards registry lists is /.well-known/agent-card.json. Lantad asked all 1,419 hostnames in this repository's committed corpus for it on 19 September 2026, and one returned a usable card. 69 of the same hosts served an OAuth protected resource document at the path an MCP client reads, and 55 of those 69 carry the signature of a Shopify or WordPress default rather than of a file written for that site.

/blog/agentic-seo-one-agent-card-in-1419-sites.md from /blog/agentic-seo-one-agent-card-in-1419-sites

A controlled study posted to arXiv on 2 May 2026 turned on both of Cloudflare's free AI bot controls and then ran seven AI browsing agents through three tasks on the authors' own site. Six finished. The one that was stopped is listed in Cloudflare's bot directory, which the authors read as self-identification rather than detection.

/blog/cloudflare-blocked-one-of-seven-browsing-agents.md from /blog/cloudflare-blocked-one-of-seven-browsing-agents

Lantad ran its own word-list sentiment grader over the 55 answers committed at core/fixtures/tracker-golden/goldens.json on 22 October 2026, using wordlistSentiment from core/src/sentiment.ts and the brand identity for lantad.co. It returned 3 positive labels, 52 neutral and 0 negative. All three positives were wrong: one scored a word describing what AI does, one scored a word describing a competitor named in the same sentence, and one read a brand dismissal as praise because the negator sat eight tokens from the term against a negation window of three.

/blog/ai-brand-sentiment-3-of-55-all-three-wrong.md from /blog/ai-brand-sentiment-3-of-55-all-three-wrong

Lantad requested the robots.txt and then the home page of all 1,419 hostnames in this repository's committed corpus on 26 September 2026 and read the raw bytes with no JavaScript executed. 1,080 home pages were both allowed and readable. 412 of them carried neither a named WebSite node in structured data nor an og:site_name, the two sources Google's site names documentation ranks above the title element, and 29 of the 291 pages that carried both gave two different names.

/blog/ai-brand-visibility-412-of-1080-named-no-site.md from /blog/ai-brand-visibility-412-of-1080-named-no-site

Ahrefs correlated search and brand metrics against brand mentions in ChatGPT, Google AI Mode and AI Overviews across 75,000 brands. YouTube mentions led at about 0.737 and number of site pages sat near 0.194. Lantad measured none of it, and the layer we do measure sits underneath the whole ranking.

/blog/ai-brand-visibility-75000-brands-top-signal-off-site.md from /blog/ai-brand-visibility-75000-brands-top-signal-off-site

The Content Telemetry Specification version 1.0, dated 2026-09-02, splits AI content use into five reported events, and assigns three of them to the AI agent alone. Lantad asked all 1,419 corpus hostnames for the discovery file it defines on 24 October 2026 and none served one.

/blog/ai-content-tracking-five-stages-one-observable.md from /blog/ai-content-tracking-five-stages-one-observable

Turning a user agent string into an AI crawler label is a matching order with named failure modes, and every one of them produces a confident wrong answer rather than an error.

/blog/ai-crawler-detection-is-an-ordering-problem.md from /blog/ai-crawler-detection-is-an-ordering-problem

Lantad requested /robots.txt once from each of 20 AI company hostnames on 7 September 2026. All 20 answered HTTP 200, five name at least one of the 15 AI crawler tokens this scanner evaluates, and one of those names is spelled with a non-breaking hyphen, which RFC 9309 does not allow in a product token, so the group refuses nothing.

/blog/ai-crawler-robots-txt-one-token-no-crawler-matches.md from /blog/ai-crawler-robots-txt-one-token-no-crawler-matches

Lantad requested robots.txt and the home page from all 1,027 hostnames in this repository's committed industry corpus seed file on 29 September 2026 as LantadBot. 706 returned a parseable robots.txt. 77 of those disallowed GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot at the root, and 42 of the 77 came from the news stratum alone.

/blog/ai-crawler-statistics-77-of-706-blocked-42-were-news.md from /blog/ai-crawler-statistics-77-of-706-blocked-42-were-news

Google published a Search post on 15 October 2026 naming five changes to how AI Mode and AI Overviews link out to the web, three of them described as tests rather than rollouts. The post gives no number or percentage for the increase in inline links it announces. The only figure in it is that Preferred Sources users have grown more than 7x since the start of the year, stated with no base count and no method. Lantad has measured none of these surfaces and does not claim to.

/blog/ai-mode-citations-five-changes-and-one-number.md from /blog/ai-mode-citations-five-changes-and-one-number

Getting quoted in an AI Overview starts with Googlebot, and Googlebot is not one of the fifteen tokens in our crawler registry. An audit of our own blind spot, read line by line against our own source.

/blog/ai-overviews-and-the-crawler-we-do-not-model.md from /blog/ai-overviews-and-the-crawler-we-do-not-model

A difference in differences study posted to arXiv on 5 February 2026 and revised on 12 May 2026 compares 52,262 English Wikipedia articles against the same articles in four language editions the rollout had not reached, across 46,534,093 daily observations, and estimates that AI Overviews exposure cut daily views to the English versions by roughly 15 percent. Wikipedia is one of the sources those answers cite most.

/blog/ai-overviews-cut-wikipedia-traffic-15-percent.md from /blog/ai-overviews-cut-wikipedia-traffic-15-percent

A SIGIR 2026 study of 11,500 queries collected on 7 and 8 December 2025 found Google AI Overviews significantly less likely than Google Search to retrieve content from sites that block Google-Extended, the token Google documents as not affecting Search. Gemini never cited the study's 21 blocking publishers at all.

/blog/ai-overviews-retrieved-less-from-google-extended-blockers.md from /blog/ai-overviews-retrieved-less-from-google-extended-blockers

A joint study by Press Ranger and OtterlyAI, announced on 20 August 2026, cross-referenced 129.3 million citations captured across seven AI platforms during June 2026 against 91 confirmed licensing agreements mapped to 314 publisher domains. Pages from publishers with an OpenAI deal collected 10.2 citations each on average on ChatGPT against 6.9 for pages from unlicensed publishers. The same pattern did not appear for Google or Perplexity deals on their own platforms.

/blog/ai-publisher-licensing-deals-10-2-citations-against-6-9.md from /blog/ai-publisher-licensing-deals-10-2-citations-against-6-9

Lantad read position and answer shape out of the 55 answers committed at core/fixtures/tracker-golden/goldens.json on 24 October 2026, using the stored labels and the recorded judge replies. 31 of the 55 name the brand. 16 of those 31 carry a position and 15 carry none. All 12 of the list answers that named the brand carried a rank, against 4 of the 17 prose answers, so the shape of the answer rather than the brand's standing decided whether there was a rank to report at all.

/blog/ai-rank-tracking-15-of-31-mentions-carried-no-rank.md from /blog/ai-rank-tracking-15-of-31-mentions-carried-no-rank

Lantad requested the home page of 392 hostnames on 11 September 2026 and read the bytes with no JavaScript executed. 380 answered HTTP 200, and 17 of those carried no visible text at all. Every one of the 17 carried a title element, 13 carried a meta description, and the two together came to a median of 17 words per site.

/blog/ai-seo-17-of-380-home-pages-sent-zero-words.md from /blog/ai-seo-17-of-380-home-pages-sent-zero-words

Lantad replayed all 55 answers committed at core/fixtures/tracker-golden/goldens.json through its own shipped extraction pipeline on 26 October 2026, using detectAliasMention, parseJudgeReply and reconcileExtraction from core/src/tracker.ts. 31 of the 55 resolved to a brand mention. Feeding those same 31 into the arithmetic the dashboard uses returns 33.3 percent, because it divides by 93, and feeding them into the arithmetic the presence report uses returns 36.9 percent, because it divides by 84. The numerator is identical. Only the denominator moved, and the function both paths call carries a comment saying one of the two denominators must never be used.

/blog/ai-share-of-voice-33-3-on-one-surface-36-9-on-the-other.md from /blog/ai-share-of-voice-33-3-on-one-surface-36-9-on-the-other

Lantad requested the home page of 1,027 hostnames on 14 September 2026 and read the bytes with no JavaScript executed. 713 answered with a 2xx status and an HTML content type, and 385 of those carried at least one JSON-LD block. On 103 of the 385 the markup declared no organization complete enough to identify a publisher, and 27 of the 103 were one property short of doing so.

/blog/ai-structured-data-103-of-385-named-no-organization.md from /blog/ai-structured-data-103-of-385-named-no-organization

Both IETF AI preferences working group drafts carry the date 19 August 2026. The vocabulary deleted the rule that let a preference stated for a broad usage category cover a narrower one, so an unstated category is now unknown and nothing else, and the companion draft defining the robots.txt Content-Usage rule is current again after 110 days behind an expiry.

/blog/ai-usage-preferences-stopped-inheriting.md from /blog/ai-usage-preferences-stopped-inheriting

Lantad ran its shipped scanner, browser and all, over every hostname in this repository's two committed corpus seed files on 13 October 2026. 972 of the 1,419 produced a composite score and a grade. Across those 972 the four sub-scores gave up 24,177 points between them, and prose parity accounted for 14,674 of them. Access, weighted at a quarter of the scale, accounted for 1,501.

/blog/ai-visibility-score-972-of-1419-were-graded.md from /blog/ai-visibility-score-972-of-1419-were-graded

A study from the University of St. Gallen posted to arXiv on 8 April 2026 queried four AI search engines with 32 prompts over 45 to 46 days, then repeated each prompt up to 10 times inside one day. A per-brand visibility rate estimated from a single run carried a standard error of 0.370, and seven runs were needed to bring it under 0.10.

/blog/ai-visibility-took-seven-runs-per-prompt-to-settle.md from /blog/ai-visibility-took-seven-runs-per-prompt-to-settle

Read out of this repository on 29 August 2026, the answer engine registry Lantad ships carries 8 engines. Five are recorded as performing live retrieval and returning the sources they grounded in. Three are recorded as having no web access and no sources at all, and one of those three is the only engine a free run asks.

/blog/ai-visibility-tracking-three-of-eight-engines-cannot-search.md from /blog/ai-visibility-tracking-three-of-eight-engines-cannot-search

An IETF Independent Submission dated 12 June 2026 asks IANA to register /.well-known/ai.txt and /.well-known/ai.json. It defines 22 field names, and a file that never mentions training has already refused it.

/blog/ai-txt-well-known-training-defaults-to-deny.md from /blog/ai-txt-well-known-training-defaults-to-deny

Lantad requested /robots.txt once from each of 1,027 hostnames on 13 September 2026. 706 responses parsed into at least one group, and 145 of those name at least one of the 15 AI crawler tokens this scanner evaluates. On 89 of the 145, every rule in those groups governs the whole site. Fifty six write at least one path rule, and on 29 of the 145 deleting the AI rules entirely would have changed no verdict we tested.

/blog/allow-ai-crawlers-robots-txt-89-of-145-whole-site.md from /blog/allow-ai-crawlers-robots-txt-89-of-145-whole-site

Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 14 October 2026. 1,016 parsed into at least one user-agent group. 108 of them named Amazonbot, 48 of those 108 disallowed it at the root, and 38 files across the whole set disallowed GPTBot at the root while leaving Amazonbot in, 35 of them without ever naming Amazonbot. Amazon's own crawler page documents three user agents, says they do not support crawl-delay, and says they may use a cached robots.txt copy from the last 30 days, against the 24 hours RFC 9309 recommends.

/blog/amazonbot-robots-txt-108-of-1016-named-it.md from /blog/amazonbot-robots-txt-108-of-1016-named-it

A controlled experiment posted to arXiv on 13 July 2026 put three browser agents through five tasks on two versions of one prototype storefront, 300 runs in total. The agent-ready version recorded 134 passes out of 150 against 74 out of 150, and both versions were driven by a real browser, so nothing the agents needed was hidden from them.

/blog/agent-ready-storefront-task-success.md from /blog/agent-ready-storefront-task-success

The IETF vocabulary for AI usage preferences defines two categories and two values, y and n, and offers no way to scope either to a named company. Two individual submissions filed on 6 August 2026 propose two different places to put the missing permission.

/blog/ai-usage-preferences-cannot-name-a-party.md from /blog/ai-usage-preferences-cannot-name-a-party

The raw HTML of five pages this scanner captured on 15 July 2026 holds 245 heading elements and our extractor recorded 166, because it discards every heading inside nav, header, footer or aside. On two of those pages the rule turned a real h1 into a reported count of zero.

/blog/an-h1-inside-a-header-counted-as-none.md from /blog/an-h1-inside-a-header-counted-as-none

draft-reilly-aipref-compliance-00, an individual submission dated 2 August 2026, defines a signed record binding a fetched asset, the preference in force when it was fetched, and the usage category the AI system assigned to it. Its own abstract says the mechanism is confined to evidence and takes no position on the legal effect of any preference.

/blog/compliance-records-are-falsifiable-not-proof.md from /blog/compliance-records-are-falsifiable-not-proof

Lantad requested the home page of all 1,027 hostnames in this repository's industry corpus on 13 September 2026, then immediately asked each one again using the validator it had just issued. 647 answered HTTP 200 with an HTML content type. 273 of those sent neither a Last-Modified header nor an ETag, 111 sent one and then ignored it, and 250 carried no machine-readable statement of change anywhere in the response.

/blog/answer-engine-optimization-250-of-647-no-change-signal.md from /blog/answer-engine-optimization-250-of-647-no-change-signal

Apple documents that when a robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot instructions. Lantad requested /robots.txt once from each of 1,419 hostnames on 19 September 2026. 1,023 parsed into at least one group, 22 named Applebot, and 64 named Googlebot without naming Applebot. On 44 of those 64 the Googlebot group and the wildcard group give different answers, and on five the wildcard refuses the whole site while the Googlebot group lets Apple in.

/blog/applebot-robots-txt-64-files-defer-to-googlebot.md from /blog/applebot-robots-txt-64-files-defer-to-googlebot

Lantad requested /robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 14 October 2026, as LantadBot/1.0, redirects followed, no JavaScript executed. 1,093 answered HTTP 200, 1,059 of those served something other than HTML, and 1,014 parsed into at least one group. 110 of the 1,014 named Applebot-Extended. 49 of the 110 disallowed it at the root and 61 left it allowed. On 63 of the 110 the root verdict was the same one the wildcard group would have given with the line deleted.

/blog/applebot-extended-110-of-1014-robots-txt-files-named-it.md from /blog/applebot-extended-110-of-1014-robots-txt-files-named-it

Lantad followed the home page's own links one level deeper on the 1,027 hostnames in this repository's committed industry corpus on 27 September 2026 and read 1,178 interior pages with no JavaScript executed. 176 of them declared exactly one Article node in JSON-LD. On 27 of the 165 that also served an h1, neither string contained the other, and 30 of the 176 declared no dateModified at all.

/blog/article-schema-27-of-165-headline-not-in-h1.md from /blog/article-schema-27-of-165-headline-not-in-h1

Lantad followed the home page's own links one level deeper on the 1,027 hostnames in this repository's committed industry corpus on 22 September 2026 and read 563 interior pages. 111 declared themselves an Article in JSON-LD. 95 of those carried an author property, 89 resolved to a name, and 8 of the 63 nodes typed as a Person named a publication, a team or a label rather than a human being.

/blog/author-schema-95-of-111-declared-an-author.md from /blog/author-schema-95-of-111-declared-an-author

Lantad requested the pricing page of seven AEO tools on 24 September 2026 as LantadBot, following redirects and executing no JavaScript. One answered twice with a body under 180 bytes and no words in it. Of the six that returned a readable page, three carried a numeric plan price in the bytes and three carried none.

/blog/best-aeo-tools-three-of-six-published-no-price.md from /blog/best-aeo-tools-three-of-six-published-no-price

Two different products are sold as AI visibility tools: one asks models a set of prompts and counts what the answers name, the other checks whether a crawler can fetch and read the pages at all. Published plans read on 1 September 2026 show the prompt allowances are not even the same unit.

/blog/best-ai-visibility-tools-two-machines-one-category.md from /blog/best-ai-visibility-tools-two-machines-one-category

Lantad requested the pricing page of the same seven GEO tools on 26 October 2026 as LantadBot, following redirects and executing no JavaScript, 32 days after reading them the first time. All seven answered HTTP 200 and five carried a numeric plan price in the bytes. Two of the seven changed verdict in that month, and one plan Lantad recorded in September has gone.

/blog/best-geo-tools-5-of-7-published-a-price.md from /blog/best-geo-tools-5-of-7-published-a-price

Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 22 October 2026, as LantadBot with redirects followed and no JavaScript executed. 1,019 parsed into at least one user-agent group. 62 named Bingbot and 4 denied it the site root, against 175 naming GPTBot and 87 denying it. 83 files closed the root to GPTBot and left Bingbot open, and not one file did the reverse.

/blog/bingbot-robots-txt-62-of-1019-named-it.md from /blog/bingbot-robots-txt-62-of-1019-named-it

Cloudflare's bots changelog entry of 1 July 2026 states that from 15 September 2026 new domains onboarding to Cloudflare receive defaults under which bots classified as Training or as Agent are blocked on pages that display ads, while Search remains allowed. That is a rule about a page, and robots.txt has no way to write one.

/blog/block-ai-crawlers-cloudflare-default-turns-on-ads.md from /blog/block-ai-crawlers-cloudflare-default-turns-on-ads

draft-ietf-aipref-vocab-08, published 14 September 2026, replaces the two usage categories of revision 07 with three: train-ai, ai-use and search. Section 4.3 states that the search category overrides any usage falling into the other categories, which speaks directly to the request site owners make most often. None of it is deployable. The aipref charter set 31 August 2026 for both working group documents to reach the IESG, neither has been sent, and the vocabulary's own Note to Readers says its contents do not reflect consensus of the working group either in whole or part.

/blog/block-ai-training-allow-ai-search-vocab-08-third-category.md from /blog/block-ai-training-allow-ai-search-vocab-08-third-category

Lantad requested /robots.txt once from each of 1,027 hostnames on 13 September 2026. 736 answered HTTP 200 and 38 of those carry a Content-Signal line. Twenty two of the 38 declare ai-train=no. Fifteen of the 22 also disallow eight or nine of the nine training crawler tokens this scanner evaluates by name, and seven disallow none of them at the site root.

/blog/block-ai-training-seven-of-22-blocked-nobody.md from /blog/block-ai-training-seven-of-22-blocked-nobody

Lantad asked all 1,419 hostnames in this repository's committed corpus for robots.txt once each on 28 September 2026 and evaluated the site root for every OpenAI token with the scanner's own parser. 1,025 files parsed into at least one group. 172 named GPTBot, 103 named ChatGPT-User, 80 named OAI-SearchBot and 4 named OAI-AdsBot, and on 38 files a rule refuses GPTBot at the root while ChatGPT-User is allowed there.

/blog/block-chatgpt-38-files-refuse-gptbot-allow-chatgpt-user.md from /blog/block-chatgpt-38-files-refuse-gptbot-allow-chatgpt-user

Lantad requested the home page and the robots.txt of 1,027 hostnames on 17 September 2026 and evaluated every same host link on those home pages against the scanner's robots parser. On the 563 sites where GPTBot could fetch the home page, 581 of the 58,813 linked paths were disallowed to it. 559 of those 581 were decided by the wildcard group, and only 22, spread over nine sites, by a group that names GPTBot at all.

/blog/blocked-by-robots-txt-559-of-581-by-a-wildcard.md from /blog/blocked-by-robots-txt-559-of-581-by-a-wildcard

A staggered difference in differences study of news publishers, arXiv 2512.24968, estimates roughly a 7 percent traffic decline within 6 weeks of first disallowing a generative AI crawler. Split by publisher rank, the decline sits in the top 50 and the point estimate turns slightly positive below rank 100.

/blog/blocking-ai-crawlers-cost-7-percent-of-traffic.md from /blog/blocking-ai-crawlers-cost-7-percent-of-traffic

Lantad asked all 1,419 hostnames in this repository's committed corpus for robots.txt on 22 September 2026, then followed each home page's own first link one level deeper and read the interior page with no JavaScript executed. 936 answered HTTP 200. 201 of them declared a BreadcrumbList in JSON-LD, 35 of the 202 trails those pages carried held a single item, and 165 of the 202 named no level between the site root and the page.

/blog/breadcrumb-schema-201-of-936-declared-a-trail.md from /blog/breadcrumb-schema-201-of-936-declared-a-trail

Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 12 October 2026 and parsed the 1,077 that answered with a robots file. 130 of them named Bytespider, ByteDance's crawler. 70 disallowed it at the root, a higher share of the files that name it than GPTBot, ClaudeBot, Amazonbot, Meta-ExternalAgent, Google-Extended or Applebot-Extended reached. It is also the one crawler in that set that Cloudflare's new training controls, announced on 15 September 2026, do not cover.

/blog/bytespider-robots-txt-130-of-1077-named-it.md from /blog/bytespider-robots-txt-130-of-1077-named-it

Lantad requested robots.txt from all 1,419 hostnames in this repository's two committed corpus seed files on 15 October 2026, then read 934 home pages and 1,186 resource pages it was permitted to read, with no JavaScript executed. 68 of those 2,120 pages showed the crawler an HTML password input. Of the 66 that answered a second read the same day, 2 met the condition this scanner uses to withhold a grade for a login or paywall, and both were facebook.com rather than the site being measured.

/blog/can-ai-crawlers-read-gated-content-2-of-66-password-pages.md from /blog/can-ai-crawlers-read-gated-content-2-of-66-password-pages

Lantad requested the home page of the 130 SaaS hostnames in this repository's committed corpus on 22 September 2026, followed each site's own link to its pricing page and read the raw bytes with no JavaScript executed. 93 pricing pages answered with HTML. 77 of them showed a price in visible text and 17 declared one inside JSON-LD, and four of those 17 declared a price of zero on a page whose text listed paid plans.

/blog/can-ai-read-my-pricing-page-17-of-93.md from /blog/can-ai-read-my-pricing-page-17-of-93

Lantad requested /robots.txt from 40 well known hostnames on 5 September 2026, twice each, and evaluated GPTBot, OAI-SearchBot and ChatGPT-User against the site root with its shipped matcher. 29 served a parseable file. On 25 of them the three agents received the same verdict, and on 4 they did not.

/blog/can-chatgpt-read-my-website-agent-changed-the-answer.md from /blog/can-chatgpt-read-my-website-agent-changed-the-answer

Lantad asked 1,419 hostnames for /robots.txt three times on 18 September 2026, changing nothing between the requests but the user agent string. 1,056 returned a robots.txt to a desktop browser string. 92 of those returned a non-200 to OpenAI's example GPTBot string, and 63 of the 92 had served the same file to this scanner's own token seconds earlier.

/blog/can-gptbot-read-your-robots-txt-92-of-1056.md from /blog/can-gptbot-read-your-robots-txt-92-of-1056

Meta's updated Commercial Terms, read at facebook.com/legal/commercial_terms_preview on 18 October 2026, carry an Effective Date of October 30, 2026 and authorise Meta to use scrapers, bots, crawlers and user agents against the pages a business associates with its products or services. The document in force until then contains no such clause, and the string robots.txt appears in neither. Lantad read all 1,419 hostnames in this repository's two committed corpus seed files on 18 October 2026 with no JavaScript executed: 100 of the 1,080 readable home pages carried an executing Meta business integration, and Meta-ExternalAds was allowed at the site root on 999 of the 1,011 robots.txt files that parsed.

/blog/can-meta-crawl-my-website-2-of-1011-named-the-crawler.md from /blog/can-meta-crawl-my-website-2-of-1011-named-the-crawler

Lantad requested the robots.txt and then the home page of all 1,419 hostnames in this repository's two committed corpus seed files on 30 September 2026 as LantadBot, following redirects and executing no JavaScript. 1,069 answered HTTP 200 with an HTML content type. 906 of those declared at least one canonical link element and 163 declared none. Of the 904 that carried a usable href, 841 named exactly the address the page had been served at and 63 named something else, among them a Framer staging host, a different university subdomain and http://localhost:38131/.

/blog/canonical-tag-163-of-1069-none-and-63-named-another-url.md from /blog/canonical-tag-163-of-1069-none-and-63-named-another-url

Cloudflare turns a non-self-referencing canonical tag into a 301 for verified AI training crawlers. On five pages this scanner captured on 15 July 2026, all five canonical tags pointed at the page's own address, so the feature would have redirected none of them, and two of the five were not byte-identical to the URL the page was served at.

/blog/canonical-tags-and-ai-crawlers-five-of-five-self-referential.md from /blog/canonical-tags-and-ai-crawlers-five-of-five-self-referential

Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 17 October 2026, as LantadBot with no JavaScript executed. 1,013 parsed into at least one user-agent group. 140 named CCBot and 81 disallowed it at the root. 15 files that closed the root to GPTBot left CCBot allowed, and not one of those 15 named CCBot anywhere in the file.

/blog/ccbot-robots-txt-140-of-1013-named-it.md from /blog/ccbot-robots-txt-140-of-1013-named-it

Lantad requested the home page of 1,419 hostnames three times each on 26 September 2026, changing nothing between the requests but the user agent string. 1,089 answered a browser with HTTP 200. 103 of those served an unknown bot user agent as well and then refused the GPTBot string, and on 76 of the 103 the site's own robots.txt permits GPTBot at the root. Sorting the refusals by the infrastructure fingerprint on the response puts 32 of 87 Akamai fronted sites in the refused column against 37 of 344 on Cloudflare.

/blog/cdn-blocking-ai-crawlers-103-of-1089.md from /blog/cdn-blocking-ai-crawlers-103-of-1089

The HTML specification requires a character encoding declaration to sit entirely within the first 1024 bytes of a document, because that is all a reader is guaranteed to have examined before it must decide how to turn bytes into text. Lantad read the home page of all 1,419 hostnames in this repository's two committed corpus seed files on 7 October 2026 with no JavaScript executed. 1,088 answered HTTP 200 with HTML, 1,073 of them declared an encoding somewhere, and 1,071 of those declared UTF-8. The 15 that declared nothing were opened one by one, and all 15 turned out to be access denied interstitials, redirect stubs or maintenance notices served with a 200.

/blog/character-encoding-for-ai-search-15-of-1088-declared-none.md from /blog/character-encoding-for-ai-search-15-of-1088-declared-none

OpenAI's product specification, read on 2 September 2026, marks 16 fields Required. Twelve apply to every non-Ads feed, and four of those twelve are settings a merchant registers rather than facts a page publishes. The file is pushed to OpenAI over SFTP, so no crawler collects any of it.

/blog/chatgpt-product-feed-twelve-required-fields.md from /blog/chatgpt-product-feed-twelve-required-fields

The European Commission designated ChatGPT a Very Large Online Search Engine under the Digital Services Act on 31 August 2026. The obligations that follow are about systemic risk, audits and public reports, and the first of them lands four months after notification. None of them tells a site owner why one page was cited and another was not.

/blog/chatgpt-seo-eu-designated-a-search-engine.md from /blog/chatgpt-seo-eu-designated-a-search-engine

Fetched on 2 September 2026 across the 14 hostnames belonging to the six real sites this scanner captured on 15 July 2026, one request per hostname returned eight distinct files. On three of the six sites the second hostname did not serve the file that governs the captured page, and on one of those it returned nothing at all.

/blog/check-for-robots-txt-fourteen-hostnames-eight-files.md from /blog/check-for-robots-txt-fourteen-hostnames-eight-files

Chrome for Developers documents an agentic browsing category in Lighthouse whose Stability and Discoverability group checks for a machine-readable summary at the domain root. Google Search Central's guide to generative AI features, carrying Last updated 2026-07-10 UTC, states Google Search ignores such files. Both pages are Google's, and they describe different consumers.

/blog/lighthouse-agentic-browsing-checks-llms-txt.md from /blog/lighthouse-agentic-browsing-checks-llms-txt

Similarweb reports the presence of citations in US ChatGPT prompts rising from about 1.6 percent in June 2025 to roughly 6.8 percent by May 2026, and separately reports the homepage share of ChatGPT referrals jumping to about 60 percent in the week after 7 May 2026. The page that earns a citation and the page that receives the visit are frequently not the same page.

/blog/chatgpt-citations-are-rare-and-land-on-the-homepage.md from /blog/chatgpt-citations-are-rare-and-land-on-the-homepage

Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 18 October 2026, as LantadBot with no JavaScript executed. 1,014 parsed into at least one user-agent group. 159 named ClaudeBot and 75 disallowed it at the root. On 43 of those 159 the ClaudeBot line heads the same group as the wildcard, so it hands the crawler the rules the wildcard had already given it, and all 43 leave the root open.

/blog/claudebot-robots-txt-159-of-1014-named-it.md from /blog/claudebot-robots-txt-159-of-1014-named-it

Lantad loaded the home page of 392 hostnames twice on 20 September 2026, from the same browser at the same moment, changing one thing between the two loads: whether JavaScript was allowed to run. On the 271 pages that answered HTTP 200 both times, the unrendered document held 173,138 words and the rendered one held 187,422. The average is undramatic and the distribution is not: 100 pages were word for word identical, and 11 returned nothing readable at all until the bundle ran.

/blog/client-side-rendering-11-of-271-returned-no-words.md from /blog/client-side-rendering-11-of-271-returned-no-words

Cloudflare's verified bots policy names an AI crawler ignoring the robots.txt crawl-delay directive as an example of a policy breach, and its post of 7 August 2026 makes verified status revocable for conduct. RFC 9309 does not mention crawl-delay, and Google's own robots.txt specification says it is not supported.

/blog/crawl-delay-can-unverify-a-crawler.md from /blog/crawl-delay-can-unverify-a-crawler

Cloudflare announced on 1 July 2026 that it is shaping pay per crawl into pay per use, with experiments running alongside Ceramic.ai and You.com. A crawl arrives as a request in your own access log. A use happens inside an answer engine, where the site that produced the page has nothing to count.

/blog/cloudflare-pay-per-use-crawl-countable-use-not.md from /blog/cloudflare-pay-per-use-crawl-countable-use-not

Cloudflare opened a developer preview on 6 August 2026 that adds a WebMCP tool interface to any site it fronts, using HTMLRewriter to insert one script reference into each HTML response. Chrome's own WebMCP documentation, last updated 2026-06-09, files tool discoverability under its limitations: clients and browsers must visit a site directly to know if it has callable tools.

/blog/webmcp-tools-need-a-browser-to-be-found.md from /blog/webmcp-tools-need-a-browser-to-be-found

Cloudflare published Kitesurf on 6 August 2026, a browser built for AI agents that runs in V8 isolates on Workers rather than as a Chromium process. Its own documentation reports DOM subtests passing at 97 percent and lists a bot-challenge handshake with real TLS fingerprints among the things it cannot do.

/blog/cloudflare-agent-browser-is-not-chromium.md from /blog/cloudflare-agent-browser-is-not-chromium

Cloudflare announced Bot Preference Sync on 21 August 2026: a setting on every plan that writes your AI bot policy for Search, Agent and Training into robots.txt, prepends it above whatever file you already have, and refreshes the crawler names inside it from Cloudflare's own bot database. RFC 9309 says a crawler merges every group matching its token and reads the wildcard group only when none matches, so a prepended group changes which rules a named crawler obeys.

/blog/cloudflare-generates-robots-txt-and-the-token-list-moves.md from /blog/cloudflare-generates-robots-txt-and-the-token-list-moves

Cloudflare announced Agent Readiness diagnostics and an Answer Engine Optimization tab on 6 August 2026. The post sorts its site checks into quick wins, technical groundwork, advanced integration and commerce, and states that the commerce group is informational for now and not counted in your score.

/blog/agent-readiness-score-counts-three-of-four-groups.md from /blog/agent-readiness-score-counts-three-of-four-groups

Cloudflare's managed robots.txt writes one Content-signal line and eight Disallow rules into a site's file. We parsed the published example on 30 July 2026: the signal was recorded as an unknown field and ignored, and the Disallow rules blocked 8 of the 15 crawler tokens in our registry.

/blog/content-signal-asks-disallow-blocks.md from /blog/content-signal-asks-disallow-blocks

Common Crawl published CC-MAIN-2026-30 on 28 July 2026, 2.14 billion pages collected between 7 and 25 July. Its own FAQ states the crawler executes no JavaScript, so what a client rendered site contributed to that archive is whatever its server returned.

/blog/common-crawl-july-archive-no-javascript.md from /blog/common-crawl-july-archive-no-javascript

A paper from three Common Crawl Foundation authors, posted to arXiv on 15 July 2026, fits 51 monthly crawls from 2020 to 2025 and reports a persistent core fraction of about 0.4 at domain granularity, with the rest of the domain population churning through a shell.

/blog/common-crawl-core-and-shell-persistence.md from /blog/common-crawl-core-and-shell-persistence

Lantad requested the robots.txt and then the home page of all 1,419 hostnames in this repository's two committed corpus seed files on 30 September 2026 as LantadBot, following redirects and executing no JavaScript. 1,074 answered HTTP 200 with an HTML content type. 191 of those offered a visitor a clickable tel: link and 119 of the 191 carried no telephone property anywhere in their JSON-LD. 512 pages declared a node of an organization type and 251 of those attached no telephone, no email and no contactPoint to it.

/blog/contactpoint-schema-119-of-191-named-no-telephone.md from /blog/contactpoint-schema-119-of-191-named-no-telephone

Lantad requested the home page of 1,027 hostnames twice on 18 September 2026, once normally and once with an Accept header naming text/markdown. 718 answered the first request with HTML. 34 answered the second with a markdown content type, six of those returned HTML bytes under that label, and four refused with 406 Not Acceptable. None of the nine AI crawler vendors behind the 15 tokens this scanner evaluates documents its crawlers asking.

/blog/content-negotiation-for-markdown-34-of-718.md from /blog/content-negotiation-for-markdown-34-of-718

Cloudflare's changelog of 31 August 2026 records that the Browser Run crawl endpoint now respects the Content Signals use directive and rejects a job with HTTP 400 when the caller declares a more permissive level than the site does. We opened all ten Browser Run quick action documentation pages on 3 September 2026: nine of them name neither Content Signals nor robots.txt.

/blog/content-signals-robots-txt-one-of-ten-endpoints.md from /blog/content-signals-robots-txt-one-of-ten-endpoints

Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 11 October 2026, read every permitted home page with no JavaScript executed, then followed each page's own link to a courses, programmes, training or classes page and read that too. 242 of those pages answered HTTP 200 with HTML on both passes. One declared a schema.org Course node. 124 carried no JSON-LD block at all, and not one of the 242 used Course in microdata or RDFa.

/blog/course-schema-markup-1-of-242-course-pages.md from /blog/course-schema-markup-1-of-242-course-pages

Google's crawl budget guide, carrying Last updated 2026-07-22 UTC, tells site owners to block crawling of low value URLs with robots.txt. The guide is scoped to Googlebot and names no AI crawler. The file it tells you to edit is scoped to nobody in particular, and RFC 9309 says a crawler with no group of its own must obey the wildcard.

/blog/crawl-budget-advice-lands-on-ai-crawlers.md from /blog/crawl-budget-advice-lands-on-ai-crawlers

Crawl-delay appears in no version of the robots exclusion standard, and it is still the most common non-standard field in this corpus. Lantad requested /robots.txt once from each of 1,419 hostnames on 20 September 2026. Of the 1,014 files that parsed into at least one group, 157 carried a Crawl-delay line, and 11 of those 157 put one inside a group naming one of the 15 AI crawler tokens this scanner evaluates.

/blog/crawl-delay-robots-txt-157-of-1014-set-one.md from /blog/crawl-delay-robots-txt-157-of-1014-set-one

The crawl to refer ratio divides a platform's crawler requests for HTML by the HTML requests whose Referer header names that platform. Cloudflare, which publishes it, states that traffic from Claude's native app carries no Referer header and that the calculations may therefore overstate the ratios, by an amount it says is unclear.

/blog/crawl-to-refer-ratio-excludes-native-app-referrals.md from /blog/crawl-to-refer-ratio-excludes-native-app-referrals

Lantad requested robots.txt from all 1,419 hostnames in this repository's committed corpus on 3 October 2026 and parsed the 1,056 that returned a readable file. 152 carried at least one Crawl-delay directive and 324 directives appeared in total, but only 11 sites put one in a group naming an AI crawler token, and 237 of the 324 directives were aimed at SEO and backlink crawlers instead.

/blog/crawl-delay-robots-txt-152-of-1056-set-one.md from /blog/crawl-delay-robots-txt-152-of-1056-set-one

Google's Search Central changelog entry of 23 October 2026 records that it updated the URL structure best practices to clarify word separation guidelines using hyphens versus underscores. Lantad resolved every same-host link on the home page of 996 corpus sites on 25 October 2026 and found 2,203 of the 80,660 distinct URLs carried an underscore in the path, against 55,220 that carried a hyphen.

/blog/crawlable-url-structure-2203-of-80660-used-an-underscore.md from /blog/crawlable-url-structure-2203-of-80660-used-an-underscore

Thirteen published crawler IP range files from Google, OpenAI, Apple, Perplexity and Microsoft, read on 19 August 2026, each answer the question of when they last changed twice, and on six the two answers differ by between 13 and 942 days.

/blog/crawler-ip-files-carry-two-dates-that-disagree.md from /blog/crawler-ip-files-carry-two-dates-that-disagree

A user-agent line in robots.txt is an exact product token, and a token that matches nothing is a rule that does nothing. Lantad requested /robots.txt once from each of 1,419 hostnames on 17 September 2026. 1,004 files parsed into at least one group and named 2,209 distinct tokens between them, 1,098 of which appear in a single file, 813 of which contain a character RFC 9309 does not let a crawler put in its own name, and 120 of which name anthropic-ai or Claude-Web, neither of which appears on Anthropic's crawler page.

/blog/crawler-user-agent-2209-tokens-in-1004-files.md from /blog/crawler-user-agent-2209-tokens-in-1004-files

Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 19 October 2026, read the home page of every host its own file permitted, and then followed the same-host links whose path or anchor text named data, statistics or a data portal. 117 of those pages answered HTTP 200 with HTML. Not one declared a Dataset, DataCatalog, DataDownload or DataFeed node, and 81 of the 117 carried no JSON-LD typed node of any kind.

/blog/dataset-schema-0-of-117-data-pages-declared-it.md from /blog/dataset-schema-0-of-117-data-pages-declared-it

Google documents that the date in structured data must match the date a reader can see, and that the date is required while the time is not. Lantad read one interior page on each of 1,419 corpus hostnames on 4 October 2026 and reached 922 of them. 228 carried a node declaring datePublished or dateModified, 196 declared a non-empty dateModified, and on 38 of the 195 that parsed the page's own visible text did not contain so much as the year of the date it claimed. On the 81 pages where the comparison was possible the Last-Modified header disagreed with the schema value 74 times, and every one of the 74 ran the same way.

/blog/datemodified-schema-38-of-195-never-showed-the-date.md from /blog/datemodified-schema-38-of-195-never-showed-the-date

A source attribution study posted to arXiv on 7 May 2026 by researchers at PricewaterhouseCoopers benchmarked 14 language models on 130 research queries. Twelve of the 14 kept cited links working above 94 percent of the time, while factual accuracy against those same sources ran from 24.4 to 76.8 percent. The dimension a publisher can change was the one already near its ceiling.

/blog/deep-research-links-worked-facts-did-not.md from /blog/deep-research-links-worked-facts-did-not

All five real pages this scanner captured on 15 July 2026 carry exactly one main element. Rewriting every one of them as a plain div and re-running the shipped extractor on 26 August 2026 cut the main content corpus from 13,615 words to 11,950. On one page the landmark was worth 43 percent of the corpus and on another it was worth 1 percent.

/blog/deleting-main-cost-1665-of-13615-words.md from /blog/deleting-main-cost-1665-of-13615-words

Google's own verification page tells site owners that a crawler identifies itself three ways: the user agent header, the source IP, and the reverse DNS hostname of that IP. Lantad read every machine readable crawler address list nine AI crawler vendors publish on 12 October 2026, then asked a public resolver for a PTR record on 11,362 of the addresses in them. 8,510 answered and 8,468 of those forward confirmed. Not one of the 2,344 addresses sampled from OpenAI's four published files answered at all, so for GPTBot the IP list is the only published check a server can run.

/blog/detect-gptbot-0-of-2344-answered-reverse-dns.md from /blog/detect-gptbot-0-of-2344-answered-reverse-dns

Google documented the UGC Fresh Data Program on 8 October 2026 and named on-page SocialMediaPosting or DiscussionForumPosting markup with interactionStatistic sub-fields among its eligibility criteria. Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 9 October 2026 and read every permitted home page with no JavaScript executed. 1,090 answered HTTP 200 with HTML and 626 carried a JSON-LD block that parsed. None declared either type. Following each site's own link to a community area reached 88 readable pages, and none of those declared either type either.

/blog/discussion-forum-structured-data-zero-of-88.md from /blog/discussion-forum-structured-data-zero-of-88

Lantad requested the home page of all 1,419 hostnames in this repository's two committed corpus seed files twice on 6 October 2026 as LantadBot, then asked a third time with a conditional request carrying the validator the first response had supplied. 1,101 answered HTTP 200 with HTML. 391 of those sent neither an ETag nor a Last-Modified header, so a returning crawler had nothing to ask against, and 514 of the 710 that did offer one answered HTTP 304 Not Modified.

/blog/do-ai-crawlers-cache-your-pages-391-of-1101.md from /blog/do-ai-crawlers-cache-your-pages-391-of-1101

Lantad requested robots.txt and then the home page of all 1,419 hostnames in this repository's two committed corpus seed files on 3 October 2026 as LantadBot, followed each home page's own link to a blog, news or newsroom index, and read that index with no JavaScript executed. 466 index pages answered HTTP 200 with HTML. 116 of them declared a second page in markup a crawler can follow, and every one of those 116 URLs returned a working page. 24 others put the next tranche behind a load more control and declared no followable path to it at all.

/blog/do-ai-crawlers-follow-pagination-116-of-466.md from /blog/do-ai-crawlers-follow-pagination-116-of-466

Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 11 October 2026, then requested the home page of the 1,324 that permitted LantadBot with automatic redirect following switched off, and walked every Location header by hand. 878 of the 1,322 that answered were reached only through at least one HTTP redirect. 1,031 redirect responses carried five different status codes, 62 chains mixed a permanent code with a temporary one, and 14 of the readable home pages moved themselves again in the markup, four of those from inside a noscript element.

/blog/do-ai-crawlers-follow-redirects-878-of-1322.md from /blog/do-ai-crawlers-follow-redirects-878-of-1322

Lantad requested the robots.txt and then the home page of all 1,419 hostnames in this repository's committed corpus on 30 September 2026 and read the delivered bytes with no JavaScript executed and no stylesheet fetched. 1,115 pages were both allowed and readable, and they carried 1,269,054 words of body text. 121,312 of those words, 9.56 percent, sat inside an element marked hidden from somebody: from a sighted visitor, or from assistive technology, or from both. On 77 of the 820 pages that declared an h1, every h1 was inside one of those elements.

/blog/do-ai-crawlers-read-hidden-text-121312-of-1269054-words.md from /blog/do-ai-crawlers-read-hidden-text-121312-of-1269054-words

Lantad requested the home page of all 1,419 hostnames in this repository's committed corpus on 21 September 2026 and read the raw bytes with no JavaScript executed. 1,076 answered HTTP 200 with HTML and 433 of those carried an iframe, 834 elements in total. Every address they pointed at was then requested on its own: 354 of the 688 resolvable ones point at a tag manager, all 636 distinct iframe documents held 9,098 words between them, and two frames on two sites held 6,072 of those words.

/blog/do-ai-crawlers-read-iframes-17-of-834-held-50-words.md from /blog/do-ai-crawlers-read-iframes-17-of-834-held-50-words

Lantad asked all 1,419 hostnames in this repository's committed corpus for robots.txt on 23 September 2026 and read each home page with no JavaScript executed. 1,076 answered HTTP 200 with HTML and carried 52,077 image elements between them. 18,171 of those declared an empty alt, 1,935 carried no alt attribute at all, and on 73 pages not one image on the page described itself.

/blog/do-ai-crawlers-read-images-18171-of-52077-empty-alt.md from /blog/do-ai-crawlers-read-images-18171-of-52077-empty-alt

Lantad requested the home page of all 1,419 hostnames in this repository's committed corpus on 24 September 2026 and read the raw bytes with no JavaScript executed. 1,085 answered HTTP 200 with HTML, and 124 of those linked a document file. 332 of the 335 document links were PDFs, 317 of the fetched files really carried PDF bytes, and 297 of those 317 sent no X-Robots-Tag, which is the only control Google documents for a file that is not HTML.

/blog/do-ai-crawlers-read-pdfs-332-on-124-home-pages.md from /blog/do-ai-crawlers-read-pdfs-332-on-124-home-pages

Lantad requested the home page of 1,027 hostnames on 19 September 2026 and asked each of the 762 that answered for /llms.txt as well. 86 declared an RSS or Atom feed in a link element and 159 served an llms.txt, so the file no crawler vendor documents reading is on nearly twice as many of these sites as the format that has been standardised since 2005.

/blog/do-ai-crawlers-read-rss-86-of-762.md from /blog/do-ai-crawlers-read-rss-86-of-762

Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 6 October 2026 and read every permitted home page as LantadBot with no JavaScript executed. 1,074 answered HTTP 200 with HTML and carried 37,019 inline SVG elements between them. 80 of those elements carried a text, tspan or textPath node, the words inside them came to 1,285 against 1,355,047 words of ordinary page text, and 35,263 of the 37,019 carried no text of any kind.

/blog/do-ai-crawlers-read-svg-text-80-of-37019-carried-any.md from /blog/do-ai-crawlers-read-svg-text-80-of-37019-carried-any

Lantad's own component library says a table is the shape an AI crawler extracts most reliably, and a post in August checked that against five stored pages and found three tables. This run checked it against the whole corpus: 1,419 hostnames asked for robots.txt and then the home page as LantadBot on 23 September 2026, 1,079 pages read as raw bytes with no JavaScript, 38 of them carrying a table at all.

/blog/do-ai-crawlers-read-tables-87-on-1079-home-pages.md from /blog/do-ai-crawlers-read-tables-87-on-1079-home-pages

Lantad requested each of the 1,419 hostnames in this repository's two committed corpus seed files once on 13 October 2026 as LantadBot, with no JavaScript executed. 1,078 answered HTTP 200 with HTML and 246 of those put a video player, an audio player or a recognised third-party embed in front of the crawler. Ten home pages declared a caption track. Twenty nine of the 40 declared tracks named no file at all, six of the named files answered 404, and three home pages served a caption file that opened.

/blog/do-ai-crawlers-read-video-captions-3-of-1078.md from /blog/do-ai-crawlers-read-video-captions-3-of-1078

Lantad requested the home page of all 1,419 hostnames in this repository's two committed corpus seed files on 8 October 2026 as LantadBot, following redirects and executing no JavaScript. 1,077 answered HTTP 200 with HTML. 219 of those carried at least one custom element, 7,124 instances across 1,090 distinct tag names, and 3,291 of those instances arrived with no text inside them.

/blog/do-ai-crawlers-read-web-components-3291-of-7124.md from /blog/do-ai-crawlers-read-web-components-3291-of-7124

Lantad fetched the nine vendor documentation pages behind the fifteen AI crawler tokens it evaluates, on 5 September 2026, and read the prose out of each with its own extractor. Two of the nine state whether their crawler executes JavaScript. Five are silent. One is a client rendered shell that returns no prose at all without the JavaScript in question.

/blog/do-ai-crawlers-render-javascript-two-of-nine.md from /blog/do-ai-crawlers-render-javascript-two-of-nine

Lantad asked all 1,419 hostnames in this repository's committed corpus for robots.txt on 25 September 2026, then read each home page with no JavaScript executed. 1,069 answered HTTP 200 with HTML and carried 158,327 links with an http or https target. 1,176 of those links declared rel=nofollow, 499 of them pointing at the site's own host. Of the nine vendors behind the 15 AI crawler tokens this scanner evaluates, two document honouring the attribute.

/blog/do-ai-crawlers-respect-nofollow-1176-of-158327.md from /blog/do-ai-crawlers-respect-nofollow-1176-of-158327

A controlled study of ten consumer AI assistants, posted to arXiv on 16 July 2026, logged 200 retrieval trials against two purpose-built domains. Six of the ten assistants never requested robots.txt in any trial, and four of those six returned correct answers taken from pages the file disallowed.

/blog/do-ai-crawlers-respect-robots-txt-six-of-ten-never-asked.md from /blog/do-ai-crawlers-respect-robots-txt-six-of-ten-never-asked

Lantad requested the robots.txt and then the home page of all 1,419 hostnames in this repository's committed corpus on 27 September 2026 and read the raw bytes with no JavaScript executed. 1,070 home pages were readable and carried 51,990 image elements. 46,544 of those declared a fetchable URL in src and a further 267 in srcset, leaving 5,179 with neither. 3,155 of the 5,179 carried the real URL in a data attribute a crawler would have to run JavaScript to use.

/blog/do-ai-crawlers-see-lazy-loaded-images-3155-of-51990.md from /blog/do-ai-crawlers-see-lazy-loaded-images-3155-of-51990

Lantad requested the home page of all 1,419 hostnames in this repository's committed corpus on 21 September 2026 and read the raw bytes with no JavaScript executed. 1,093 answered HTTP 200 with HTML. 269 of them loaded one of 25 named consent platforms and 257 carried consent wording in the delivered HTML, but that wording came to 6,143 words against 1,381,167 visible words in total, and no page in the corpus gave a cookie notice more than half its text.

/blog/do-cookie-banners-block-ai-crawlers-269-of-1093.md from /blog/do-cookie-banners-block-ai-crawlers-269-of-1093

Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 16 October 2026, read every permitted home page with no JavaScript executed, then followed each page's own link to a documentation, developer, help or knowledge base surface and read that too. 239 of the 1,069 readable home pages linked one, 187 of those sat on a different hostname, and of the 220 pages that answered HTTP 200 with HTML, 43 delivered fewer than 100 words and 15 delivered none.

/blog/documentation-for-ai-search-43-of-220-under-100-words.md from /blog/documentation-for-ai-search-43-of-220-under-100-words

Cloudflare announced a Disallow AI Training setting and an Accountable designation on 15 September 2026, built on the promise that refusing training costs a site nothing in search. Lantad requested /robots.txt once from each of 1,027 hostnames on 16 September 2026. 700 files parsed into at least one group, 54 of them disallow Google-Extended at the site root, and on 52 of those 54 Googlebot is still allowed.

/blog/does-blocking-ai-training-affect-search-rankings-52-of-54.md from /blog/does-blocking-ai-training-affect-search-rankings-52-of-54

Lantad read the nine vendor documentation pages behind the fifteen AI crawler tokens it evaluates on 9 September 2026. The word noindex appears in the bytes of two of them, Apple's and Amazon's, and on both it governs indexing rather than the fetch.

/blog/does-noindex-block-ai-crawlers-two-of-nine.md from /blog/does-noindex-block-ai-crawlers-two-of-nine

Google's crawl budget documentation, last updated 22 July 2026, names Time to First Byte as an input to how much of a site it crawls. Lantad requested the home page of 392 hostnames on 14 September 2026. The median answered in 0.467 seconds and none took longer than two, but 184 of the 379 that returned HTML were reached through a redirect, and that redirect accounted for a median 50.8 percent of the wait.

/blog/does-page-speed-affect-ai-search-184-of-379-redirected.md from /blog/does-page-speed-affect-ai-search-184-of-379-redirected

Lantad requested the home page of 1,027 hostnames on 17 September 2026 and found 86 served by Drupal. All 86 robots.txt files answered HTTP 200, 68 of them carry the header comment Drupal core ships verbatim, and 79 name none of the 15 AI crawler tokens this scanner evaluates. Seventy three of the 86 are government, university or hospital sites.

/blog/drupal-ai-crawlers-79-of-86-named-none.md from /blog/drupal-ai-crawlers-79-of-86-named-none

DuckDuckGo documents that DuckAssistBot fetches pages in real time for answers that cite their sources, that the data never trains a model, and that disallowing it costs a site nothing in search. That makes it the cheapest citation crawler on the web to refuse, and the only lever is the product token in robots.txt. Lantad requested the robots.txt of all 1,419 hostnames in this repository's two committed corpus seed files on 7 October 2026. 1,063 returned a file that parsed as plain text, 63 of those named DuckAssistBot, and 48 of the 63 left the root open to it.

/blog/duckassistbot-63-of-1063-robots-txt-files-named-it.md from /blog/duckassistbot-63-of-1063-robots-txt-files-named-it

Lantad requested robots.txt from all 1,419 hostnames in this repository's two committed corpus seed files on 15 October 2026, then asked the 313 home pages that answer at their own apex for four more URL forms of the same page. 305 of the 313 served the same home page at two or more live URLs, 45 of those 305 declared no canonical at all, and the documentation of three AI crawler vendors never uses the word canonical.

/blog/duplicate-content-305-of-313-answered-at-more-than-one-url.md from /blog/duplicate-content-305-of-313-answered-at-more-than-one-url

draft-hood-aipref-earmark-00, an individual submission dated 12 August 2026, embeds a signed usage preference in the content itself so the preference survives copying, syndication and aggregation. Its own text states that a Mark does not prove copyright ownership, that Earmarks do not make preferences self-enforcing, and that Earmark prevents no use. It names the failure it is answering: a preference written at the perimeter attaches to the location rather than to the content.

/blog/earmark-signs-the-preference-into-the-content.md from /blog/earmark-signs-the-preference-into-the-content

A public URL box is a server-side request forgery machine unless something stops it. Lantad's guard names eight rejections, refuses 14 IPv4 ranges and applies eight IPv6 rules, re-runs in full on every redirect hop, and carries one gap we document rather than close.

/blog/eight-reasons-a-scanner-refuses-a-url.md from /blog/eight-reasons-a-scanner-refuses-a-url

Lantad asked all 1,419 hostnames in this repository's committed corpus for robots.txt on 8 October 2026 and read each home page it was allowed to read, with no JavaScript executed. 1,074 answered HTTP 200 with HTML and 612 of those carried parseable JSON-LD. 129 of the 612 used about, mainEntity or knowsAbout to name a subject, and of the 163 about and mainEntity values across 104 sites, 107 pointed at the site's own organisation, service or founder rather than at a topic.

/blog/entity-seo-107-of-163-topic-statements-named-the-site.md from /blog/entity-seo-107-of-163-topic-statements-named-the-site

Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 5 October 2026, read every permitted home page with no JavaScript executed, then followed each site's own link to an events page and read that too. 1,078 home pages and 519 events pages answered HTTP 200 with HTML. 19 hostnames carried an Event node anywhere, 442 nodes across 21 pages, and 171 of the 429 start dates that parsed as ISO 8601 were already in the past.

/blog/event-schema-markup-19-of-1419-sites-declared-it.md from /blog/event-schema-markup-19-of-1419-sites-declared-it

A study posted to arXiv on 29 June 2026 pointed six LLM web agents, three scrapers and three browser automation frameworks at ten honeysites carrying seven anti-bot defences. Combining network, TLS and browser fingerprints classified the client at 0.993 accuracy. The robots.txt honeysite blocked exactly one of the twelve tools, and it was the one whose framework obeys robots.txt by default.

/blog/web-agents-identifiable-robots-txt-stopped-one.md from /blog/web-agents-identifiable-robots-txt-stopped-one

Lantad asked all 1,419 hostnames in this repository's committed corpus for robots.txt on 23 September 2026, read each home page, then followed the site's own link to an FAQ, help or support page and read that too, with no JavaScript executed. 1,851 pages answered HTTP 200 with HTML. 83 sites declared a FAQPage, the 97 nodes between them held 1,277 questions, and 298 of the 1,274 answer texts that could be checked did not appear anywhere in the page the crawler was served.

/blog/faq-schema-298-of-1277-answers-not-on-the-page.md from /blog/faq-schema-298-of-1277-answers-not-on-the-page

We re-measured the five structured-data signals behind Entity Confidence against 30 pages fetched in one day, removed two of them, and added two nobody was checking.

/blog/five-signals-that-tell-ai-who-you-are.md from /blog/five-signals-that-tell-ai-who-you-are

A controlled study posted to arXiv on 25 May 2026 ran 252,000 trials across six language models, showing each of them two competing sources that differed in exactly one of 18 content factors. Four factors moved the first citation in every model, one of them was list position, and the two formatting factors are grouped by the paper among the seven with no consistent effect.

/blog/four-factors-decided-the-first-citation.md from /blog/four-factors-decided-the-first-citation

OpenAI publishes a JSON file of IP address ranges for each of its four crawlers. Read on 29 July 2026, the ChatGPT-User file covered 37,328 IPv4 addresses against GPTBot's 3,472, and it is the one crawler OpenAI says robots.txt rules may not apply to.

/blog/openai-crawler-ip-ranges-counted.md from /blog/openai-crawler-ip-ranges-counted

Google's crawling documentation sorts everything it sends at a website into three classes. Nine user-triggered fetchers generally ignore robots.txt, and five special-case crawlers ignore the global user agent group, so fourteen named Google agents are not decided by the group you head with an asterisk.

/blog/google-fetchers-ignore-the-robots-txt-wildcard.md from /blog/google-fetchers-ignore-the-robots-txt-wildcard

Counted with an HTML parser on 25 August 2026 over five real pages this scanner captured and stored on 15 July 2026, the initial HTTP responses hold 14 noscript elements. Between them they carry 11 img tags, one tag manager iframe, one CSS rule of 57 characters and zero characters of text. The one element HTML defines for a reader that does not run scripts is the one place on these pages where nobody wrote a sentence.

/blog/fourteen-noscript-elements-held-no-text.md from /blog/fourteen-noscript-elements-held-no-text

Lantad requested the home page of 392 hostnames on 11 September 2026 and found 36 served by Framer. Asked with an Accept header naming text/markdown, 35 of the 36 returned a markdown document. Asked for robots.txt, 32 of the 33 that answered HTTP 200 returned the same four line file, and one of the 33 named an AI crawler token at all.

/blog/framer-ai-crawlers-35-of-36-served-markdown.md from /blog/framer-ai-crawlers-35-of-36-served-markdown

Google's new channel files a visit under AI Assistant when the referrer matches a list it does not publish. Lantad fetched the six assistant hosts in its own referrer list on 9 September 2026: two returned a readable page, and both declare a policy that sends the host and discards the path.

/blog/ga4-ai-assistant-channel-two-of-six-hosts.md from /blog/ga4-ai-assistant-channel-two-of-six-hosts

Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for a home page carrying utm_source=gemini on 24 October 2026, as LantadBot with redirects followed and no JavaScript executed. 1,047 answered HTTP 200. Of the 749 that reached the page through at least one redirect, 171 arrived with both parameters gone, and every one of the 298 that made no redirect kept them.

/blog/gemini-utm-parameters-171-of-1047-dropped-in-a-redirect.md from /blog/gemini-utm-parameters-171-of-1047-dropped-in-a-redirect

A paper posted to arXiv on 31 March 2026 reports that editing document structure alone raised citation rate across six generative engines. Its own table puts the move at 45.0 percent to 52.8 percent, and the ablation attributes 84.6 percent of that gain to heading hierarchy and chunking rather than to emphasis.

/blog/generative-engine-optimization-structure-7-8-points.md from /blog/generative-engine-optimization-structure-7-8-points

A study published on 28 July 2026 by the AI visibility vendor Goodie, covering 31 million AI citations recorded between October 2025 and July 2026, puts the largest share of news citations on Grok at 23 percent, ahead of AI Overviews at 19.5 and Gemini at 13.5. Lantad's own answer engine registry, read on 7 September 2026, asks Grok with no retrieval and reads no citations back from it.

/blog/geo-engine-grok-23-percent-of-news-citations.md from /blog/geo-engine-grok-23-percent-of-news-citations

Google documents four preview controls that decide how much of a page it may use as direct input for AI Overviews and AI Mode, and one of them works on three element types only. Lantad requested robots.txt and then the home page of all 1,419 hostnames in this repository's two committed corpus seed files on 2 October 2026 as LantadBot, following redirects and executing no JavaScript. 1,079 answered HTTP 200 with an HTML content type. 54 of those carried a data-nosnippet attribute, 225 attributes between them, and 76 of the 225 sat on an element type the documentation does not list. On three pages every attribute on the page was one of them.

/blog/geo-optimization-76-of-225-data-nosnippet-attributes-inert.md from /blog/geo-optimization-76-of-225-data-nosnippet-attributes-inert

A paper posted to arXiv on 11 August 2026 by researchers at Carnegie Mellon and UC San Diego ran an automated generative engine optimization tool against a defended answer engine for five consecutive rounds, on three benchmarks and three model backends. In the worked example the rewritten document accumulated four to six unsupported claims, and the strategy the platform read back out of the rewrites moved from formatting advice at round one to hidden GEO intent at round five.

/blog/geo-rewriting-degraded-the-document-over-five-rounds.md from /blog/geo-rewriting-degraded-the-document-over-five-rounds

Both disciplines depend on one HTTP fetch governed by one file. The measured divergences are narrow: a Washington University team found nearly 30 percent of AI Overview cited domains rank nowhere on page one, a 252,000 trial experiment put formatting near zero, and Google's own guide says SEO best practices still apply because its generative features run on its core ranking systems.

/blog/geo-vs-seo-same-crawl-four-differences.md from /blog/geo-vs-seo-same-crawl-four-differences

Google's robots meta tag documentation states that nosnippet prevents content being used as a direct input for AI Overviews and AI Mode, and that max-snippet limits how much may be used. Lantad read the home page of all 1,419 hostnames in this repository's two committed corpus seed files on 7 October 2026 with no JavaScript executed. 1,083 answered HTTP 200 with HTML. 132 sent one of those two directives. 129 of them set max-snippet to -1, which Google defines as no limit at all.

/blog/google-ai-mode-132-of-1083-sent-the-snippet-directive.md from /blog/google-ai-mode-132-of-1083-sent-the-snippet-directive

Google's guidance on Search-related JavaScript problems, carrying Last updated 2025-12-18 UTC, states under the heading Don't rely on data persistence to serve content that its Web Rendering Service does not retain state across page loads, and that Local Storage, Session Storage and HTTP Cookies are cleared across page loads. A gate that waits for a stored consent decision is therefore waiting on something the crawler discards, and the crawler pages published by OpenAI, Anthropic and Perplexity, read on 13 August 2026, contain the word cookie zero times between them.

/blog/consent-gates-never-open-for-a-stateless-crawler.md from /blog/consent-gates-never-open-for-a-stateless-crawler

Google's snippet documentation, carrying Last updated 2026-04-20 UTC, describes a read more deep link that lands a searcher on a specific section of a page, and names removing the hash fragment from the URL as something that breaks it. Counted with an HTML parser over five real pages this scanner captured on 15 July 2026, 55 of 245 headings carried an id of their own, and 26 headings could not be reached by any element identifier at all.

/blog/read-more-deep-links-and-heading-ids.md from /blog/read-more-deep-links-and-heading-ids

Google's documentation changelog records the FAQ rich result leaving Search on 7 May 2026 and its documentation being removed on 15 June 2026. Read on 5 August 2026 the old URL answers 301, schema.org still defines FAQPage, and Lantad's schema check still counts it as a recognised business type.

/blog/faq-rich-result-gone-faqpage-still-scored.md from /blog/faq-rich-result-gone-faqpage-still-scored

Google's lazy-loading documentation, carrying Last updated 2025-12-10 UTC, tells site owners to load content when it enters the viewport rather than when somebody scrolls, and gives the reason in one clause: Google Search does not interact with your page. The crawler pages published by OpenAI, Anthropic and Perplexity, fetched and searched on 14 August 2026, contain the words JavaScript, render, scroll, viewport and lazy zero times between them.

/blog/lazy-loading-and-crawlers-that-do-not-scroll.md from /blog/lazy-loading-and-crawlers-that-do-not-scroll

Google's file types documentation, carrying Last updated 2026-02-03 UTC, lists nine flat and eleven encoded formats it can index and states that the Content-Type header decides which parser runs. The crawler documentation published by OpenAI, Anthropic and Perplexity, read on 19 August 2026, names no file type at all, and this scanner returns no score for a URL that is not HTML.

/blog/google-indexes-twenty-file-types-ai-docs-name-none.md from /blog/google-indexes-twenty-file-types-ai-docs-name-none

Google added a guide on 16 September 2026 describing how to put a Search profile badge on a website, and it asks for a downloaded image and a link rather than any markup. Lantad captured the home pages of all 1,419 hostnames in this repository's committed corpus on 3 October 2026 and read the 1,093 that returned HTML. Two referenced profile.google.com, both inside a sameAs array on an organisation node, both using a URL form the guide does not document, and neither carried the text link the guide suggests.

/blog/google-search-profile-badge-2-of-1093-referenced-it.md from /blog/google-search-profile-badge-2-of-1093-referenced-it

Google Search Central updated its preferred sources documentation on 20 August 2026 to add an interactive button that loads a script from news.google.com, and named a plain URL alternative for sites that cannot run third party scripts.

/blog/preferred-sources-button-and-plain-url.md from /blog/preferred-sources-button-and-plain-url

Google documents three Search features for the European Economic Area, and they are opened by three different things: an approved data feed, an interest form, and nothing but an ordinary crawl. Lantad measured the one site-side condition the Google supplier unit names across all 1,419 hostnames in this repository's two committed corpus seed files on 16 October 2026. 347 delivered no HTML at all and 71 of the 1,072 that did delivered zero words, so 418 gave a plainly identified crawler nothing to read.

/blog/google-supplier-unit-418-of-1419-gave-a-crawler-no-words.md from /blog/google-supplier-unit-418-of-1419-gave-a-crawler-no-words

Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 23 October 2026, as LantadBot with redirects followed and no JavaScript executed, and parsed the 1,023 that resolved into at least one user-agent group. 26 named Google-CloudVertexBot, the crawler Google documents as serving crawls requested by site owners for building Vertex AI Agents, and 23 of the 26 disallowed it at the root. Every one of the 26 also named Google-Extended, and no file named the Vertex token without the training token.

/blog/google-cloudvertexbot-robots-txt-26-of-1023-named-it.md from /blog/google-cloudvertexbot-robots-txt-26-of-1023-named-it

Google Search Central publishes an official guide to appearing in AI Overviews and AI Mode, and a section of it headed Mythbusting lists llms.txt, chunking, rewriting for AI, inauthentic mentions and structured data among the things you do not need to do.

/blog/google-names-five-geo-tactics-to-ignore.md from /blog/google-names-five-geo-tactics-to-ignore

Google's paywalled content documentation, last updated 10 December 2025, tells a publisher to let Googlebot and Googlebot-News reach the paywalled text so the markup can show the gap is not cloaking. Read on 4 August 2026, those two names are the only crawler names on the page.

/blog/paywall-markup-names-two-google-crawlers.md from /blog/paywall-markup-names-two-google-crawlers

Google's review snippet documentation, carrying Last updated 2026-07-24 UTC, tells site owners not to include fake or undisclosed incentivized reviews in the page or in the structured data markup. Of the nine properties schema.org defines specifically on Review, none records whether the reviewer was paid. The rule is about provenance and every automated check reads shape.

/blog/review-authenticity-is-not-in-the-markup.md from /blog/review-authenticity-is-not-in-the-markup

Google's crawler documentation, read on 10 August 2026, states that Googlebot crawls the first 2MB of a supported file type, stops the fetch at the cutoff, and applies the limit to uncompressed data. The crawler documentation published by OpenAI, Anthropic and Perplexity states no size limit at all, which leaves site owners with a documented ceiling from one vendor and silence from three.

/blog/googlebot-reads-the-first-two-megabytes.md from /blog/googlebot-reads-the-first-two-megabytes

Google's locale-adaptive pages documentation, carrying Last updated 2025-12-10 UTC, states that the crawler sends HTTP requests without setting Accept-Language and that its default IP addresses appear to be based in the USA. Read on 13 August 2026, the crawler pages published by OpenAI, Anthropic and Perplexity name no language, locale or region behaviour at all, so a site that chooses a language from the request has nothing published to plan against.

/blog/googlebot-sets-no-accept-language.md from /blog/googlebot-sets-no-accept-language

Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 17 October 2026, as LantadBot with no JavaScript executed, and parsed the 1,008 that resolved into at least one user-agent group. 81 allowed Googlebot at the site root and refused GPTBot, and none did the reverse. Beyond the root, 639 paths across 40 hostnames were open to GPTBot and closed to Googlebot, and every divergence in the corpus came from a file that names one of the two tokens.

/blog/googlebot-vs-gptbot-81-of-1008-files-differed.md from /blog/googlebot-vs-gptbot-81-of-1008-files-differed

Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 21 October 2026, as LantadBot with no JavaScript executed. 1,015 parsed into at least one user-agent group. 63 named GoogleOther, the crawler Google documents as generic and available to various product teams, and 27 of the 1,015 disallowed it at the root, against 68 that disallowed Google-Extended. 43 files closed the root to Google-Extended and left GoogleOther open.

/blog/googleother-robots-txt-63-of-1015-named-it.md from /blog/googleother-robots-txt-63-of-1015-named-it

A robots.txt product token and an HTTP User-Agent header are two different strings, and only the second one is what your access log holds. Read at nine vendors' own documentation on 27 August 2026, 6 of the 15 tokens in Lantad's crawler registry carry a complete User-Agent string that can be matched literally.

/blog/gptbot-detection-six-of-fifteen-tokens-publish-a-ua.md from /blog/gptbot-detection-six-of-fifteen-tokens-publish-a-ua

The 2025 Web Almanac, written from HTTP Archive's July 2025 crawl of 16,213,084 websites and published on 15 January 2026, reports the catch-all user agent in 77.04 percent of desktop robots.txt files and gptbot in 4.5 percent. Google's advertising crawler is named more than twice as often as any AI crawler.

/blog/robots-txt-names-the-wildcard-not-the-ai-crawler.md from /blog/robots-txt-names-the-wildcard-not-the-ai-crawler

Lantad read every page of xAI's published documentation on 18 September 2026, 182 pages and 200,241 words, and the strings robots, crawler, crawl and user agent appear in it zero times each. In the same day's sweep of 1,419 hostnames, ten robots.txt files tried to write a rule for an xAI crawler anyway, between them spelling its name eight different ways.

/blog/grok-seo-xai-documents-no-crawler.md from /blog/grok-seo-xai-documents-no-crawler

Lantad asked all 1,419 hostnames in this repository's committed corpus for robots.txt on 25 September 2026, read each home page with no JavaScript executed, followed one article shaped link from it, and read that page the same way. 706 article pages answered, of which 704 arrived whole. 311 of the 704 carried at least one unbroken stretch of more than 300 words with no heading anywhere in it, 237 skipped a heading level, and not one of the 704 carried a single text block longer than 1,200 words.

/blog/heading-structure-for-ai-search-311-of-704.md from /blog/heading-structure-for-ai-search-311-of-704

Three researchers at the University of Bamberg loaded 10,000 homepages in four browser configurations and recorded who got refused. Headless Chromium drew a 15.2 percent soft block rate against 7.2 percent for the same browser with a window, and removing one string from two headers unblocked 75 percent of the difference.

/blog/headless-chromium-blocked-on-15-percent-of-sites.md from /blog/headless-chromium-blocked-on-15-percent-of-sites

Lantad asked the 123 healthcare hostnames in this repository's committed industry seed file for robots.txt on 19 October 2026 and read every permitted home page with no JavaScript executed. 91 answered HTTP 200 with HTML. 43 of those carried parseable JSON-LD, 12 named a type from schema.org's medical vocabulary, and the 95 medical nodes between them declared none of the six properties those types define: not medicalSpecialty, not isAcceptingNewPatients, not healthPlanNetworkId, not availableService, not healthcareReportingData, not medicalAudience.

/blog/healthcare-schema-markup-95-nodes-no-specialty.md from /blog/healthcare-schema-markup-95-nodes-no-specialty

A controlled study posted to arXiv on 15 June 2026 ran five attack modes against 13 LLM backends, 308 cases each. Hidden instructions planted in the machine-readable page averaged 0.3 percent success. Three fabricated sources agreeing with each other averaged 24.8 percent.

/blog/hidden-text-was-the-weakest-attack-on-ai-search.md from /blog/hidden-text-was-the-weakest-attack-on-ai-search

Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 21 October 2026, read the home page of every host its own file permitted, and then followed the same-host links whose path named a hotel, a room, a destination or a tour. One of the 1,093 readable home pages declared a Hotel node. Across those home pages and the 134 travel pages reached from them, 111 Hotel nodes arrived from four hosts, and not one of them declared checkinTime, numberOfRooms or amenityFeature.

/blog/hotel-schema-markup-1-of-1093-home-pages.md from /blog/hotel-schema-markup-1-of-1093-home-pages

Perplexity documents two named crawlers with different robots.txt behaviour, and its Sonar Chat Completions endpoint is deprecated with support ending on 27 September 2026 in favour of the Agent API.

/blog/how-do-i-get-cited-by-perplexity-two-crawlers.md from /blog/how-do-i-get-cited-by-perplexity-two-crawlers

Cloudflare and OpenAI published early results on 15 October 2026 from a research pilot begun in July 2026, in which signals from Cloudflare's network helped OpenAI's search crawler reach pages that links never did. The post states the pilot ran on a participating cohort of about 1 percent of Cloudflare's total network and enabled OpenAI to discover and index hundreds of millions of new web pages. It also states that a signal about a page does not grant permission to access it, and lists measuring what happens after content is discovered as next-phase work. Discovery moved. Permission and readability did not.

/blog/how-does-chatgpt-find-my-website-a-1-percent-cohort.md from /blog/how-does-chatgpt-find-my-website-a-1-percent-cohort

A study of 374,052 citations across Perplexity, OpenAI SearchGPT and Google Gemini found repeated runs of the same query share as little as 0.29 of their cited domains, and that a confidence interval five percentage points wide on citation share needs roughly 40 queries on one platform and 150 or more on another.

/blog/how-many-prompts-an-ai-visibility-measurement-needs.md from /blog/how-many-prompts-an-ai-visibility-measurement-needs

Lantad requested the robots.txt and then the home page of all 1,419 hostnames in this repository's committed corpus on 27 September 2026 and read the raw bytes with no JavaScript executed. 1,071 home pages were both allowed and readable, and they carried 20,664 list elements holding 104,474 list items. 80,761 of those items, 77.30 percent, sat inside a nav, header, footer or aside element, or inside an element declaring a navigation, banner or contentinfo role. 428 of the 1,071 pages carried no list anywhere else.

/blog/how-to-format-content-for-ai-search-80761-of-104474.md from /blog/how-to-format-content-for-ai-search-80761-of-104474

Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt and then for the home page on 26 October 2026, every request sent as LantadBot/1.0, and evaluated OpenAI's three robots.txt tokens against the site root with the matcher in core/src/robots.ts. 980 sites cleared all four conditions OpenAI's own crawler documentation implies. 25 were closed by a robots.txt rule. 217 permitted the crawler in the file and then answered the home page with HTTP 403.

/blog/how-to-get-cited-by-chatgpt-980-of-1419-cleared-every-gate.md from /blog/how-to-get-cited-by-chatgpt-980-of-1419-cleared-every-gate

Anthropic's help centre article of 7 April 2026 names ClaudeBot, Claude-User and Claude-SearchBot, and says that disabling either of the last two may reduce your site's visibility. Lantad fetched the IP range file that page points at on 7 September 2026. It holds 26 prefixes covering 1,094 IPv4 addresses, and it is a single list for all three bots.

/blog/how-to-get-cited-by-claude-one-ip-list-three-bots.md from /blog/how-to-get-cited-by-claude-one-ip-list-three-bots

Google documents 11 common crawlers and none of them is a Gemini crawler. The only entry that names Gemini is Google-Extended, which sends no request of its own, and Google's page of 14 July 2026 says the same token governs training and grounding together.

/blog/how-to-get-cited-by-gemini-one-token-two-systems.md from /blog/how-to-get-cited-by-gemini-one-token-two-systems

Google publishes one hard eligibility rule for a supporting link in AI Overviews, which is that the page must be indexed and eligible to be shown in Search with a snippet. Lantad requested the home page of 120 real hostnames on 6 September 2026. Of the 81 that answered HTTP 200, none carried a nosnippet directive, three carried max-snippet and all three set it to -1, and 48 carried no robots meta tag at all.

/blog/how-to-get-content-cited-in-google-ai-overviews-zero-nosnippet.md from /blog/how-to-get-content-cited-in-google-ai-overviews-zero-nosnippet

A paper posted to arXiv on 24 July 2026 measured where a conversation's request state actually sits across its user turns. In 670 commercial multi-turn conversations the final prompt carried a median 35.6 percent of the session's unique user-side content vocabulary, and in 50.3 percent of them a rule set detected at least one request-state dimension in the history that the final prompt never repeats.

/blog/how-to-measure-geo-final-prompt-held-35-6-percent.md from /blog/how-to-measure-geo-final-prompt-held-35-6-percent

On 15 October 2026 Google said it is adding more inline links to AI Overviews and AI Mode, showing them higher, and testing a top-of-answer section for official pages. Being the page an answer links is now a contest decided before the organic position you already fight for.

/blog/how-to-rank-in-ai-overviews-google-adds-inline-links.md from /blog/how-to-rank-in-ai-overviews-google-adds-inline-links

Lantad read all 1,419 hostnames in this repository's committed corpus on 11 October 2026, took every same-host link whose URL path or anchor text explicitly said it was instructional, and fetched the 288 pages that produced. 8 of the 272 that answered with HTML declared a HowTo or Recipe node. 115 of the 272 laid their steps out as an ordered list or a numbered heading, and 109 of those 115 declared no HowTo node at all. Google deleted its HowTo documentation on 14 September 2023 and now states that no special structured data is needed for its AI features.

/blog/howto-schema-8-of-272-how-to-pages.md from /blog/howto-schema-8-of-272-how-to-pages

Lantad requested the home page of 1,027 hostnames on 15 September 2026 and read the bytes with no JavaScript executed. 730 answered with a 2xx and an HTML content type, 255 of those declared a hreflang alternate, and asking the same 730 again with a German Accept-Language header changed the declared language on 27 of them. Four of the 27 publish no alternate at all.

/blog/hreflang-and-ai-crawlers-27-of-719-changed-on-a-header.md from /blog/hreflang-and-ai-crawlers-27-of-719-changed-on-a-header

Lantad asked all 1,419 hostnames in this repository's committed corpus for robots.txt on 8 October 2026 and read each home page it was allowed to read, with no JavaScript executed. 1,084 answered HTTP 200 with HTML, 618 of those carried parseable JSON-LD, and 253 declared ImageObject schema across 943 nodes. 685 of the 943 carried only an image URL and pixel dimensions, 128 carried a caption, and 33 nodes on 2 sites carried the pair of properties Google's image metadata documentation requires.

/blog/imageobject-schema-685-of-943-nodes-url-and-size.md from /blog/imageobject-schema-685-of-943-nodes-url-and-size

IndexNow lets a site push changed URLs to search engines instead of waiting to be crawled. Read at indexnow.org on 12 August 2026, its supporter list names Microsoft Bing, Naver, Seznam.cz, Yandex and Yep, and its own documentation states that an HTTP 200 indicates only that the search engine has received your URL. The protocol moves the moment a crawler learns a page changed. It changes nothing about what that crawler receives when it arrives.

/blog/indexnow-speeds-discovery-not-readability.md from /blog/indexnow-speeds-discovery-not-readability

Lantad asked all 1,419 hostnames in this repository's committed corpus for robots.txt on 26 September 2026, then read the home page of every host that allowed it with no JavaScript executed. 1,077 answered HTTP 200 with HTML. 615 of them carried at least one JSON-LD block that parsed, and 186 of those 615 put an inLanguage value anywhere in it. The html lang attribute was on 1,016 of the 1,077, and Google's own documentation says it reads none of these tags.

/blog/inlanguage-schema-429-of-615-named-no-language.md from /blog/inlanguage-schema-429-of-615-named-no-language

Lantad requested the home page of all 1,419 hostnames in this repository's committed corpus on 21 September 2026 and read the raw bytes with no JavaScript executed. 1,091 answered HTTP 200 with HTML. 84 of those carried no internal link a crawler could follow, and 55 of the 84 contained no anchor element at all. On the 1,007 that did offer a path, the first one was requested too.

/blog/internal-links-ai-crawlers-84-of-1091-no-path.md from /blog/internal-links-ai-crawlers-84-of-1091-no-path

Lantad requested the home page of 1,015 hostnames five times each on 13 September 2026, once as LantadBot and once as each of GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. On 891 hostnames all five requests came back. 700 of those answered the baseline with a 2xx, and 115 of the 700 refused at least one crawler user agent. Re-reading each of those 115 robots.txt files the same day, 79 of them allow at the site root a crawler their server turned away.

/blog/is-my-site-blocking-ai-crawlers-79-of-115.md from /blog/is-my-site-blocking-ai-crawlers-79-of-115

Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 14 October 2026. 1,052 returned a text file, carrying 88,364 directive lines and 74,626 allow and disallow rules. 84 user-agent lines on 58 files spelled a documented AI crawler token in the wrong case and this scanner's evaluator bound every one of them. 15,864 rule paths on 418 files carried a capital letter, and on the half of the file where case does decide the answer, nine pages across five hosts answered a crawler at the lowercase address their own Disallow line failed to cover.

/blog/is-robots-txt-case-sensitive-84-tokens-bound-9-pages-open.md from /blog/is-robots-txt-case-sensitive-84-tokens-bound-9-pages-open

The European Data Protection Board adopted draft Guidelines 03/2026 on web scraping in the context of generative AI on 7 July 2026 and has them out for public consultation until 30 October 2026. The draft never makes robots.txt binding. It names robots.txt, ai.txt and CAPTCHA as technical measures a scraper should check, and makes their presence a factor in whether a person can reasonably expect their data to be scraped. Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for both ai.txt paths and for robots.txt on 20 October 2026. Five served a plain text file at an ai.txt path that was not a catch-all, four of those five declared an AI usage preference, and one of the four grants training outright.

/blog/is-robots-txt-legally-binding-4-of-1419-served-ai-txt.md from /blog/is-robots-txt-legally-binding-4-of-1419-served-ai-txt

An answer engine asked for the best five of something has to find five things, and ItemList is the one schema.org construct built to say a page holds a list of named things in a stated order. Lantad requested the home page of all 1,419 hostnames in this repository's committed corpus on 1 October 2026 and parsed every JSON-LD block. 605 pages carried markup that parsed. 17 of those declared an ItemList, 58 nodes between them, and 525 of the 766 items inside those nodes carried no name at all.

/blog/itemlist-schema-17-of-605-and-525-items-unnamed.md from /blog/itemlist-schema-17-of-605-and-525-items-unnamed

Counted with an HTML parser over five real pages this scanner captured and stored on 15 July 2026, the raw HTTP response held 991 distinct same-host URLs and the rendered DOM held 982. Three URLs existed only after JavaScript ran, and every one of the three was a rewrite of a link already present. Nine of the twelve that disappeared were one page's translations of itself.

/blog/javascript-added-no-new-crawl-paths.md from /blog/javascript-added-no-new-crawl-paths

Lantad requested robots.txt from all 1,419 hostnames in this repository's committed corpus on 29 September 2026, then read every permitted home page twice: once as raw HTML with no JavaScript executed, and once out of the DOM of a real browser sent the same crawler user agent. 969 hosts produced a clean pair. 404 of those carried no JSON-LD block in the HTML, and 27 of the 404 carried one after the browser had run the page.

/blog/javascript-structured-data-27-of-404-gained-it.md from /blog/javascript-structured-data-27-of-404-gained-it

Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 10 October 2026, read every permitted home page with no JavaScript executed, then followed each page's own link to a careers or jobs page and read that too. 380 of the 1,079 readable home pages linked one, and 374 answered HTTP 200 with HTML on two passes the same day. 5 of the 374 declared a JobPosting node, carrying 52 postings between them.

/blog/job-posting-schema-5-of-374-careers-pages.md from /blog/job-posting-schema-5-of-374-careers-pages

Lantad requested the home page of 392 hostnames on 14 September 2026 and read the bytes with no JavaScript executed. 385 answered with a 2xx and an HTML content type. 239 carried JSON-LD, 22 carried microdata, one carried RDFa, and 19 of the 22 microdata pages carried JSON-LD as well. Reading all three syntaxes instead of one rescued four pages out of the 146 that published no JSON-LD.

/blog/json-ld-vs-microdata-22-of-385-carried-microdata.md from /blog/json-ld-vs-microdata-22-of-385-carried-microdata

Cloudflare's rebuilt bot submission form, launched on 28 August 2026, asks an operator to declare what its bot does, how it uses what it reads, and who runs it. The automated review that follows fetches an IP list, confirms reverse DNS or validates a Web Bot Auth signature, so the one claim a machine settles is identity, and the three that describe intent are taken on the operator's word.

/blog/llm-crawler-detection-declared-three-verified-one.md from /blog/llm-crawler-detection-declared-three-verified-one

Lantad requested /llms.txt and /llms-full.txt once each from 54 documentation and developer tool hostnames on 12 September 2026. 44 returned an llms.txt as plain text or markdown, 32 of those also returned an llms-full.txt, and not one returned the second file without the first. The llms.txt proposal, modified on 10 August 2026, does not contain the string llms-full.txt anywhere.

/blog/llms-full-txt-32-of-44-served-one.md from /blog/llms-full-txt-32-of-44-served-one

Lantad requested /llms.txt once from each of 40 hostnames on 5 September 2026, chosen because they were the sites most likely to publish one. 31 returned a plain text file, 7 returned HTTP 404, and 2 returned HTTP 200 with an HTML document. The 31 files hold 5,567 markdown links between them, and every one of the 75 links in the file at docs.mistral.ai returned HTTP 404.

/blog/llms-txt-example-31-files-and-75-dead-links.md from /blog/llms-txt-example-31-files-and-75-dead-links

The llms.txt proposal was revised to version 2 and the page now carries a modified date of 10 August 2026. The change its author names first is discoverability: two standard link relations, carried in an HTML link element or an HTTP Link header, so an agent can find a page's markdown version and the llms.txt covering it without guessing. The context expansion tool that version 1 described as the consumer of these files is no longer part of the proposal.

/blog/llms-txt-v2-link-relations-and-discovery.md from /blog/llms-txt-v2-link-relations-and-discovery

Lantad requested /robots.txt and /llms.txt once each from 200 hostnames on 7 September 2026. 145 returned a parseable robots.txt and 23 returned an llms.txt as plain text. Of those 23, 19 name none of the 15 AI crawler tokens this scanner evaluates anywhere in their robots.txt.

/blog/llms-txt-vs-robots-txt-19-of-23.md from /blog/llms-txt-vs-robots-txt-19-of-23

Lantad requested /llms.txt from all 1,419 hostnames in this repository's two committed corpus seed files on 29 September 2026 as LantadBot, following redirects and executing no JavaScript. 305 answered HTTP 200 with a text body that was not HTML, and a control path that cannot exist returned text on none of them. 298 were real files. On the 239 of those whose sitemap resolved completely, the llms.txt named a median of 23 URLs and the sitemap declared a median of 969.

/blog/llms-txt-vs-sitemap-xml-23-urls-against-969.md from /blog/llms-txt-vs-sitemap-xml-23-urls-against-969

Lantad requested the home page of 253 healthcare and travel hostnames on 14 September 2026 and read the bytes with no JavaScript executed. 172 answered with a 2xx and an HTML content type. 25 carried a street address inside JSON-LD, 31 carried a telephone number, and 2 carried opening hours.

/blog/localbusiness-schema-two-of-172-carried-hours.md from /blog/localbusiness-schema-two-of-172-carried-hours

The Model Context Protocol revision of 28 July 2026 requires every HTTP POST to an MCP endpoint to name its protocol version, its method and the resource it is reading. Those three headers describe one hop, and the site whose content is being read is not on it.

/blog/mcp-labels-stop-at-the-mcp-server.md from /blog/mcp-labels-stop-at-the-mcp-server

Lantad requested the robots.txt of all 1,419 hostnames in this repository's corpus seeds on 24 September 2026 as LantadBot, and 1,065 returned a plain-text file. 117 named Meta-ExternalAgent, the training crawler. 6 named Meta-WebIndexer, the crawler Meta's own documentation says has to be allowed for Meta AI to cite and link to a site. All six of those refused it.

/blog/meta-ai-crawler-6-of-1065-named-the-citation-token.md from /blog/meta-ai-crawler-6-of-1065-named-the-citation-token

Lantad asked the 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 19 October 2026 and read every permitted home page with no JavaScript executed. 1,068 answered HTTP 200 with HTML. 930 of those carried both a title and a meta description with something in it, 118 carried a title and no usable description, and 20 carried neither.

/blog/meta-description-for-ai-search-138-of-1068-served-none.md from /blog/meta-description-for-ai-search-138-of-1068-served-none

Meta's web crawler documentation, read on 8 August 2026, names facebookexternalhit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher. Lantad's crawler registry carries one Meta token, and a post we published on 1 August 2026 stated that Meta has no second published name to allow separately. The name most worth having is Meta-WebIndexer, which Meta describes as serving Meta AI search.

/blog/meta-documents-five-crawler-tokens.md from /blog/meta-documents-five-crawler-tokens

Lantad requested the robots.txt and then the home page of all 1,419 hostnames in this repository's committed corpus on 25 September 2026 and read the head of every response with no JavaScript executed. 1,083 answered HTTP 200 with HTML. 38 of those carried neither a usable title element nor an og:title, so nothing in the bytes named the page, and a second independent request the same day returned the same result on all 38.

/blog/meta-tags-for-ai-search-38-of-1083-sent-no-name.md from /blog/meta-tags-for-ai-search-38-of-1083-sent-no-name

DataDome's press release of 16 July 2026 states that its network processed 17.7 billion AI agent requests between April and June, up from 12.2 billion in the previous quarter, with Meta's two agents taking the majority of that traffic. In the same release, ChatGPT-User's absolute request volume fell 6 percent while ChatGPT held 80 to 88 percent of all AI referrals every month. Crawl volume and referral value are not the same population, and the tokens are not interchangeable.

/blog/meta-took-the-crawl-volume-chatgpt-kept-the-referrals.md from /blog/meta-took-the-crawl-volume-chatgpt-kept-the-referrals

Microsoft's AI Performance report in Bing Webmaster Tools, announced on 10 February 2026 and extended on 16 June 2026, shows the grounding queries behind citations in Microsoft Copilot. Google's generative AI performance report, read at its help documentation on 14 August 2026, publishes impressions grouped by page, country, date and device, and no query at all.

/blog/microsoft-copilot-seo-bing-names-the-grounding-query.md from /blog/microsoft-copilot-seo-bing-names-the-grounding-query

Every other major vendor gave site owners a robots.txt name to disallow. Microsoft's published guidance for keeping a page out of its AI answers is NOCACHE or NOARCHIVE in a meta robots tag, which is why none of the 15 tokens in our crawler registry belongs to Microsoft.

/blog/microsoft-ai-opt-out-is-a-meta-tag.md from /blog/microsoft-ai-opt-out-is-a-meta-tag

Mistral's crawler page, read on 12 September 2026, documents three crawlers, one each for user fetches, search indexing and training. Lantad requested /robots.txt once from 392 hostnames the same day. Of the 323 files that parsed, one named a Mistral token, and the file that tried to block Mistral used a token Mistral does not publish.

/blog/mistral-ai-crawler-one-of-323-robots-txt-files.md from /blog/mistral-ai-crawler-one-of-323-robots-txt-files

Google indexes the mobile version of a page and says so. The AI crawlers that decide whether ChatGPT and Perplexity can read a site publish user agent strings that name a desktop or no device at all. Lantad asked all 1,419 hostnames in this repository's committed corpus for their home page twice on 1 October 2026, once with a desktop platform token and once with a phone one, and 1,087 answered both. 989 of those returned an identical word count. 29 differed by a tenth or more, confirmed over two further rounds of fetching, and 26 of the 29 gave the phone less.

/blog/mobile-first-indexing-29-of-1087-home-pages-differed.md from /blog/mobile-first-indexing-29-of-1087-home-pages-differed

A measurement framework posted to arXiv on 29 April 2026 scores how much of each cited page reaches the answer text. Across 21,143 valid search-layer citations, Perplexity averaged 16.35 citations per prompt to ChatGPT's 6.88, and mean influence among fetched pages ran the other way, 0.0646 against 0.2713.

/blog/citation-count-is-not-answer-influence.md from /blog/citation-count-is-not-answer-influence

Lantad requested /robots.txt once from each of 1,027 hostnames across eight industry sectors on 12 September 2026, and 718 of the responses parsed as robots.txt. GPTBot was disallowed at the site root by 82 of them, seven more than the fourth placed token. The wider gap is inside each vendor: 59 of the 718 refused GPTBot while leaving OAI-SearchBot allowed.

/blog/most-blocked-ai-crawler-gptbot-82-of-718.md from /blog/most-blocked-ai-crawler-gptbot-82-of-718

Lantad asked the 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 21 October 2026 and read every permitted home page with no JavaScript executed. 1,059 answered HTTP 200 with HTML and carried 1,495 h1 elements between them. 630 of those pages carried exactly one, 170 carried more than one, 259 carried none, and one news home page carried 85. Google's own guidance files the number of headings under things it believes you should not focus on.

/blog/multiple-h1-tags-170-of-1059-declared-more-than-one.md from /blog/multiple-h1-tags-170-of-1059-declared-more-than-one

A Washington University in St. Louis team issued 55,393 trending queries over 40 days and found that 29.8 percent of the domains an AI Overview cites appear nowhere on the first page of results for the same query.

/blog/ai-overview-citations-are-not-page-one.md from /blog/ai-overview-citations-are-not-page-one

Lantad requested the home page of 392 hostnames on 12 September 2026, and the scanner's own platform detector found 28 of them built with Next.js. Across those 28 pages the React Server Component payload and the __NEXT_DATA__ blocks came to 3,897,015 characters. The visible prose came to 253,179, which is 2.2 percent of the characters that arrived.

/blog/nextjs-ai-crawlers-payload-15-times-the-prose.md from /blog/nextjs-ai-crawlers-payload-15-times-the-prose

Lantad's prompt tracker reported a 12.6 percent brand mention rate on its first live run on 13 August 2026. The truth was near 1 percent: of the 19 counted mentions, eight were an engine defining a Tagalog adjective, three resolved the name to a different business, and six were engines saying they had never heard of it. A name in an answer is presence, and presence is not identity.

/blog/nineteen-counted-mentions-about-two-were-the-company.md from /blog/nineteen-counted-mentions-about-two-were-the-company

Microsoft introduced NLWeb on 19 May 2025 as a way to reuse the Schema.org and RSS a site already publishes, and stated that every instance is also an MCP server. Cloudflare's deployment page for it, last updated 10 July 2026, documents an /ask and an /mcp endpoint and instructs the reader to ingest pages by crawling them.

/blog/nlweb-ask-endpoint-schema-org-data-model.md from /blog/nlweb-ask-endpoint-schema-org-data-model

Lantad requested the home page of all 392 hostnames in this repository's committed platform corpus on 28 September 2026 as LantadBot, following redirects and executing no JavaScript. 383 answered HTTP 200 with HTML and 27 of those carried zero words of readable text. 11 of the 27 sat in the 42 site no code stratum, the worst result of the ten platform groups read that day, and once one bot challenge is set aside 10 of them were real pages that served nothing. Not one of those 10 sites disallowed GPTBot in robots.txt.

/blog/no-code-seo-10-of-42-served-zero-words.md from /blog/no-code-seo-10-of-42-served-zero-words

Lantad fetched robots.txt and the home page of all 42 hostnames in the bubble-nocode stratum of this repository's committed platform seed file on 23 October 2026, with no JavaScript executed and no browser involved. 40 answered HTTP 200 with an HTML content type. The shipped extractor read zero words from 10 of those 40, and every one of the 10 served a title and a meta description. The shipped platform detector returned no match on 20 of the 40, including all 5 pages whose served HTML referenced bubble.io, because the 15 signals it carries include none for Bubble.

/blog/no-code-seo-10-of-40-served-zero-words.md from /blog/no-code-seo-10-of-40-served-zero-words

Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 5 October 2026 and read every permitted home page with no JavaScript executed. 1,077 answered HTTP 200 with HTML. 7 of them sent noarchive, the page level directive Amazon's own crawler page reads as do not train and Microsoft's reads as exclusion from Bing Chat answers, and Google's current documentation says it no longer uses at all. None sent nocache.

/blog/noarchive-7-of-1077-home-pages-sent-it.md from /blog/noarchive-7-of-1077-home-pages-sent-it

Lantad requested /robots.txt from all 1,419 hostnames in this repository's two committed corpus seed files on 13 October 2026 as LantadBot. 1,014 answered with a file that parsed into at least one group. Four of those files named OAI-AdsBot, the crawler OpenAI documents as validating pages submitted as ads on ChatGPT, against 170 that named GPTBot. On three of the four, deleting the named group changes no verdict.

/blog/oai-adsbot-4-of-1014-robots-txt-files-named-it.md from /blog/oai-adsbot-4-of-1014-robots-txt-files-named-it

Lantad requested /robots.txt once from each of 170 hostnames on 8 September 2026. 140 answered with a file that parsed as text. 25 of those name GPTBot, 16 name OAI-SearchBot, and 11 name GPTBot without ever naming the crawler OpenAI's documentation ties to appearing in ChatGPT search answers.

/blog/oai-searchbot-16-of-140-files-named-it.md from /blog/oai-searchbot-16-of-140-files-named-it

Schema.org began publishing domain counts for its whole vocabulary on 4 June 2026. Counting the July 2026 file on 31 July 2026 gives 958 types and 4,587 properties, of which 4,217 sit in the lowest bucket of fewer than 1,000 domains and 16 types reach the top bucket of more than 10 million.

/blog/sixteen-schema-types-reach-ten-million-domains.md from /blog/sixteen-schema-types-reach-ten-million-domains

Lantad requested the home page of all 1,419 hostnames in this repository's committed corpus on 29 September 2026 and read the head of every response with no JavaScript executed. 1,135 answered HTTP 200 with an HTML content type, and 948 of those carried at least one og: property. Of those 948, 578 carried all four properties the Open Graph protocol calls required and 370 did not. On 14 of the 767 pages declaring an og:url, the URL the page gave as its own permanent identifier did not point at the page that served it.

/blog/open-graph-tags-370-of-948-missing-a-required-property.md from /blog/open-graph-tags-370-of-948-missing-a-required-property

Lantad asked all 1,419 hostnames in this repository's committed corpus for robots.txt on 28 September 2026, read the home page of every host that allowed it with no JavaScript executed, then requested every logo URL the markup declared. 1,093 home pages answered HTTP 200 with HTML, 520 carried an Organization node, and 433 of those declared a logo. 389 of the 433 URLs returned image bytes: 22 answered 404, eight returned a web page, five never answered.

/blog/organization-schema-logo-44-of-433-returned-no-image.md from /blog/organization-schema-logo-44-of-433-returned-no-image

Lantad requested the home page of each of the 1,419 hostnames in this repository's two committed corpus seed files on 17 October 2026, as LantadBot with no JavaScript executed, and ran the shipped entity assessor over the 1,084 that delivered HTML. 448 declared an identity node. 171 stated what kind of organisation they are and 277 did not. Of the 63 industry-corpus sites that stated it through the type itself, 55 chose a subtype belonging to their own sector.

/blog/organization-schema-type-277-of-448-declared-no-category.md from /blog/organization-schema-type-277-of-448-declared-no-category

Lantad requested the home page of every hostname its robots.txt permitted across this repository's two committed corpus seed files on 15 October 2026, as LantadBot/1.0, redirects followed, no JavaScript executed. 1,093 answered HTTP 200 with HTML. 25 of them delivered more than 2,000,000 bytes of uncompressed HTML, the page size limit Google's Googlebot documentation states, and not one of the 1,093 exceeded that figure on the wire. Truncating each of the 25 at the documented cutoff and re-running this scanner's own extractor removed 12,129 of their 45,554 words, 11,441 of them main content, and left 14 of the 25 with nothing missing at all.

/blog/page-size-limit-25-of-1093-passed-2mb.md from /blog/page-size-limit-25-of-1093-passed-2mb

Cloudflare's pay per crawl answers an unpaid AI crawler with HTTP 402 Payment Required and a crawler-price header naming the amount. RFC 9110 defines that status code in one sentence: it is reserved for future use.

/blog/pay-per-crawl-runs-on-an-undefined-status-code.md from /blog/pay-per-crawl-runs-on-an-undefined-status-code

Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 16 October 2026, as LantadBot with no JavaScript executed. 1,013 parsed into at least one user-agent group. 102 named PerplexityBot and 51 named Perplexity-User. 45 files disallowed PerplexityBot at the root, and 20 of those 45 left Perplexity-User allowed, only one of which named it anywhere. Perplexity's own documentation states that the user fetcher generally ignores robots.txt rules.

/blog/perplexitybot-robots-txt-102-of-1013-named-it.md from /blog/perplexitybot-robots-txt-102-of-1013-named-it

Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 6 October 2026 as LantadBot, read every permitted home page with no JavaScript executed, then followed each page's own link to an about, team or company page and read that too. 1,079 home pages and 709 about pages answered HTTP 200 with HTML. 111 of the 1,079 sites declared a Person node on one of those two pages, 418 declared an organisation and no person at all, and of the 518 named Person nodes found, 272 carried neither of the two properties Google's own documentation names for working out who an author is.

/blog/person-schema-111-of-1079-sites-named-a-human.md from /blog/person-schema-111-of-1079-sites-named-a-human

Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 10 October 2026, read every permitted home page with no JavaScript executed, then followed each page's own link to a pricing page and read that too. 205 home pages linked one, 196 answered HTTP 200 with HTML, and 195 answered again on a second pass the same day. 146 of the 195 showed a currency amount in the text the extractor read, and 28 declared a price in JSON-LD.

/blog/pricing-page-schema-120-of-195-declared-none.md from /blog/pricing-page-schema-120-of-195-declared-none

Lantad asked the 164 commerce hostnames in this repository's two committed corpus seed files for robots.txt on 22 September 2026, then walked each site to a page carrying an add to cart control and read that page with no JavaScript executed. 50 product pages answered. 42 declared a Product node in JSON-LD, 41 of those 42 carried a price, a currency and an availability value, and 22 carried no GTIN, MPN or other manufacturer identifier.

/blog/product-schema-42-of-50-declared-it.md from /blog/product-schema-42-of-50-declared-it

Google's recipe documentation marks exactly two properties required, name and image, and leaves recipeIngredient and recipeInstructions as recommendations. Lantad asked all 1,419 corpus hostnames for robots.txt on 18 October 2026, read every permitted home page with no JavaScript executed, followed each site's own link to a recipe surface and read that too. 60 Recipe nodes answered on 60 pages across 13 hosts. 59 carried both required properties, and 5 declared a method or an ingredient list that contained no text at all.

/blog/recipe-schema-5-of-60-nodes-declared-an-empty-field.md from /blog/recipe-schema-5-of-60-nodes-declared-an-empty-field

Google added Retry-After to its reduce crawl rate documentation on 6 October 2026, and when Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for their home page once on 22 October 2026, 63 of the 1,388 that answered returned one of the three status codes that page names and not one of them carried the header.

/blog/reduce-crawl-rate-0-of-63-sent-retry-after.md from /blog/reduce-crawl-rate-0-of-63-sent-retry-after

Reflection AI published crawler documentation for a new training bot in late September 2026. Lantad requested the robots.txt of all 1,419 hostnames in this repository's two committed corpus seed files on 3 October 2026 as LantadBot, and 1,068 returned a parseable file. One named Reflectionbot. On 1,010 of the rest the wildcard group decided, and of the 77 files that refuse GPTBot at the site root, 72 allow Reflectionbot there.

/blog/reflectionbot-1-of-1068-robots-txt-files-named-it.md from /blog/reflectionbot-1-of-1068-robots-txt-files-named-it

HTTP defines a response header that lets an overloaded server tell a client exactly when to retry. Read on 14 August 2026, six crawler documentation pages published by Google, OpenAI, Anthropic and Perplexity describe what a server error does to crawling and none of the six names Retry-After anywhere.

/blog/no-crawler-vendor-documents-retry-after.md from /blog/no-crawler-vendor-documents-retry-after

Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 10 October 2026, read every permitted home page with no JavaScript executed, then followed each page's own link to a returns, refunds, shipping or delivery page and read that too. 6 of the 1,072 readable home pages declared a MerchantReturnPolicy node, 129 linked a policy page, and across the 201 policy pages that answered, not one declared a return or shipping policy in its markup.

/blog/return-policy-structured-data-6-of-1072-home-pages.md from /blog/return-policy-structured-data-6-of-1072-home-pages

Lantad asked all 1,419 hostnames in this repository's committed corpus for robots.txt on 27 September 2026, then read the home page of every host that allowed it with no JavaScript executed. 1,075 answered HTTP 200 with HTML and 615 of those carried at least one JSON-LD block that parsed. 33 of the 615 declared an aggregateRating, 73 nodes between them, and on 14 of the 33 every rating sat on a LocalBusiness or Organization type naming the site's own business, which Google's review snippet documentation says is ineligible for the star review feature.

/blog/review-schema-33-of-615-declared-a-rating.md from /blog/review-schema-33-of-615-declared-a-rating

Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 22 October 2026 and parsed the 1,061 files that resolved into at least one user-agent group. Those files carried 18,427 Allow rules against 60,514 Disallow rules. Taking each Allow line out and re-running the shipped evaluator against the line's own path, 15,591 of the 18,427 changed the answer and 2,714 did not. At file level the split is much closer: of the 716 files whose Allow lines could be probed, 343 held not one line that changed anything.

/blog/robots-txt-allow-343-of-716-changed-no-verdict.md from /blog/robots-txt-allow-343-of-716-changed-no-verdict

Read out of the golden fixtures this scanner captured on 15 July 2026, robots.txt permitted every one of the 60 user agent probes that ran against five real pages. Four of those probes came back HTTP 403, and seven came back HTTP 200 carrying visible text that matched the honest baseline only 87 percent.

/blog/robots-txt-allowed-sixty-probes-eleven-differed.md from /blog/robots-txt-allowed-sixty-probes-eleven-differed

Lantad asked all 1,419 hostnames in this repository's committed corpus for robots.txt on 10 October 2026, then read every home page it was allowed to read with no JavaScript executed. 1,090 answered HTTP 200 with HTML, 997 of those served a robots.txt that parsed, and 925 of the 997 allowed GPTBot the home page itself. 78 of those 925 then disallowed at least one script or stylesheet the same page loads from the same host, 495 references in total. A further 183 loaded a blocked resource from a host they do not control.

/blog/robots-txt-blocked-resources-78-of-925-home-pages.md from /blog/robots-txt-blocked-resources-78-of-925-home-pages

Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 13 October 2026. 1,057 returned a text file. Seven of them carried a Noindex line, 33 lines between them, and this scanner's own evaluator confirmed that 12 of those lines name a path GPTBot is still allowed to fetch.

/blog/robots-txt-noindex-7-of-1057-files-wrote-it.md from /blog/robots-txt-noindex-7-of-1057-files-wrote-it

Lantad requested /robots.txt once from each of 392 hostnames on 10 September 2026 with redirect following switched off. 391 answered, 122 of them with a 3xx, and every one of those 122 Location headers pointed at a hostname other than the one asked. RFC 9309 says to follow such a redirect and apply the file it reaches to the original authority.

/blog/robots-txt-redirect-122-of-391-crossed-hosts.md from /blog/robots-txt-redirect-122-of-391-crossed-hosts

A Sitemap line is not scoped to any user-agent group, which suggests a site could refuse an AI crawler and still hand it a full URL inventory in the same file. Lantad requested /robots.txt once from each of 1,419 hostnames on 17 September 2026 and then fetched the first sitemap each file declared. 1,016 files parsed into at least one group, 839 declared a sitemap, 108 refuse at least one of the 15 AI crawler tokens this scanner evaluates, and on 92 of the 93 refusing files that also publish a map, the refusal already covers the map.

/blog/robots-txt-sitemap-839-of-1016-declare-one.md from /blog/robots-txt-sitemap-839-of-1016-declare-one

Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 23 October 2026 and parsed every answer with the shipped parser in core/src/robots.ts. 1,063 files parsed, carrying 75,739 Allow and Disallow rules. 51 files held a byte outside ASCII somewhere, 46 confined it to a comment or a Sitemap line, and 5 put one inside a rule path. Those 5 wrote 22 such rules, and the shipped matcher bound none of the 22 against the percent-encoded form of the path they name.

/blog/robots-txt-special-characters-22-raw-paths-none-bound-encoded.md from /blog/robots-txt-special-characters-22-raw-paths-none-bound-encoded

A robots.txt file governs one host and nothing else, so the rule you wrote on example.com does not reach blog.example.com. Lantad resolved six content subdomain names under each of 1,027 hostnames on 18 September 2026 and found 277 pairs where a subdomain and its apex each served a robots.txt of its own. 73 of those pairs disagreed about at least one of the 15 AI crawler tokens this scanner evaluates, and 262 of the 277 named no AI crawler at all.

/blog/robots-txt-subdomains-73-of-277-differ.md from /blog/robots-txt-subdomains-73-of-277-differ

One GET per hostname on 5 September 2026 for the robots.txt of the six real sites this scanner captured on 15 July 2026. All six answered HTTP 200. Between them the files declare 48 user-agent groups, and not one of those groups names any of the 15 AI crawler tokens this scanner evaluates.

/blog/robots-txt-testen-six-files-no-ai-crawler-named.md from /blog/robots-txt-testen-six-files-no-ai-crawler-named

Lantad asked 1,419 hostnames for /robots.txt on 19 September 2026 and checked every answer against RFC 9309 and Google's published parsing rules. 1,104 returned HTTP 200. 33 of those returned HTML rather than robots.txt text, 12 returned nothing at all, and of the 1,059 that held real directives, 232 carried at least one defect a validator can name.

/blog/robots-txt-validator-232-of-1059-carried-a-defect.md from /blog/robots-txt-validator-232-of-1059-carried-a-defect

The Really Simple Licensing 1.0 specification, published on 10 December 2025, adds a directive named License to robots.txt, and states in the same section that it does not modify the access permissions expressed by Allow or Disallow. RFC 9309 tells crawlers they MAY interpret records it does not define. Four vendor crawler documentation pages, read on 9 August 2026, do not mention RSL at all.

/blog/rsl-license-directive-changes-no-access.md from /blog/rsl-license-directive-changes-no-access

Lantad read the home page of 1,419 hostnames on 25 September 2026 with no JavaScript executed. 1,068 answered HTTP 200 with HTML, 433 carried an identity node the scanner recognises, and 317 of those declared sameAs on it. Those 317 pointed at 185 distinct hosts, of which the six largest are social networks: 310 named a social profile and 85 named a Wikipedia or Wikidata entry.

/blog/sameas-schema-85-of-317-named-a-reference-entry.md from /blog/sameas-schema-85-of-317-named-a-reference-entry

Six pages captured on 15 July 2026 exposed 985 unique internal paths between them. On 29 August 2026 all six of those sites named a sitemap in robots.txt, five of the named sitemaps resolved, every one of the five was an index rather than a page list, and together they declared 200,712 URLs.

/blog/scan-a-website-for-all-pages-985-linked-200712-declared.md from /blog/scan-a-website-for-all-pages-985-linked-200712-declared

Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 16 October 2026, then read every permitted home page as LantadBot with no JavaScript executed. 1,085 answered a 2xx with HTML, 631 carried at least one JSON-LD block and 626 held one that parsed. Across the 2,181 typed nodes those pages declared, 88 distinct type strings appeared, against the 939 classes schema.org release 30.1 defines. WebSite, Organization and WebPage accounted for 1,029 of the 2,227 type declarations, and five of the 88 strings are not types schema.org defines at all.

/blog/schema-markup-types-88-of-939-on-626-home-pages.md from /blog/schema-markup-types-88-of-939-on-626-home-pages

Lantad requested the home page of all 1,419 hostnames in this repository's committed corpus on 21 September 2026 and read the raw bytes with no JavaScript executed. 1,072 answered HTTP 200 with HTML, and 612 of those carried at least one JSON-LD block, 1,143 blocks between them. 117 of the 612 carried a defect, 16 blocks never parsed as JSON, and on 7 sites not one block on the page did.

/blog/schema-markup-validator-117-of-612-carried-a-defect.md from /blog/schema-markup-validator-117-of-612-carried-a-defect

Counted with an HTML parser on 28 August 2026 across the raw and rendered captures of five real pages this scanner stored on 15 July 2026, microdata appears zero times. RDFa appears 204 times and 5 of those name a schema.org type, all of them on one page, which publishes a BreadcrumbList and scores 0 for schema because this scanner reads only JSON-LD.

/blog/schema-structured-data-syntax-decided-the-score.md from /blog/schema-structured-data-syntax-decided-the-score

Google made platform properties globally available on 29 July 2026, so Instagram, TikTok, X and YouTube content now reports inside Search Console. We fetched all four robots.txt files on 31 July 2026 and parsed them with our own parser: Instagram and X blocked 15 of our 15 crawler tokens, TikTok blocked 14, and YouTube blocked none.

/blog/search-console-platforms-block-ai-crawlers.md from /blog/search-console-platforms-block-ai-crawlers

Google's generative AI performance report, read at its Search Console help documentation on 14 August 2026, offers one metric and four dimensions. There is no clicks column, no click-through rate, no average position and no query dimension, and the Search and Discover versions count an impression by opposite rules.

/blog/generative-ai-report-counts-impressions-only.md from /blog/generative-ai-report-counts-impressions-only

Lantad asked all 1,419 hostnames in this repository's committed corpus for robots.txt on 28 September 2026, then read the home page of every host that allowed it with no JavaScript executed. 1,076 answered HTTP 200 with HTML and 619 of those carried at least one JSON-LD block that parsed. 231 of the 619 declared a SearchAction, 224 of the 231 hosts gave an address a machine could actually build, and 57 of those addresses point at a path the same site's robots.txt disallows.

/blog/searchaction-schema-57-of-224-endpoints-robots-closed.md from /blog/searchaction-schema-57-of-224-endpoints-robots-closed

Lantad asked all 1,419 hostnames in this repository's committed corpus for robots.txt on 9 October 2026 and read each home page it was allowed to read, with no JavaScript executed. 1,066 answered HTTP 200 with HTML. 687 declared at least one main element, 379 declared none, and 301 declared neither a main nor an article. Across the 1,066 pages the shipped extractor classified 1,335,841 words, of which 418,642 were boilerplate inside nav, header, footer or aside.

/blog/semantic-html-for-ai-379-of-1066-declared-no-main.md from /blog/semantic-html-for-ai-379-of-1066-declared-no-main

Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 7 October 2026, then read every permitted home page as LantadBot with no JavaScript executed. 1,078 answered HTTP 200 with HTML and 616 of those carried at least one JSON-LD block that parsed. 20 of the 616 declared a Service node, 89 nodes between them, and 46 of the 89 carried none of the 22 properties schema.org defines on the Service type itself.

/blog/service-schema-markup-20-of-616-declared-it.md from /blog/service-schema-markup-20-of-616-declared-it

Parsed with an HTML parser on 24 August 2026 out of five real pages this scanner captured and stored on 15 July 2026, 7 of the 102 typed JSON-LD nodes carry an author property. Six of those seven sit on Review nodes and name the customer who gave a testimonial. The seventh names an organisation. No page in the set says which person wrote it, and all five carry zero meta author tags, zero rel=author links and zero address elements.

/blog/seven-schema-nodes-named-an-author.md from /blog/seven-schema-nodes-named-an-author

A single-author paper from a Cisco DevNet engineer logged 15 AI clients fetching one documentation page across three trials in February and March 2026. Most of the coding agents identified themselves as an HTTP library, and none of the 15 asked for llms.txt.

/blog/coding-agents-arrive-as-curl-and-axios.md from /blog/coding-agents-arrive-as-curl-and-axios

Lantad requested /products.json from the 49 hostnames in a 123 host sample whose home pages loaded Shopify storefront assets, on 9 September 2026. 39 answered HTTP 200, 38 of those parsed as a product array holding 1,112 products, and not one of the 49 robots.txt files mentions the path.

/blog/shopify-ai-crawlers-1112-products-as-open-json.md from /blog/shopify-ai-crawlers-1112-products-as-open-json

Lantad requested /robots.txt from all 1,419 hostnames in this repository's two committed corpus seed files on 29 September 2026 as LantadBot. 859 declared a sitemap and 785 of those resolved to a URL set holding 2,768,047 entries, of which 2,004,336 carried a lastmod. Two of those URLs per site were then fetched, and on the 440 responses that carried a Last-Modified header, 340 disagreed with the sitemap's claim about the same page by more than a day.

/blog/sitemap-lastmod-340-of-440-disagreed-with-the-header.md from /blog/sitemap-lastmod-340-of-440-disagreed-with-the-header

Lantad requested /robots.txt from all 1,419 hostnames in this repository's two committed corpus seed files on 1 October 2026, then fetched the first sitemap each file declared. 1,104 answered HTTP 200 and 871 of those declared a sitemap. 827 of the declared sitemaps delivered a body, 38 answered a non-200 status, one timed out, and five named a value that is not an absolute URL.

/blog/sitemap-validator-44-of-871-delivered-no-file.md from /blog/sitemap-validator-44-of-871-delivered-no-file

The standard advice is to disallow training crawlers and allow search crawlers. That instruction needs two published names to write, and Lantad's registry records six of nine vendors publishing exactly one, while Cloudflare reported on 1 July 2026 that mixed-use crawlers are over 36 percent of crawler activity on its network.

/blog/six-vendors-publish-one-crawler-token.md from /blog/six-vendors-publish-one-crawler-token

Lantad asked all 1,419 hostnames in this repository's committed corpus for a path that cannot exist on 20 September 2026. 1,094 answered both that request and their own home page. 936 returned 404, 125 returned HTTP 200, and on a second nonexistent path of a different shape 111 of those 125 returned 200 again.

/blog/soft-404-125-of-1094-answered-200.md from /blog/soft-404-125-of-1094-answered-200

Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 12 October 2026 and read every permitted home page once with no JavaScript executed. 1,074 answered HTTP 200 with HTML and 617 of those held at least one JSON-LD block that parsed. 57 declared a SoftwareApplication, WebApplication or MobileApplication node, 77 nodes between them. All 77 carried a name, 76 carried the recommended applicationCategory, and 4 carried all three of the properties Google's documentation lists as required.

/blog/softwareapplication-schema-57-of-1074-declared-4-complete.md from /blog/softwareapplication-schema-57-of-1074-declared-4-complete

Lantad requested the robots.txt and then the home page of all 1,419 hostnames in this repository's committed corpus on 30 September 2026, read the delivered bytes with no JavaScript executed, and analysed the 1,114 pages that were both allowed and readable. 8 of them declared the schema.org speakable property, and a second pass over the news strata found it on 5 of the 69 hostnames where an Article typed page could be identified. Of the 12 CSS selectors those declarations used that name a single element by id, class or tag name, 4 matched no element in the delivered HTML at all and 1 matched an element carrying no text.

/blog/speakable-schema-8-of-1114-and-5-of-12-selectors.md from /blog/speakable-schema-8-of-1114-and-5-of-12-selectors

Lantad fetched the robots.txt of 72 Squarespace served hostnames on 10 September 2026. 66 answered HTTP 200, and 64 of those carried the platform's generated file naming 26 AI crawler tokens. On 61 of the 66 those 26 names sat in the same group as the wildcard, which under RFC 9309 gives them exactly the access every other crawler gets.

/blog/squarespace-ai-crawlers-61-of-66-named-not-blocked.md from /blog/squarespace-ai-crawlers-61-of-66-named-not-blocked

A paper posted to arXiv on 13 September 2025 scored 1,100 pages that Brave Summary, Google AI Overviews and Perplexity had cited across 70 prompts, using a sixteen pillar framework it calls GEO-16. Structured data placed third of the six pillars the paper reports, at r = 0.63, behind Metadata Freshness at 0.68 and Semantic HTML at 0.65.

/blog/structured-data-came-third-in-a-citation-audit.md from /blog/structured-data-came-third-in-a-citation-audit

Parsed with an HTML parser on 26 August 2026 out of five real pages this scanner captured and stored on 15 July 2026, the four JSON-LD blocks in the set carry 146 distinct human readable text values. 76 of those values appear somewhere in the visible text of the page they sit on. 70 do not. One page contributes 140 of the 146.

/blog/structured-data-for-ai-70-of-146-values-off-page.md from /blog/structured-data-for-ai-70-of-146-values-off-page

Lantad requested the home page of 392 hostnames on 11 September 2026 and parsed the bytes that came back with no JavaScript executed. 382 answered HTTP 200, 241 of those carried at least one JSON-LD block and 141 carried none. The 241 shipped 442 blocks between them, and eight of the 442 do not parse as JSON at all.

/blog/structured-data-141-of-382-home-pages-carried-none.md from /blog/structured-data-141-of-382-home-pages-carried-none

Lantad asked all 1,419 hostnames in this repository's committed corpus for robots.txt on 26 September 2026, then read the home page of every host that allowed it with no JavaScript executed. 1,081 answered HTTP 200 with HTML and 615 of them carried at least one JSON-LD block that parsed. 313 of those 615 put an @id on no node at all, 545 described more than one typed thing, and 193 of the 545 joined two of those things with an @id reference that resolved on the page.

/blog/structured-data-313-of-615-gave-no-node-an-id.md from /blog/structured-data-313-of-615-gave-no-node-an-id

Dries Buytaert published a month of Cloudflare logs from his own site on 5 March 2026 covering ten AI crawlers and 58,279 requests, of which 4,967 fetched a markdown file and zero used content negotiation, three weeks after Cloudflare shipped a feature that converts pages to markdown only when a request carries an Accept header naming text/markdown.

/blog/no-ai-crawler-used-content-negotiation.md from /blog/no-ai-crawler-used-content-negotiation

Ten products sold into AI visibility were read at their own published pages on 17 August 2026. Three of them examine the customer's own pages, four read a record of requests that already arrived, and six put prompts to models and read the answers back. The crawler half of the category is bundled free or sold as an annual licence, and the answer half is sold monthly by prompt volume.

/blog/ten-ai-visibility-products-three-fetch-your-pages.md from /blog/ten-ai-visibility-products-three-fetch-your-pages

Lantad requested robots.txt, then the home page, then every legal page the home page linked, for all 1,419 hostnames in this repository's two committed corpus seed files on 14 October 2026, executing no JavaScript. 656 hostnames delivered a readable legal page. Reading all 32 candidate clauses by hand confirmed 15 whose own terms restrict the use of site content for AI or machine learning training, and 11 of those 15 leave all nine training crawler tokens this scanner evaluates allowed at the root. Six hostnames in the whole corpus serve a TDMRep file at /.well-known/tdmrep.json, and none of the six is one of the 15.

/blog/text-and-data-mining-opt-out-15-in-prose-none-in-a-file.md from /blog/text-and-data-mining-opt-out-15-in-prose-none-in-a-file

Two golden fixtures stored in this repository produce an identical composite of 34.4, an identical grade of F, and identical access, structure and schema sub-scores. One of them ships zero words in its initial HTML. The other ships all 485 of them, inside a JSON payload. The only stored number that separates the two is retrievability, and retrievability is worth nothing in the grade.

/blog/hydration-json-scored-the-same-as-no-text.md from /blog/hydration-json-scored-the-same-as-no-text

Declarative shadow DOM ships component text in the HTML, inside a template element. Lantad's extraction ruleset drops template subtrees, the default DOM serialization omits shadow roots, and the shadow piercing fallback that recovers the text runs in one of our two renderers.

/blog/shadow-dom-text-and-the-extractor.md from /blog/shadow-dom-text-and-the-extractor

Article 50 of Regulation (EU) 2024/1689 has applied since 2 August 2026 and obliges providers of AI systems that generate synthetic text to mark the output in a machine-readable format. The Regulation names no format. Every technique it mentions sits in one sentence of one recital, inside a list introduced by the words such as, and the string HTML appears in the published text zero times.

/blog/ai-act-marks-ai-text-and-names-no-format.md from /blog/ai-act-marks-ai-text-and-names-no-format

Google began testing a Search Console setting on 3 June 2026 that excludes a site from AI Overviews and AI Mode. It is an account setting rather than a file, so no external scan of your site can tell you whether it is set.

/blog/ai-overviews-opt-out-does-not-live-on-your-site.md from /blog/ai-overviews-opt-out-does-not-live-on-your-site

The IETF vocabulary for expressing AI usage preferences reached its sixth draft on 28 April 2026. The companion draft that defines where to write one down expired on 1 May 2026, so there is still nothing on your site for a crawler to read.

/blog/the-ai-preferences-standard-you-cannot-deploy-yet.md from /blog/the-ai-preferences-standard-you-cannot-deploy-yet

A live index built from real scans, showing how readable the measured web is to AI crawlers, with the sample size stated beside every figure.

/research.md from /research

Regulation (EU) 2024/1689 obliges general-purpose AI model providers to identify and comply with machine-readable rights reservations, and the phrase robots.txt appears nowhere in it. The Commission's Code of Practice names the file and the RFC that defines it, and the Commission's power to fine under Article 101 applies from 2 August 2026.

/blog/eu-ai-act-never-writes-robots-txt.md from /blog/eu-ai-act-never-writes-robots-txt

Dated, argued posts built on measured scan data, plus guides and product notes. Every figure carries its measurement date.

/blog.md from /blog

Every rival name on a Lantad report is still extracted from the answer text, through screens we built after shipping two pieces of output that should never have passed. Since 2026-08-07 you can also pin competitors to a roster, which reverses a position this post originally presented as a design choice, so the post now narrates that change instead of quietly absorbing it.

/blog/rival-names-are-extracted-not-entered.md from /blog/rival-names-are-extracted-not-entered

A robots.txt study posted to arXiv in October 2025 reports 60.0 percent of reputable news sites disallowing at least one AI crawler against 9.1 percent of misinformation sites, and an average of 15.5 named AI user agents against 0.77.

/blog/who-blocks-ai-crawlers-splits-by-credibility.md from /blog/who-blocks-ai-crawlers-splits-by-credibility

Google-Extended, Applebot-Extended and anthropic-ai are robots.txt product tokens that send no user agent of their own, so a log search for them returns zero forever, and that zero does not mean what it looks like.

/blog/the-crawler-tokens-that-never-appear-in-your-logs.md from /blog/the-crawler-tokens-that-never-appear-in-your-logs

Parsed with an HTML parser out of five real pages this scanner captured and stored on 15 July 2026, the whole set holds one time element and one JSON-LD block carrying a date. Three of the five score a perfect 1.0 on our schema sub-score, and only one of those three says anywhere in its markup when it was written or last changed.

/blog/full-schema-marks-and-one-page-carried-a-date.md from /blog/full-schema-marks-and-one-page-carried-a-date

The W3C TDM Reservation Protocol defines four places a site can record that AI mining rights are reserved. Read on 4 August 2026, the crawler documentation published by OpenAI, Anthropic and Google names robots.txt in every case and mentions none of the four.

/blog/tdmrep-not-named-in-crawler-documentation.md from /blog/tdmrep-not-named-in-crawler-documentation

Parsed out of five real pages this scanner captured and stored on 15 July 2026, four JSON-LD blocks hold 26,299 bytes and 102 typed nodes. One page accounts for 24,865 of those bytes and 96 of those nodes, 39 of them describing a navigation menu.

/blog/three-of-five-pages-carried-json-ld.md from /blog/three-of-five-pages-carried-json-ld

Lantad's own component library states that a table is the shape an AI crawler extracts most reliably, so the claim was checked against the pages this scanner has actually stored. Counted with an HTML parser on 21 August 2026 over five real pages captured on 15 July 2026, the whole set holds three table elements, all three on one page, all three navigation boxes, and none of the five pages carries a single caption element.

/blog/three-tables-on-five-pages-all-navigation.md from /blog/three-tables-on-five-pages-all-navigation

Digiday reported on 30 July 2026 that Time has converted its pages to markdown for AI systems and is selling FAQ shaped sponsored content inside those files, which turns the machine copy of a page into something that can differ from the human one by design.

/blog/ads-inside-the-markdown-ai-crawlers-read.md from /blog/ads-inside-the-markdown-ai-crawlers-read

SAGEO Arena ran ten published generative engine optimization strategies through retrieval, reranking and generation over a corpus of 171,003 crawled web documents. We relayed its body-text result here on 31 July 2026, at second hand, from a survey that cited it. The primary source, revised on 7 August 2026, also holds the scope the survey did not relay: rewriting the title, meta description, headings and JSON-LD instead moved the average retrieval hit rate from 0.58 to 0.71.

/blog/metadata-rewrites-gained-22-percent-of-retrievals.md from /blog/metadata-rewrites-gained-22-percent-of-retrievals

Cloudflare said on 1 July 2026 that from 15 September newly onboarded domains will block Training and Agent bots by default on pages that display ads. Your robots.txt will not mention it, because robots.txt was never the layer that answers.

/blog/two-layers-decide-if-ai-can-read-your-site.md from /blog/two-layers-decide-if-ai-can-read-your-site

Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt and then for their home page on 6 October 2026, as LantadBot, with no JavaScript executed. 1,072 answered HTTP 200 with HTML. 32 of those declared Vary: User-Agent, 30 of the 32 returned byte identical responses to a crawler user agent string and a desktop Chrome one, and the three pages whose word count actually changed between the two declared no User-Agent at all.

/blog/vary-user-agent-32-of-1072-home-pages-declared-it.md from /blog/vary-user-agent-32-of-1072-home-pages-declared-it

Cloudflare published Continuous accountability for Verified bots on 16 October 2026, naming two behaviours it observes after verification: whether a bot honours robots.txt in line with RFC 9309, and whether it slows down after an HTTP 429. Both signals are emitted by the site rather than by the bot, and Lantad has already published measurements of both across the 1,419 hostnames in this repository's two committed corpus seed files.

/blog/verified-bots-cloudflare-grades-robots-txt-and-429.md from /blog/verified-bots-cloudflare-grades-robots-txt-and-429

Lantad asked all 1,419 hostnames in this repository's committed corpus for robots.txt on 23 September 2026 and read each home page with no JavaScript executed. 1,084 answered HTTP 200 with HTML and 270 of those put a video in front of the crawler. 21 of the 270 declared video schema, the 43 VideoObject nodes between them all carried the three properties Google requires, and 2 carried a transcript.

/blog/video-schema-21-of-270-home-pages-declared-it.md from /blog/video-schema-21-of-270-home-pages-declared-it

Lantad requested /.well-known/http-message-signatures-directory once from each of 51 AI crawler operator hostnames on 15 September 2026. Four served a key directory, 40 answered 404, and three answered HTTP 200 with an HTML page. Of the nine vendors behind the 15 crawler tokens this scanner evaluates, two publish one.

/blog/web-bot-authentication-two-of-nine-publish-a-directory.md from /blog/web-bot-authentication-two-of-nine-publish-a-directory

Lantad fetched the robots.txt of 52 Webflow served hostnames on 11 September 2026. Of the 40 that answered on the owner's own domain, 7 returned HTTP 200 with zero bytes and 13 held a Sitemap line and no User-agent line, so 20 carried no rule of any kind. Six named GPTBot, and two of those six files were written by Cloudflare rather than by anyone at the site.

/blog/webflow-ai-crawlers-20-of-40-held-no-rule.md from /blog/webflow-ai-crawlers-20-of-40-held-no-rule

Lantad asked all 1,419 hostnames in this repository's committed corpus for robots.txt on 9 October 2026 and read each home page it was allowed to read, with no JavaScript executed. 1,072 answered HTTP 200 with HTML, 620 carried a JSON-LD block and 613 held at least one block that parsed. 407 of those declared a WebSite node and 510 declared an organisation node. 338 declared both, and on 158 of them no property in the document joined the two.

/blog/website-schema-markup-158-of-338-connected-neither.md from /blog/website-schema-markup-158-of-338-connected-neither

Six real captures, measured on one day in July. Two stores on the same platform produced opposite outcomes, and the difference was not their content.

/blog/what-a-crawler-meets-on-a-real-storefront.md from /blog/what-a-crawler-meets-on-a-real-storefront

Ahrefs measured 137,210 domains in May 2026 and found 97 percent of valid llms.txt files were never fetched at all. We ship an llms.txt tool, and we think you should know this.

/blog/what-the-evidence-says-about-llms-txt.md from /blog/what-the-evidence-says-about-llms-txt

The IETF working group on crawler authentication listed ten active Internet-Drafts on 29 July 2026, two of them filed on 19 July. The core protocol draft carries intended status Informational, and every mechanism it describes sits in places an external scan of your site cannot reach.

/blog/what-web-bot-auth-actually-specifies.md from /blog/what-web-bot-auth-actually-specifies

Lantad asked all 1,027 hostnames in the eight industry strata of this repository's committed corpus seed file for robots.txt on 20 October 2026 and computed a root verdict for seven AI crawler tokens with the scanner's own parser. 750 files parsed. News disallowed GPTBot at the root on 41 of its 69 files, against 2 of 97 in finance, 2 of 87 in travel and 3 of 120 in SaaS, and 616 of the 750 files named none of the seven tokens at all.

/blog/which-industries-block-ai-crawlers-41-of-69-news-files.md from /blog/which-industries-block-ai-crawlers-41-of-69-news-files

Sites sampled by the platform they are built on, from Framer and Webflow to Shopify and WordPress, each fetched as an AI crawler sees it and again in a browser, then compared. Per-platform figures, each with its sample size.

/research/crawlability-study.md from /research/crawlability-study

Every AI visibility tool prints a number. When the measurement failed, that number is a guess wearing a lab coat, so ours refuses to appear.

/blog/why-we-withhold-a-grade.md from /blog/why-we-withhold-a-grade

Run against Lantad's shipped robots.txt matcher on 27 August 2026, appending a trailing asterisk to six rule patterns changed the verdict on none of 168 path comparisons. Appending a trailing dollar sign to four patterns changed 36 of 112.

/blog/wildcard-robots-txt-trailing-star-changed-nothing.md from /blog/wildcard-robots-txt-trailing-star-changed-nothing

Lantad requested the robots.txt of 71 Wix served hostnames on 10 September 2026. 69 answered HTTP 200 and not one of them named an AI crawler token. Wix's help centre tells owners to paste four user agents into that file, and the fourth, BingAI, appears in no crawler documentation OpenAI or Microsoft publishes.

/blog/wix-ai-crawlers-zero-of-69-named-one.md from /blog/wix-ai-crawlers-zero-of-69-named-one

Lantad requested the home page and the robots.txt of 120 hostnames on 8 September 2026. 91 home pages answered HTTP 200, and 24 of those carried a WordPress signal. All 24 served a parseable robots.txt, and 16 of the 24 name none of the 15 AI crawler tokens this scanner evaluates.

/blog/wordpress-ai-crawlers-16-of-24-named-none.md from /blog/wordpress-ai-crawlers-16-of-24-named-none

Lantad requested the home page of 392 hostnames on 9 September 2026 and recorded the response headers. Seven responses carried an X-Robots-Tag. The five that said noindex all arrived as an HTTP 202 of 168 or 169 bytes whose body was a redirect to a captcha, and the same response comes back from the hosting company's own domain.

/blog/x-robots-tag-noindex-five-of-391-were-a-captcha.md from /blog/x-robots-tag-noindex-five-of-391-were-a-captcha

Lantad is an independent AI-visibility scanner. It measures whether AI crawlers can read your site and whether AI answers cite it, and it withholds any grade it could not measure.

/about.md from /about

Plain answers about what the AI Visibility Score measures, which AI crawlers Lantad checks, whether blocking AI hurts Google, and whether llms.txt helps.

/faq.md from /faq

What Lantad stores, for how long, and the rights that apply. Written from what the code does rather than from a template.

/privacy.md from /privacy

The terms that apply to using Lantad, including acceptable use of the scanner and the limits of what a report asserts.

/terms.md from /terms

Lantad's privacy policy, terms of service, security contact, and crawler conduct policy, in one place.

/legal.md from /legal

Every public page on lantad.co in markdown, grouped and linked, so a crawler can take the content without parsing HTML.

/md.md from /md