Every page on this site, in markdown

Paid engagements where Lantad fixes what the scan found, rather than only reporting it.

/services.md from /services

The scan is free and needs no account. Paid plans are there for what happens after: AI readability breaks quietly on a redeploy, and monitoring is what tells you the week it does.

/pricing.md from /pricing

Lantad is an AI visibility checker: it fetches your page the way a non-rendering AI crawler does, renders it the way a browser does, and reports the difference. Free, one URL, no signup.

/index.md from /

Puts two reports side by side so a change is attributable rather than remembered.

/compare.md from /compare

The full method: what is fetched, how prose parity is computed, how access is probed per crawler, and every weight in the composite.

/methodology.md from /methodology

Programmatic access to scans and reports, including the JSON a scan returns.

/developers.md from /developers

What LantadBot fetches, how it identifies itself, and how to block it. It never spoofs another crawler's user agent on the baseline fetch.

/bot.md from /bot

Tracks a set of prompts over time across several answer engines, so a change in what they say is visible rather than anecdotal.

/prompts.md from /prompts

Runs the same measurement across a set of URLs and reports them together, so a pattern across a site becomes visible.

/multiscan.md from /multiscan

Asks answer engines real questions about your brand and reports what comes back, including whether you are named at all.

/what-ai-says.md from /what-ai-says

The 15 AI crawler tokens Lantad tracks, grouped by purpose: training, search indexing, and fetches triggered by a person asking an assistant.

/tools/ai-crawlers.md from /tools/ai-crawlers

Eight checks, all free, and seven of them never ask for an email. Each one answers a single question, and each one links to the full scan when the answer turns out to be complicated.

/tools.md from /tools

Fetches your /llms.txt and reports whether it exists and how it is structured. Ships beside our own published finding that the evidence for llms.txt is weak.

/tools/llms-txt.md from /tools/llms-txt

Parses a robots.txt and reports, for each of the 15 named AI crawler tokens, whether it is allowed at the site root and which rule decided.

/tools/robots-txt-tester.md from /tools/robots-txt-tester

Fetches your page with GPTBot's published user agent and no JavaScript, then shows the text a non-rendering crawler actually receives.

/tools/what-gptbot-sees.md from /tools/what-gptbot-sees

An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.

/glossary/ai-crawler.md from /glossary/ai-crawler

AI visibility is whether AI systems can access, read, and faithfully represent your web pages, the prerequisite for being cited in an AI answer.

/glossary/ai-visibility.md from /glossary/ai-visibility

Answer Engine Optimization (AEO) is the practice of making a page readable, allowed, and structured enough that AI answer engines can use it as a source when they compose an answer.

/glossary/aeo.md from /glossary/aeo

Entity confidence is a free diagnostic on every graded Lantad report that scores, out of 100, how completely the JSON-LD in your crawler-visible HTML declares one identity and how far that identity is anchored by identifiers which resolve outside the page.

/glossary/entity-confidence.md from /glossary/entity-confidence

Generative Engine Optimization (GEO) is the practice of structuring and publishing content so that generative AI engines can fetch it, read it in full, and cite it when they answer a question.

/glossary/geo.md from /glossary/geo

llms.txt is a proposed plain-text markdown file placed at a site's root that gives AI systems a curated map of its most important content, so models can find and read what matters without parsing every page.

/glossary/llms-txt.md from /glossary/llms-txt

Prose Parity is Lantad's metric for the share of a page's rendered, human-visible text that is present in the raw HTML an AI crawler receives, where 100% means a crawler sees everything a human visitor sees.

/glossary/prose-parity.md from /glossary/prose-parity

Structured data is schema.org markup, usually written as JSON-LD, that labels a page's facts in a machine-readable format so AI systems can extract and attribute them without having to interpret the surrounding prose.

/glossary/structured-data.md from /glossary/structured-data

AI search has collected a lot of acronyms in a short time, several of which mean the same thing. These are the terms that appear on this site, with the definitions Lantad uses when it uses them.

/glossary.md from /glossary

A published Framer site is pre-rendered to static HTML and is usually readable without JavaScript, so when ChatGPT misses your content the cause is specific: a code component, an embed, or an interaction that keeps text out of the HTML.

/fix/framer.md from /fix/framer

Lovable server-renders new projects by default and serves older client-rendered projects a rendered copy to recognized crawlers, so many sites are readable. The gap is browser-fetched content and any crawler Lovable does not recognize.

/fix/lovable.md from /fix/lovable

Next.js hands AI crawlers fully rendered HTML when you use SSR, static generation, or server components, and an empty shell only when your content is rendered client-side or loaded in useEffect after the page mounts.

/fix/nextjs.md from /fix/nextjs

React is not invisible to AI, but a client-rendered single-page app ships an empty div and builds the page in the browser, so a crawler that does not run JavaScript receives a shell with almost no text.

/fix/react.md from /fix/react

Shopify themes are server-rendered Liquid, so most of your store is already readable to AI crawlers, and the real gap is the content third-party apps and widgets inject with JavaScript after the page loads.

/fix/shopify.md from /fix/shopify

Six patterns account for most of the readable content Lantad measures as missing. Each one has a specific cause in a specific stack, and a change your developer can make this week.

/fix.md from /fix

ChatGPT, Claude and Perplexity build answers from pages they could fetch and read. Five steps, worked in order, make a page one of them; nothing on the page promises a citation.

/how-to-get-cited.md from /how-to-get-cited

ChatGPT can only cite a page its crawlers are allowed to fetch and can read without JavaScript, so being permitted and readable comes first, and any citation follows from there.

/how-to-get-cited/chatgpt.md from /how-to-get-cited/chatgpt

Claude reaches your page through named Anthropic crawlers that honor robots.txt and work from your raw HTML, so being cited starts with being fetchable, allowed, and readable without JavaScript.

/how-to-get-cited/claude.md from /how-to-get-cited/claude

Google AI Overviews are built from Google's Search index, so getting cited starts with a page Googlebot can crawl, index, and read, and no site can promise the citation itself.

/how-to-get-cited/google-ai-overviews.md from /how-to-get-cited/google-ai-overviews

To be cited by Perplexity, a page first has to be readable: it must fetch cleanly, carry its real text in the raw HTML before any JavaScript runs, and stay open to PerplexityBot in robots.txt.

/how-to-get-cited/perplexity.md from /how-to-get-cited/perplexity

Ahrefs Brand Radar tracks brand mentions and citations across AI platforms at very large scale; Lantad measures whether an AI crawler could read the page those citations would point at, and names the fix when it could not.

/vs/ahrefs.md from /vs/ahrefs

Honest comparisons against named alternatives, including what each of them does better than Lantad.

/vs.md from /vs

Otterly.AI tracks what AI engines already say about your brand; Lantad measures and fixes whether AI crawlers can read your site in the first place.

/vs/otterly.md from /vs/otterly

Peec AI tracks what AI assistants already say about your brand; Lantad measures and fixes whether AI crawlers can read your site in the first place.

/vs/peec.md from /vs/peec

Profound tracks what AI assistants say about your brand across many engines; Lantad tests and fixes whether AI crawlers can read your pages in the first place.

/vs/profound.md from /vs/profound

PromptAlpha bundles AI answer tracking with done-for-you AEO articles; Lantad measures whether AI can read your pages, tracks what 8 answer engines say, and deliberately does not sell content production.

/vs/promptalpha.md from /vs/promptalpha

Semrush's AI Visibility Toolkit reports how often AI answers mention your brand; Lantad measures whether an AI crawler could read the page those answers would have to cite, and names the fix when it could not.

/vs/semrush.md from /vs/semrush

Counted on 25 August 2026 from five real pages this scanner captured and stored on 15 July 2026, the crawler view holds 16,636 visible words and 2,729 text blocks. 1,395 of those blocks sit inside nav, header, footer or aside, and between them they carry 2,920 words. More than half the blocks hold under a fifth of the text.

/blog/half-the-text-blocks-were-navigation.md from /blog/half-the-text-blocks-were-navigation

An audit posted to arXiv on 22 May 2026 put 712 queries to ChatGPT, Copilot, Gemini and Perplexity through their own interfaces, collected 26,266 cited URLs and scraped 19,154 of them. A detection classifier labelled 3,056 of those, approximately 16 percent, as AI-generated, ranging from 7.3 percent of ChatGPT's cited sources to 27.8 percent of Copilot's. The authors present the figure as a lower bound, because the same classifier missed 31.4 percent of a sample of articles already documented as AI-generated.

/blog/sixteen-percent-of-cited-sources-were-ai-generated.md from /blog/sixteen-percent-of-cited-sources-were-ai-generated

A probe of 185 media properties published on 4 August 2026 reports that 17.0 percent of the outlets giving a clean baseline answer at least one AI vendor with HTTP 402 Payment Required, that the gate is selective by vendor, and that none of it is declared in robots.txt.

/blog/publisher-tolls-are-invisible-in-robots-txt.md from /blog/publisher-tolls-are-invisible-in-robots-txt

Counted with an HTML parser over five real pages this scanner captured and stored on 15 July 2026, the rendered DOM held 466 img elements: 173 with a non-empty alt attribute, 246 with an explicitly empty one, and 47 with no alt attribute at all. Our own extractor reads none of them.

/blog/alt-text-on-466-captured-images.md from /blog/alt-text-on-466-captured-images

HasData fetched as well as read. Across a 2,096 domain enforcement subset measured in July 2026, 39.5 percent of the sites disallowing GPTBot returned it a live 200 when a request carrying that name actually knocked.

/blog/gptbot-bans-served-a-200-anyway.md from /blog/gptbot-bans-served-a-200-anyway

Counted on 26 August 2026 across the five real pages this scanner captured on 15 July 2026, the details element appears 31 times, every one collapsed by default, and not a single summary label contains a question mark.

/blog/thirty-one-disclosure-widgets-held-no-faq.md from /blog/thirty-one-disclosure-widgets-held-no-faq

Counted with an HTML parser over five real pages this scanner captured and stored on 15 July 2026, the rendered DOM holds 1,951 links with an href and 85 of them carry no text between the tags. Google's link documentation names two fallbacks for an empty link, the title attribute and a nested image's alt text. Thirty four of the 85 carry neither, and 23 carry none of the three.

/blog/textless-links-named-only-by-aria-label.md from /blog/textless-links-named-only-by-aria-label

Counted with an HTML parser on 24 August 2026 over five real pages this scanner captured and stored on 15 July 2026, the head of those documents holds 85 meta elements. Two are a robots meta tag, and both of those carry nothing but an image preview size. Five of the five tell a social network how to draw a card.

/blog/eighty-five-meta-elements-two-addressed-a-crawler.md from /blog/eighty-five-meta-elements-two-addressed-a-crawler

RFC 9309 gives a missing robots.txt and a broken robots.txt opposite meanings: a 4xx lets a crawler access any resource on the server, a 5xx obliges it to assume complete disallow. Google's documentation carves 429 out of that rule. Lantad's code does not.

/blog/robots-txt-404-and-503-are-opposites.md from /blog/robots-txt-404-and-503-are-opposites

A preprint posted to arXiv on 18 June 2026 by a single author at the AI visibility vendor Ranqo breaks 149,912 citations from five AI engines into nine source classes. The brand's own domain accounts for 4,392 of them, 2.9 percent, while 75.2 percent point at other companies in the same category.

/blog/ai-citations-mostly-point-at-other-companies.md from /blog/ai-citations-mostly-point-at-other-companies

What actually comes back when a tool asks Gemini a grounded question, why the cited URI can be a Google redirect rather than a publisher, and which parts of a citation table survive that.

/blog/a-gemini-citation-tool-is-not-an-ai-overview-tool.md from /blog/a-gemini-citation-tool-is-not-an-ai-overview-tool

A benchmark posted to arXiv on 17 August 2026 fetched 10,095 pages out of Google Search and Gemini results between 28 and 31 July 2026, flagged 898 of them as optimised for generative engines, and found that the off-the-shelf detectors it tested were partly reading who wrote a page rather than whether anyone had tuned it.

/blog/geo-detector-flagged-8-9-percent-of-retrieved-pages.md from /blog/geo-detector-flagged-8-9-percent-of-retrieved-pages

Lantad reads the same JSON-LD twice with two different traversals, and on our stored Wikipedia capture one reading returns a perfect 1.0 while the other returns 0 out of 100. Both are right, and the gap is the honest answer to what structured data does for AI.

/blog/structured-data-measured-twice.md from /blog/structured-data-measured-twice

Google replaced Google-NotebookLM with Google-GeminiNotebook on 16 July 2026 and supports the old name only until August 2026. A group headed by a name no crawler answers to falls through to the wildcard, or to no rules at all, and nothing on your site reports it.

/blog/renamed-crawler-tokens-leave-stale-robots-rules.md from /blog/renamed-crawler-tokens-leave-stale-robots-rules

A canary token study of 20 websites, posted to arXiv on 13 May 2026, mapped 22 AI chatbots to the scrapers that fed them. After the sites were taken offline and after a robots.txt file disallowed every bot, 12 of the 18 measurable systems still returned their content, and 10 of those 18 had been fed by a search engine's crawler rather than by any AI crawler of the vendor's own.

/blog/robots-txt-block-did-not-stop-twelve-chatbots.md from /blog/robots-txt-block-did-not-stop-twelve-chatbots

A controlled experiment run over 40 days on university web logs reports fewer than 40 percent of AI assistant and AI search bots re-fetching robots.txt inside a 168 hour window, and 15 of 47 bots never fetching it at all during two of its experiments. RFC 9309 asks crawlers not to use a cached copy for more than 24 hours.

/blog/when-a-robots-txt-edit-reaches-a-crawler.md from /blog/when-a-robots-txt-edit-reaches-a-crawler

Lantad's stored fixture for a site that disallows the six AI crawlers capable of citing it scores 78.8 out of 100 and grades B, because access is weighted at 25 points and only 20 of them are lost. An audit of our own scale, read out of our own repository.

/blog/a-robots-txt-that-blocks-every-citation-crawler-still-grades-b.md from /blog/a-robots-txt-that-blocks-every-citation-crawler-still-grades-b

Google's sitemap documentation, carrying Last updated 2026-07-08 UTC, says it uses the lastmod value if it is consistently and verifiably accurate, and that it ignores changefreq and priority. The crawler documentation published by OpenAI, Anthropic and Perplexity, read on 8 August 2026, does not mention sitemaps at all. The mechanism that can settle whether a page changed lives in HTTP.

/blog/sitemap-lastmod-is-an-assertion.md from /blog/sitemap-lastmod-is-an-assertion

A paper revised on arXiv on 1 August 2026 asks whether an LLM agent honours a server's request to stand down. At the access door four frontier models recused at rates from 100 percent to 55 percent. Delivered mid task to two OpenAI models it stopped nobody, 0 of 40, while a later run of the same apparatus had Gemini 2.5 Flash stopping 16 of 20.

/blog/mid-task-stop-request-stopped-zero-of-forty.md from /blog/mid-task-stop-request-stopped-zero-of-forty

A critical survey posted to arXiv on 15 July 2026 reviewed 45 generative engine optimization studies and reported that none of them shows a stable, cross-platform causal effect on organic discoverability. Its own pipeline diagram puts crawling and indexing at stage two of seven, and its caption says far fewer studies look there.

/blog/geo-survey-45-studies-crawling-stage.md from /blog/geo-survey-45-studies-crawling-stage

A pre-launch expert evaluation of Evrópuvefur, a government-funded Icelandic service answering questions about the EU, compared a curated corpus against open web search across 449 answers. Naming roughly sixty trusted domains in the system prompt raised the share of citations landing on one of them from 12 percent to 21 percent, and the country's most widely used news source was never cited across 287 web-search answers.

/blog/trusted-domain-list-moved-citations-12-to-21-percent.md from /blog/trusted-domain-list-moved-citations-12-to-21-percent

Every per-crawler number you have ever seen, including ours, counts requests that said they were a crawler. Verifying the claim needs an IP check, and most tools do not do one.

/blog/a-user-agent-is-a-claim-not-an-identity.md from /blog/a-user-agent-is-a-claim-not-an-identity

A controlled study posted to arXiv on 2 May 2026 turned on both of Cloudflare's free AI bot controls and then ran seven AI browsing agents through three tasks on the authors' own site. Six finished. The one that was stopped is listed in Cloudflare's bot directory, which the authors read as self-identification rather than detection.

/blog/cloudflare-blocked-one-of-seven-browsing-agents.md from /blog/cloudflare-blocked-one-of-seven-browsing-agents

Ahrefs correlated search and brand metrics against brand mentions in ChatGPT, Google AI Mode and AI Overviews across 75,000 brands. YouTube mentions led at about 0.737 and number of site pages sat near 0.194. Lantad measured none of it, and the layer we do measure sits underneath the whole ranking.

/blog/ai-brand-visibility-75000-brands-top-signal-off-site.md from /blog/ai-brand-visibility-75000-brands-top-signal-off-site

Turning a user agent string into an AI crawler label is a matching order with named failure modes, and every one of them produces a confident wrong answer rather than an error.

/blog/ai-crawler-detection-is-an-ordering-problem.md from /blog/ai-crawler-detection-is-an-ordering-problem

Lantad requested /robots.txt once from each of 20 AI company hostnames on 7 September 2026. All 20 answered HTTP 200, five name at least one of the 15 AI crawler tokens this scanner evaluates, and one of those names is spelled with a non-breaking hyphen, which RFC 9309 does not allow in a product token, so the group refuses nothing.

/blog/ai-crawler-robots-txt-one-token-no-crawler-matches.md from /blog/ai-crawler-robots-txt-one-token-no-crawler-matches

Getting quoted in an AI Overview starts with Googlebot, and Googlebot is not one of the fifteen tokens in our crawler registry. An audit of our own blind spot, read line by line against our own source.

/blog/ai-overviews-and-the-crawler-we-do-not-model.md from /blog/ai-overviews-and-the-crawler-we-do-not-model

A difference in differences study posted to arXiv on 5 February 2026 and revised on 12 May 2026 compares 52,262 English Wikipedia articles against the same articles in four language editions the rollout had not reached, across 46,534,093 daily observations, and estimates that AI Overviews exposure cut daily views to the English versions by roughly 15 percent. Wikipedia is one of the sources those answers cite most.

/blog/ai-overviews-cut-wikipedia-traffic-15-percent.md from /blog/ai-overviews-cut-wikipedia-traffic-15-percent

A SIGIR 2026 study of 11,500 queries collected on 7 and 8 December 2025 found Google AI Overviews significantly less likely than Google Search to retrieve content from sites that block Google-Extended, the token Google documents as not affecting Search. Gemini never cited the study's 21 blocking publishers at all.

/blog/ai-overviews-retrieved-less-from-google-extended-blockers.md from /blog/ai-overviews-retrieved-less-from-google-extended-blockers

A joint study by Press Ranger and OtterlyAI, announced on 20 August 2026, cross-referenced 129.3 million citations captured across seven AI platforms during June 2026 against 91 confirmed licensing agreements mapped to 314 publisher domains. Pages from publishers with an OpenAI deal collected 10.2 citations each on average on ChatGPT against 6.9 for pages from unlicensed publishers. The same pattern did not appear for Google or Perplexity deals on their own platforms.

/blog/ai-publisher-licensing-deals-10-2-citations-against-6-9.md from /blog/ai-publisher-licensing-deals-10-2-citations-against-6-9

Both IETF AI preferences working group drafts carry the date 19 August 2026. The vocabulary deleted the rule that let a preference stated for a broad usage category cover a narrower one, so an unstated category is now unknown and nothing else, and the companion draft defining the robots.txt Content-Usage rule is current again after 110 days behind an expiry.

/blog/ai-usage-preferences-stopped-inheriting.md from /blog/ai-usage-preferences-stopped-inheriting

A study from the University of St. Gallen posted to arXiv on 8 April 2026 queried four AI search engines with 32 prompts over 45 to 46 days, then repeated each prompt up to 10 times inside one day. A per-brand visibility rate estimated from a single run carried a standard error of 0.370, and seven runs were needed to bring it under 0.10.

/blog/ai-visibility-took-seven-runs-per-prompt-to-settle.md from /blog/ai-visibility-took-seven-runs-per-prompt-to-settle

Read out of this repository on 29 August 2026, the answer engine registry Lantad ships carries 8 engines. Five are recorded as performing live retrieval and returning the sources they grounded in. Three are recorded as having no web access and no sources at all, and one of those three is the only engine a free run asks.

/blog/ai-visibility-tracking-three-of-eight-engines-cannot-search.md from /blog/ai-visibility-tracking-three-of-eight-engines-cannot-search

An IETF Independent Submission dated 12 June 2026 asks IANA to register /.well-known/ai.txt and /.well-known/ai.json. It defines 22 field names, and a file that never mentions training has already refused it.

/blog/ai-txt-well-known-training-defaults-to-deny.md from /blog/ai-txt-well-known-training-defaults-to-deny

A controlled experiment posted to arXiv on 13 July 2026 put three browser agents through five tasks on two versions of one prototype storefront, 300 runs in total. The agent-ready version recorded 134 passes out of 150 against 74 out of 150, and both versions were driven by a real browser, so nothing the agents needed was hidden from them.

/blog/agent-ready-storefront-task-success.md from /blog/agent-ready-storefront-task-success

The IETF vocabulary for AI usage preferences defines two categories and two values, y and n, and offers no way to scope either to a named company. Two individual submissions filed on 6 August 2026 propose two different places to put the missing permission.

/blog/ai-usage-preferences-cannot-name-a-party.md from /blog/ai-usage-preferences-cannot-name-a-party

The raw HTML of five pages this scanner captured on 15 July 2026 holds 245 heading elements and our extractor recorded 166, because it discards every heading inside nav, header, footer or aside. On two of those pages the rule turned a real h1 into a reported count of zero.

/blog/an-h1-inside-a-header-counted-as-none.md from /blog/an-h1-inside-a-header-counted-as-none

draft-reilly-aipref-compliance-00, an individual submission dated 2 August 2026, defines a signed record binding a fetched asset, the preference in force when it was fetched, and the usage category the AI system assigned to it. Its own abstract says the mechanism is confined to evidence and takes no position on the legal effect of any preference.

/blog/compliance-records-are-falsifiable-not-proof.md from /blog/compliance-records-are-falsifiable-not-proof

Two different products are sold as AI visibility tools: one asks models a set of prompts and counts what the answers name, the other checks whether a crawler can fetch and read the pages at all. Published plans read on 1 September 2026 show the prompt allowances are not even the same unit.

/blog/best-ai-visibility-tools-two-machines-one-category.md from /blog/best-ai-visibility-tools-two-machines-one-category

Cloudflare's bots changelog entry of 1 July 2026 states that from 15 September 2026 new domains onboarding to Cloudflare receive defaults under which bots classified as Training or as Agent are blocked on pages that display ads, while Search remains allowed. That is a rule about a page, and robots.txt has no way to write one.

/blog/block-ai-crawlers-cloudflare-default-turns-on-ads.md from /blog/block-ai-crawlers-cloudflare-default-turns-on-ads

A staggered difference in differences study of news publishers, arXiv 2512.24968, estimates roughly a 7 percent traffic decline within 6 weeks of first disallowing a generative AI crawler. Split by publisher rank, the decline sits in the top 50 and the point estimate turns slightly positive below rank 100.

/blog/blocking-ai-crawlers-cost-7-percent-of-traffic.md from /blog/blocking-ai-crawlers-cost-7-percent-of-traffic

Lantad requested /robots.txt from 40 well known hostnames on 5 September 2026, twice each, and evaluated GPTBot, OAI-SearchBot and ChatGPT-User against the site root with its shipped matcher. 29 served a parseable file. On 25 of them the three agents received the same verdict, and on 4 they did not.

/blog/can-chatgpt-read-my-website-agent-changed-the-answer.md from /blog/can-chatgpt-read-my-website-agent-changed-the-answer

Cloudflare turns a non-self-referencing canonical tag into a 301 for verified AI training crawlers. On five pages this scanner captured on 15 July 2026, all five canonical tags pointed at the page's own address, so the feature would have redirected none of them, and two of the five were not byte-identical to the URL the page was served at.

/blog/canonical-tags-and-ai-crawlers-five-of-five-self-referential.md from /blog/canonical-tags-and-ai-crawlers-five-of-five-self-referential

OpenAI's product specification, read on 2 September 2026, marks 16 fields Required. Twelve apply to every non-Ads feed, and four of those twelve are settings a merchant registers rather than facts a page publishes. The file is pushed to OpenAI over SFTP, so no crawler collects any of it.

/blog/chatgpt-product-feed-twelve-required-fields.md from /blog/chatgpt-product-feed-twelve-required-fields

The European Commission designated ChatGPT a Very Large Online Search Engine under the Digital Services Act on 31 August 2026. The obligations that follow are about systemic risk, audits and public reports, and the first of them lands four months after notification. None of them tells a site owner why one page was cited and another was not.

/blog/chatgpt-seo-eu-designated-a-search-engine.md from /blog/chatgpt-seo-eu-designated-a-search-engine

Fetched on 2 September 2026 across the 14 hostnames belonging to the six real sites this scanner captured on 15 July 2026, one request per hostname returned eight distinct files. On three of the six sites the second hostname did not serve the file that governs the captured page, and on one of those it returned nothing at all.

/blog/check-for-robots-txt-fourteen-hostnames-eight-files.md from /blog/check-for-robots-txt-fourteen-hostnames-eight-files

Chrome for Developers documents an agentic browsing category in Lighthouse whose Stability and Discoverability group checks for a machine-readable summary at the domain root. Google Search Central's guide to generative AI features, carrying Last updated 2026-07-10 UTC, states Google Search ignores such files. Both pages are Google's, and they describe different consumers.

/blog/lighthouse-agentic-browsing-checks-llms-txt.md from /blog/lighthouse-agentic-browsing-checks-llms-txt

Similarweb reports the presence of citations in US ChatGPT prompts rising from about 1.6 percent in June 2025 to roughly 6.8 percent by May 2026, and separately reports the homepage share of ChatGPT referrals jumping to about 60 percent in the week after 7 May 2026. The page that earns a citation and the page that receives the visit are frequently not the same page.

/blog/chatgpt-citations-are-rare-and-land-on-the-homepage.md from /blog/chatgpt-citations-are-rare-and-land-on-the-homepage

Cloudflare's verified bots policy names an AI crawler ignoring the robots.txt crawl-delay directive as an example of a policy breach, and its post of 7 August 2026 makes verified status revocable for conduct. RFC 9309 does not mention crawl-delay, and Google's own robots.txt specification says it is not supported.

/blog/crawl-delay-can-unverify-a-crawler.md from /blog/crawl-delay-can-unverify-a-crawler

Cloudflare announced on 1 July 2026 that it is shaping pay per crawl into pay per use, with experiments running alongside Ceramic.ai and You.com. A crawl arrives as a request in your own access log. A use happens inside an answer engine, where the site that produced the page has nothing to count.

/blog/cloudflare-pay-per-use-crawl-countable-use-not.md from /blog/cloudflare-pay-per-use-crawl-countable-use-not

Cloudflare opened a developer preview on 6 August 2026 that adds a WebMCP tool interface to any site it fronts, using HTMLRewriter to insert one script reference into each HTML response. Chrome's own WebMCP documentation, last updated 2026-06-09, files tool discoverability under its limitations: clients and browsers must visit a site directly to know if it has callable tools.

/blog/webmcp-tools-need-a-browser-to-be-found.md from /blog/webmcp-tools-need-a-browser-to-be-found

Cloudflare published Kitesurf on 6 August 2026, a browser built for AI agents that runs in V8 isolates on Workers rather than as a Chromium process. Its own documentation reports DOM subtests passing at 97 percent and lists a bot-challenge handshake with real TLS fingerprints among the things it cannot do.

/blog/cloudflare-agent-browser-is-not-chromium.md from /blog/cloudflare-agent-browser-is-not-chromium

Cloudflare announced Bot Preference Sync on 21 August 2026: a setting on every plan that writes your AI bot policy for Search, Agent and Training into robots.txt, prepends it above whatever file you already have, and refreshes the crawler names inside it from Cloudflare's own bot database. RFC 9309 says a crawler merges every group matching its token and reads the wildcard group only when none matches, so a prepended group changes which rules a named crawler obeys.

/blog/cloudflare-generates-robots-txt-and-the-token-list-moves.md from /blog/cloudflare-generates-robots-txt-and-the-token-list-moves

Cloudflare announced Agent Readiness diagnostics and an Answer Engine Optimization tab on 6 August 2026. The post sorts its site checks into quick wins, technical groundwork, advanced integration and commerce, and states that the commerce group is informational for now and not counted in your score.

/blog/agent-readiness-score-counts-three-of-four-groups.md from /blog/agent-readiness-score-counts-three-of-four-groups

Cloudflare's managed robots.txt writes one Content-signal line and eight Disallow rules into a site's file. We parsed the published example on 30 July 2026: the signal was recorded as an unknown field and ignored, and the Disallow rules blocked 8 of the 15 crawler tokens in our registry.

/blog/content-signal-asks-disallow-blocks.md from /blog/content-signal-asks-disallow-blocks

Common Crawl published CC-MAIN-2026-30 on 28 July 2026, 2.14 billion pages collected between 7 and 25 July. Its own FAQ states the crawler executes no JavaScript, so what a client rendered site contributed to that archive is whatever its server returned.

/blog/common-crawl-july-archive-no-javascript.md from /blog/common-crawl-july-archive-no-javascript

A paper from three Common Crawl Foundation authors, posted to arXiv on 15 July 2026, fits 51 monthly crawls from 2020 to 2025 and reports a persistent core fraction of about 0.4 at domain granularity, with the rest of the domain population churning through a shell.

/blog/common-crawl-core-and-shell-persistence.md from /blog/common-crawl-core-and-shell-persistence

Cloudflare's changelog of 31 August 2026 records that the Browser Run crawl endpoint now respects the Content Signals use directive and rejects a job with HTTP 400 when the caller declares a more permissive level than the site does. We opened all ten Browser Run quick action documentation pages on 3 September 2026: nine of them name neither Content Signals nor robots.txt.

/blog/content-signals-robots-txt-one-of-ten-endpoints.md from /blog/content-signals-robots-txt-one-of-ten-endpoints

Google's crawl budget guide, carrying Last updated 2026-07-22 UTC, tells site owners to block crawling of low value URLs with robots.txt. The guide is scoped to Googlebot and names no AI crawler. The file it tells you to edit is scoped to nobody in particular, and RFC 9309 says a crawler with no group of its own must obey the wildcard.

/blog/crawl-budget-advice-lands-on-ai-crawlers.md from /blog/crawl-budget-advice-lands-on-ai-crawlers

The crawl to refer ratio divides a platform's crawler requests for HTML by the HTML requests whose Referer header names that platform. Cloudflare, which publishes it, states that traffic from Claude's native app carries no Referer header and that the calculations may therefore overstate the ratios, by an amount it says is unclear.

/blog/crawl-to-refer-ratio-excludes-native-app-referrals.md from /blog/crawl-to-refer-ratio-excludes-native-app-referrals

Thirteen published crawler IP range files from Google, OpenAI, Apple, Perplexity and Microsoft, read on 19 August 2026, each answer the question of when they last changed twice, and on six the two answers differ by between 13 and 942 days.

/blog/crawler-ip-files-carry-two-dates-that-disagree.md from /blog/crawler-ip-files-carry-two-dates-that-disagree

A source attribution study posted to arXiv on 7 May 2026 by researchers at PricewaterhouseCoopers benchmarked 14 language models on 130 research queries. Twelve of the 14 kept cited links working above 94 percent of the time, while factual accuracy against those same sources ran from 24.4 to 76.8 percent. The dimension a publisher can change was the one already near its ceiling.

/blog/deep-research-links-worked-facts-did-not.md from /blog/deep-research-links-worked-facts-did-not

All five real pages this scanner captured on 15 July 2026 carry exactly one main element. Rewriting every one of them as a plain div and re-running the shipped extractor on 26 August 2026 cut the main content corpus from 13,615 words to 11,950. On one page the landmark was worth 43 percent of the corpus and on another it was worth 1 percent.

/blog/deleting-main-cost-1665-of-13615-words.md from /blog/deleting-main-cost-1665-of-13615-words

Lantad fetched the nine vendor documentation pages behind the fifteen AI crawler tokens it evaluates, on 5 September 2026, and read the prose out of each with its own extractor. Two of the nine state whether their crawler executes JavaScript. Five are silent. One is a client rendered shell that returns no prose at all without the JavaScript in question.

/blog/do-ai-crawlers-render-javascript-two-of-nine.md from /blog/do-ai-crawlers-render-javascript-two-of-nine

A controlled study of ten consumer AI assistants, posted to arXiv on 16 July 2026, logged 200 retrieval trials against two purpose-built domains. Six of the ten assistants never requested robots.txt in any trial, and four of those six returned correct answers taken from pages the file disallowed.

/blog/do-ai-crawlers-respect-robots-txt-six-of-ten-never-asked.md from /blog/do-ai-crawlers-respect-robots-txt-six-of-ten-never-asked

Lantad read the nine vendor documentation pages behind the fifteen AI crawler tokens it evaluates on 9 September 2026. The word noindex appears in the bytes of two of them, Apple's and Amazon's, and on both it governs indexing rather than the fetch.

/blog/does-noindex-block-ai-crawlers-two-of-nine.md from /blog/does-noindex-block-ai-crawlers-two-of-nine

draft-hood-aipref-earmark-00, an individual submission dated 12 August 2026, embeds a signed usage preference in the content itself so the preference survives copying, syndication and aggregation. Its own text states that a Mark does not prove copyright ownership, that Earmarks do not make preferences self-enforcing, and that Earmark prevents no use. It names the failure it is answering: a preference written at the perimeter attaches to the location rather than to the content.

/blog/earmark-signs-the-preference-into-the-content.md from /blog/earmark-signs-the-preference-into-the-content

A public URL box is a server-side request forgery machine unless something stops it. Lantad's guard names eight rejections, refuses 14 IPv4 ranges and applies eight IPv6 rules, re-runs in full on every redirect hop, and carries one gap we document rather than close.

/blog/eight-reasons-a-scanner-refuses-a-url.md from /blog/eight-reasons-a-scanner-refuses-a-url

A study posted to arXiv on 29 June 2026 pointed six LLM web agents, three scrapers and three browser automation frameworks at ten honeysites carrying seven anti-bot defences. Combining network, TLS and browser fingerprints classified the client at 0.993 accuracy. The robots.txt honeysite blocked exactly one of the twelve tools, and it was the one whose framework obeys robots.txt by default.

/blog/web-agents-identifiable-robots-txt-stopped-one.md from /blog/web-agents-identifiable-robots-txt-stopped-one

We re-measured the five structured-data signals behind Entity Confidence against 30 pages fetched in one day, removed two of them, and added two nobody was checking.

/blog/five-signals-that-tell-ai-who-you-are.md from /blog/five-signals-that-tell-ai-who-you-are

A controlled study posted to arXiv on 25 May 2026 ran 252,000 trials across six language models, showing each of them two competing sources that differed in exactly one of 18 content factors. Four factors moved the first citation in every model, one of them was list position, and the two formatting factors are grouped by the paper among the seven with no consistent effect.

/blog/four-factors-decided-the-first-citation.md from /blog/four-factors-decided-the-first-citation

OpenAI publishes a JSON file of IP address ranges for each of its four crawlers. Read on 29 July 2026, the ChatGPT-User file covered 37,328 IPv4 addresses against GPTBot's 3,472, and it is the one crawler OpenAI says robots.txt rules may not apply to.

/blog/openai-crawler-ip-ranges-counted.md from /blog/openai-crawler-ip-ranges-counted

Google's crawling documentation sorts everything it sends at a website into three classes. Nine user-triggered fetchers generally ignore robots.txt, and five special-case crawlers ignore the global user agent group, so fourteen named Google agents are not decided by the group you head with an asterisk.

/blog/google-fetchers-ignore-the-robots-txt-wildcard.md from /blog/google-fetchers-ignore-the-robots-txt-wildcard

Counted with an HTML parser on 25 August 2026 over five real pages this scanner captured and stored on 15 July 2026, the initial HTTP responses hold 14 noscript elements. Between them they carry 11 img tags, one tag manager iframe, one CSS rule of 57 characters and zero characters of text. The one element HTML defines for a reader that does not run scripts is the one place on these pages where nobody wrote a sentence.

/blog/fourteen-noscript-elements-held-no-text.md from /blog/fourteen-noscript-elements-held-no-text

Lantad requested the home page of 392 hostnames on 11 September 2026 and found 36 served by Framer. Asked with an Accept header naming text/markdown, 35 of the 36 returned a markdown document. Asked for robots.txt, 32 of the 33 that answered HTTP 200 returned the same four line file, and one of the 33 named an AI crawler token at all.

/blog/framer-ai-crawlers-35-of-36-served-markdown.md from /blog/framer-ai-crawlers-35-of-36-served-markdown

Google's new channel files a visit under AI Assistant when the referrer matches a list it does not publish. Lantad fetched the six assistant hosts in its own referrer list on 9 September 2026: two returned a readable page, and both declare a policy that sends the host and discards the path.

/blog/ga4-ai-assistant-channel-two-of-six-hosts.md from /blog/ga4-ai-assistant-channel-two-of-six-hosts

A paper posted to arXiv on 31 March 2026 reports that editing document structure alone raised citation rate across six generative engines. Its own table puts the move at 45.0 percent to 52.8 percent, and the ablation attributes 84.6 percent of that gain to heading hierarchy and chunking rather than to emphasis.

/blog/generative-engine-optimization-structure-7-8-points.md from /blog/generative-engine-optimization-structure-7-8-points

A study published on 28 July 2026 by the AI visibility vendor Goodie, covering 31 million AI citations recorded between October 2025 and July 2026, puts the largest share of news citations on Grok at 23 percent, ahead of AI Overviews at 19.5 and Gemini at 13.5. Lantad's own answer engine registry, read on 7 September 2026, asks Grok with no retrieval and reads no citations back from it.

/blog/geo-engine-grok-23-percent-of-news-citations.md from /blog/geo-engine-grok-23-percent-of-news-citations

A paper posted to arXiv on 11 August 2026 by researchers at Carnegie Mellon and UC San Diego ran an automated generative engine optimization tool against a defended answer engine for five consecutive rounds, on three benchmarks and three model backends. In the worked example the rewritten document accumulated four to six unsupported claims, and the strategy the platform read back out of the rewrites moved from formatting advice at round one to hidden GEO intent at round five.

/blog/geo-rewriting-degraded-the-document-over-five-rounds.md from /blog/geo-rewriting-degraded-the-document-over-five-rounds

Both disciplines depend on one HTTP fetch governed by one file. The measured divergences are narrow: a Washington University team found nearly 30 percent of AI Overview cited domains rank nowhere on page one, a 252,000 trial experiment put formatting near zero, and Google's own guide says SEO best practices still apply because its generative features run on its core ranking systems.

/blog/geo-vs-seo-same-crawl-four-differences.md from /blog/geo-vs-seo-same-crawl-four-differences

Google's guidance on Search-related JavaScript problems, carrying Last updated 2025-12-18 UTC, states under the heading Don't rely on data persistence to serve content that its Web Rendering Service does not retain state across page loads, and that Local Storage, Session Storage and HTTP Cookies are cleared across page loads. A gate that waits for a stored consent decision is therefore waiting on something the crawler discards, and the crawler pages published by OpenAI, Anthropic and Perplexity, read on 13 August 2026, contain the word cookie zero times between them.

/blog/consent-gates-never-open-for-a-stateless-crawler.md from /blog/consent-gates-never-open-for-a-stateless-crawler

Google's snippet documentation, carrying Last updated 2026-04-20 UTC, describes a read more deep link that lands a searcher on a specific section of a page, and names removing the hash fragment from the URL as something that breaks it. Counted with an HTML parser over five real pages this scanner captured on 15 July 2026, 55 of 245 headings carried an id of their own, and 26 headings could not be reached by any element identifier at all.

/blog/read-more-deep-links-and-heading-ids.md from /blog/read-more-deep-links-and-heading-ids

Google's documentation changelog records the FAQ rich result leaving Search on 7 May 2026 and its documentation being removed on 15 June 2026. Read on 5 August 2026 the old URL answers 301, schema.org still defines FAQPage, and Lantad's schema check still counts it as a recognised business type.

/blog/faq-rich-result-gone-faqpage-still-scored.md from /blog/faq-rich-result-gone-faqpage-still-scored

Google's lazy-loading documentation, carrying Last updated 2025-12-10 UTC, tells site owners to load content when it enters the viewport rather than when somebody scrolls, and gives the reason in one clause: Google Search does not interact with your page. The crawler pages published by OpenAI, Anthropic and Perplexity, fetched and searched on 14 August 2026, contain the words JavaScript, render, scroll, viewport and lazy zero times between them.

/blog/lazy-loading-and-crawlers-that-do-not-scroll.md from /blog/lazy-loading-and-crawlers-that-do-not-scroll

Google's file types documentation, carrying Last updated 2026-02-03 UTC, lists nine flat and eleven encoded formats it can index and states that the Content-Type header decides which parser runs. The crawler documentation published by OpenAI, Anthropic and Perplexity, read on 19 August 2026, names no file type at all, and this scanner returns no score for a URL that is not HTML.

/blog/google-indexes-twenty-file-types-ai-docs-name-none.md from /blog/google-indexes-twenty-file-types-ai-docs-name-none

Google Search Central updated its preferred sources documentation on 20 August 2026 to add an interactive button that loads a script from news.google.com, and named a plain URL alternative for sites that cannot run third party scripts.

/blog/preferred-sources-button-and-plain-url.md from /blog/preferred-sources-button-and-plain-url

Google Search Central publishes an official guide to appearing in AI Overviews and AI Mode, and a section of it headed Mythbusting lists llms.txt, chunking, rewriting for AI, inauthentic mentions and structured data among the things you do not need to do.

/blog/google-names-five-geo-tactics-to-ignore.md from /blog/google-names-five-geo-tactics-to-ignore

Google's paywalled content documentation, last updated 10 December 2025, tells a publisher to let Googlebot and Googlebot-News reach the paywalled text so the markup can show the gap is not cloaking. Read on 4 August 2026, those two names are the only crawler names on the page.

/blog/paywall-markup-names-two-google-crawlers.md from /blog/paywall-markup-names-two-google-crawlers

Google's review snippet documentation, carrying Last updated 2026-07-24 UTC, tells site owners not to include fake or undisclosed incentivized reviews in the page or in the structured data markup. Of the nine properties schema.org defines specifically on Review, none records whether the reviewer was paid. The rule is about provenance and every automated check reads shape.

/blog/review-authenticity-is-not-in-the-markup.md from /blog/review-authenticity-is-not-in-the-markup

Google's crawler documentation, read on 10 August 2026, states that Googlebot crawls the first 2MB of a supported file type, stops the fetch at the cutoff, and applies the limit to uncompressed data. The crawler documentation published by OpenAI, Anthropic and Perplexity states no size limit at all, which leaves site owners with a documented ceiling from one vendor and silence from three.

/blog/googlebot-reads-the-first-two-megabytes.md from /blog/googlebot-reads-the-first-two-megabytes

Google's locale-adaptive pages documentation, carrying Last updated 2025-12-10 UTC, states that the crawler sends HTTP requests without setting Accept-Language and that its default IP addresses appear to be based in the USA. Read on 13 August 2026, the crawler pages published by OpenAI, Anthropic and Perplexity name no language, locale or region behaviour at all, so a site that chooses a language from the request has nothing published to plan against.

/blog/googlebot-sets-no-accept-language.md from /blog/googlebot-sets-no-accept-language

A robots.txt product token and an HTTP User-Agent header are two different strings, and only the second one is what your access log holds. Read at nine vendors' own documentation on 27 August 2026, 6 of the 15 tokens in Lantad's crawler registry carry a complete User-Agent string that can be matched literally.

/blog/gptbot-detection-six-of-fifteen-tokens-publish-a-ua.md from /blog/gptbot-detection-six-of-fifteen-tokens-publish-a-ua

The 2025 Web Almanac, written from HTTP Archive's July 2025 crawl of 16,213,084 websites and published on 15 January 2026, reports the catch-all user agent in 77.04 percent of desktop robots.txt files and gptbot in 4.5 percent. Google's advertising crawler is named more than twice as often as any AI crawler.

/blog/robots-txt-names-the-wildcard-not-the-ai-crawler.md from /blog/robots-txt-names-the-wildcard-not-the-ai-crawler

Three researchers at the University of Bamberg loaded 10,000 homepages in four browser configurations and recorded who got refused. Headless Chromium drew a 15.2 percent soft block rate against 7.2 percent for the same browser with a window, and removing one string from two headers unblocked 75 percent of the difference.

/blog/headless-chromium-blocked-on-15-percent-of-sites.md from /blog/headless-chromium-blocked-on-15-percent-of-sites

A controlled study posted to arXiv on 15 June 2026 ran five attack modes against 13 LLM backends, 308 cases each. Hidden instructions planted in the machine-readable page averaged 0.3 percent success. Three fabricated sources agreeing with each other averaged 24.8 percent.

/blog/hidden-text-was-the-weakest-attack-on-ai-search.md from /blog/hidden-text-was-the-weakest-attack-on-ai-search

Perplexity documents two named crawlers with different robots.txt behaviour, and its Sonar Chat Completions endpoint is deprecated with support ending on 27 September 2026 in favour of the Agent API.

/blog/how-do-i-get-cited-by-perplexity-two-crawlers.md from /blog/how-do-i-get-cited-by-perplexity-two-crawlers

A study of 374,052 citations across Perplexity, OpenAI SearchGPT and Google Gemini found repeated runs of the same query share as little as 0.29 of their cited domains, and that a confidence interval five percentage points wide on citation share needs roughly 40 queries on one platform and 150 or more on another.

/blog/how-many-prompts-an-ai-visibility-measurement-needs.md from /blog/how-many-prompts-an-ai-visibility-measurement-needs

Anthropic's help centre article of 7 April 2026 names ClaudeBot, Claude-User and Claude-SearchBot, and says that disabling either of the last two may reduce your site's visibility. Lantad fetched the IP range file that page points at on 7 September 2026. It holds 26 prefixes covering 1,094 IPv4 addresses, and it is a single list for all three bots.

/blog/how-to-get-cited-by-claude-one-ip-list-three-bots.md from /blog/how-to-get-cited-by-claude-one-ip-list-three-bots

Google documents 11 common crawlers and none of them is a Gemini crawler. The only entry that names Gemini is Google-Extended, which sends no request of its own, and Google's page of 14 July 2026 says the same token governs training and grounding together.

/blog/how-to-get-cited-by-gemini-one-token-two-systems.md from /blog/how-to-get-cited-by-gemini-one-token-two-systems

Google publishes one hard eligibility rule for a supporting link in AI Overviews, which is that the page must be indexed and eligible to be shown in Search with a snippet. Lantad requested the home page of 120 real hostnames on 6 September 2026. Of the 81 that answered HTTP 200, none carried a nosnippet directive, three carried max-snippet and all three set it to -1, and 48 carried no robots meta tag at all.

/blog/how-to-get-content-cited-in-google-ai-overviews-zero-nosnippet.md from /blog/how-to-get-content-cited-in-google-ai-overviews-zero-nosnippet

A paper posted to arXiv on 24 July 2026 measured where a conversation's request state actually sits across its user turns. In 670 commercial multi-turn conversations the final prompt carried a median 35.6 percent of the session's unique user-side content vocabulary, and in 50.3 percent of them a rule set detected at least one request-state dimension in the history that the final prompt never repeats.

/blog/how-to-measure-geo-final-prompt-held-35-6-percent.md from /blog/how-to-measure-geo-final-prompt-held-35-6-percent

IndexNow lets a site push changed URLs to search engines instead of waiting to be crawled. Read at indexnow.org on 12 August 2026, its supporter list names Microsoft Bing, Naver, Seznam.cz, Yandex and Yep, and its own documentation states that an HTTP 200 indicates only that the search engine has received your URL. The protocol moves the moment a crawler learns a page changed. It changes nothing about what that crawler receives when it arrives.

/blog/indexnow-speeds-discovery-not-readability.md from /blog/indexnow-speeds-discovery-not-readability

Counted with an HTML parser over five real pages this scanner captured and stored on 15 July 2026, the raw HTTP response held 991 distinct same-host URLs and the rendered DOM held 982. Three URLs existed only after JavaScript ran, and every one of the three was a rewrite of a link already present. Nine of the twelve that disappeared were one page's translations of itself.

/blog/javascript-added-no-new-crawl-paths.md from /blog/javascript-added-no-new-crawl-paths

Cloudflare's rebuilt bot submission form, launched on 28 August 2026, asks an operator to declare what its bot does, how it uses what it reads, and who runs it. The automated review that follows fetches an IP list, confirms reverse DNS or validates a Web Bot Auth signature, so the one claim a machine settles is identity, and the three that describe intent are taken on the operator's word.

/blog/llm-crawler-detection-declared-three-verified-one.md from /blog/llm-crawler-detection-declared-three-verified-one

Lantad requested /llms.txt once from each of 40 hostnames on 5 September 2026, chosen because they were the sites most likely to publish one. 31 returned a plain text file, 7 returned HTTP 404, and 2 returned HTTP 200 with an HTML document. The 31 files hold 5,567 markdown links between them, and every one of the 75 links in the file at docs.mistral.ai returned HTTP 404.

/blog/llms-txt-example-31-files-and-75-dead-links.md from /blog/llms-txt-example-31-files-and-75-dead-links

The llms.txt proposal was revised to version 2 and the page now carries a modified date of 10 August 2026. The change its author names first is discoverability: two standard link relations, carried in an HTML link element or an HTTP Link header, so an agent can find a page's markdown version and the llms.txt covering it without guessing. The context expansion tool that version 1 described as the consumer of these files is no longer part of the proposal.

/blog/llms-txt-v2-link-relations-and-discovery.md from /blog/llms-txt-v2-link-relations-and-discovery

Lantad requested /robots.txt and /llms.txt once each from 200 hostnames on 7 September 2026. 145 returned a parseable robots.txt and 23 returned an llms.txt as plain text. Of those 23, 19 name none of the 15 AI crawler tokens this scanner evaluates anywhere in their robots.txt.

/blog/llms-txt-vs-robots-txt-19-of-23.md from /blog/llms-txt-vs-robots-txt-19-of-23

The Model Context Protocol revision of 28 July 2026 requires every HTTP POST to an MCP endpoint to name its protocol version, its method and the resource it is reading. Those three headers describe one hop, and the site whose content is being read is not on it.

/blog/mcp-labels-stop-at-the-mcp-server.md from /blog/mcp-labels-stop-at-the-mcp-server

Meta's web crawler documentation, read on 8 August 2026, names facebookexternalhit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher. Lantad's crawler registry carries one Meta token, and a post we published on 1 August 2026 stated that Meta has no second published name to allow separately. The name most worth having is Meta-WebIndexer, which Meta describes as serving Meta AI search.

/blog/meta-documents-five-crawler-tokens.md from /blog/meta-documents-five-crawler-tokens

DataDome's press release of 16 July 2026 states that its network processed 17.7 billion AI agent requests between April and June, up from 12.2 billion in the previous quarter, with Meta's two agents taking the majority of that traffic. In the same release, ChatGPT-User's absolute request volume fell 6 percent while ChatGPT held 80 to 88 percent of all AI referrals every month. Crawl volume and referral value are not the same population, and the tokens are not interchangeable.

/blog/meta-took-the-crawl-volume-chatgpt-kept-the-referrals.md from /blog/meta-took-the-crawl-volume-chatgpt-kept-the-referrals

Microsoft's AI Performance report in Bing Webmaster Tools, announced on 10 February 2026 and extended on 16 June 2026, shows the grounding queries behind citations in Microsoft Copilot. Google's generative AI performance report, read at its help documentation on 14 August 2026, publishes impressions grouped by page, country, date and device, and no query at all.

/blog/microsoft-copilot-seo-bing-names-the-grounding-query.md from /blog/microsoft-copilot-seo-bing-names-the-grounding-query

Every other major vendor gave site owners a robots.txt name to disallow. Microsoft's published guidance for keeping a page out of its AI answers is NOCACHE or NOARCHIVE in a meta robots tag, which is why none of the 15 tokens in our crawler registry belongs to Microsoft.

/blog/microsoft-ai-opt-out-is-a-meta-tag.md from /blog/microsoft-ai-opt-out-is-a-meta-tag

A measurement framework posted to arXiv on 29 April 2026 scores how much of each cited page reaches the answer text. Across 21,143 valid search-layer citations, Perplexity averaged 16.35 citations per prompt to ChatGPT's 6.88, and mean influence among fetched pages ran the other way, 0.0646 against 0.2713.

/blog/citation-count-is-not-answer-influence.md from /blog/citation-count-is-not-answer-influence

A Washington University in St. Louis team issued 55,393 trending queries over 40 days and found that 29.8 percent of the domains an AI Overview cites appear nowhere on the first page of results for the same query.

/blog/ai-overview-citations-are-not-page-one.md from /blog/ai-overview-citations-are-not-page-one

Lantad's prompt tracker reported a 12.6 percent brand mention rate on its first live run on 13 August 2026. The truth was near 1 percent: of the 19 counted mentions, eight were an engine defining a Tagalog adjective, three resolved the name to a different business, and six were engines saying they had never heard of it. A name in an answer is presence, and presence is not identity.

/blog/nineteen-counted-mentions-about-two-were-the-company.md from /blog/nineteen-counted-mentions-about-two-were-the-company

Microsoft introduced NLWeb on 19 May 2025 as a way to reuse the Schema.org and RSS a site already publishes, and stated that every instance is also an MCP server. Cloudflare's deployment page for it, last updated 10 July 2026, documents an /ask and an /mcp endpoint and instructs the reader to ingest pages by crawling them.

/blog/nlweb-ask-endpoint-schema-org-data-model.md from /blog/nlweb-ask-endpoint-schema-org-data-model

Lantad requested /robots.txt once from each of 170 hostnames on 8 September 2026. 140 answered with a file that parsed as text. 25 of those name GPTBot, 16 name OAI-SearchBot, and 11 name GPTBot without ever naming the crawler OpenAI's documentation ties to appearing in ChatGPT search answers.

/blog/oai-searchbot-16-of-140-files-named-it.md from /blog/oai-searchbot-16-of-140-files-named-it

Schema.org began publishing domain counts for its whole vocabulary on 4 June 2026. Counting the July 2026 file on 31 July 2026 gives 958 types and 4,587 properties, of which 4,217 sit in the lowest bucket of fewer than 1,000 domains and 16 types reach the top bucket of more than 10 million.

/blog/sixteen-schema-types-reach-ten-million-domains.md from /blog/sixteen-schema-types-reach-ten-million-domains

Cloudflare's pay per crawl answers an unpaid AI crawler with HTTP 402 Payment Required and a crawler-price header naming the amount. RFC 9110 defines that status code in one sentence: it is reserved for future use.

/blog/pay-per-crawl-runs-on-an-undefined-status-code.md from /blog/pay-per-crawl-runs-on-an-undefined-status-code

HTTP defines a response header that lets an overloaded server tell a client exactly when to retry. Read on 14 August 2026, six crawler documentation pages published by Google, OpenAI, Anthropic and Perplexity describe what a server error does to crawling and none of the six names Retry-After anywhere.

/blog/no-crawler-vendor-documents-retry-after.md from /blog/no-crawler-vendor-documents-retry-after

Read out of the golden fixtures this scanner captured on 15 July 2026, robots.txt permitted every one of the 60 user agent probes that ran against five real pages. Four of those probes came back HTTP 403, and seven came back HTTP 200 carrying visible text that matched the honest baseline only 87 percent.

/blog/robots-txt-allowed-sixty-probes-eleven-differed.md from /blog/robots-txt-allowed-sixty-probes-eleven-differed

Lantad requested /robots.txt once from each of 392 hostnames on 10 September 2026 with redirect following switched off. 391 answered, 122 of them with a 3xx, and every one of those 122 Location headers pointed at a hostname other than the one asked. RFC 9309 says to follow such a redirect and apply the file it reaches to the original authority.

/blog/robots-txt-redirect-122-of-391-crossed-hosts.md from /blog/robots-txt-redirect-122-of-391-crossed-hosts

One GET per hostname on 5 September 2026 for the robots.txt of the six real sites this scanner captured on 15 July 2026. All six answered HTTP 200. Between them the files declare 48 user-agent groups, and not one of those groups names any of the 15 AI crawler tokens this scanner evaluates.

/blog/robots-txt-testen-six-files-no-ai-crawler-named.md from /blog/robots-txt-testen-six-files-no-ai-crawler-named

The Really Simple Licensing 1.0 specification, published on 10 December 2025, adds a directive named License to robots.txt, and states in the same section that it does not modify the access permissions expressed by Allow or Disallow. RFC 9309 tells crawlers they MAY interpret records it does not define. Four vendor crawler documentation pages, read on 9 August 2026, do not mention RSL at all.

/blog/rsl-license-directive-changes-no-access.md from /blog/rsl-license-directive-changes-no-access

Six pages captured on 15 July 2026 exposed 985 unique internal paths between them. On 29 August 2026 all six of those sites named a sitemap in robots.txt, five of the named sitemaps resolved, every one of the five was an index rather than a page list, and together they declared 200,712 URLs.

/blog/scan-a-website-for-all-pages-985-linked-200712-declared.md from /blog/scan-a-website-for-all-pages-985-linked-200712-declared

Counted with an HTML parser on 28 August 2026 across the raw and rendered captures of five real pages this scanner stored on 15 July 2026, microdata appears zero times. RDFa appears 204 times and 5 of those name a schema.org type, all of them on one page, which publishes a BreadcrumbList and scores 0 for schema because this scanner reads only JSON-LD.

/blog/schema-structured-data-syntax-decided-the-score.md from /blog/schema-structured-data-syntax-decided-the-score

Google made platform properties globally available on 29 July 2026, so Instagram, TikTok, X and YouTube content now reports inside Search Console. We fetched all four robots.txt files on 31 July 2026 and parsed them with our own parser: Instagram and X blocked 15 of our 15 crawler tokens, TikTok blocked 14, and YouTube blocked none.

/blog/search-console-platforms-block-ai-crawlers.md from /blog/search-console-platforms-block-ai-crawlers

Google's generative AI performance report, read at its Search Console help documentation on 14 August 2026, offers one metric and four dimensions. There is no clicks column, no click-through rate, no average position and no query dimension, and the Search and Discover versions count an impression by opposite rules.

/blog/generative-ai-report-counts-impressions-only.md from /blog/generative-ai-report-counts-impressions-only

Parsed with an HTML parser on 24 August 2026 out of five real pages this scanner captured and stored on 15 July 2026, 7 of the 102 typed JSON-LD nodes carry an author property. Six of those seven sit on Review nodes and name the customer who gave a testimonial. The seventh names an organisation. No page in the set says which person wrote it, and all five carry zero meta author tags, zero rel=author links and zero address elements.

/blog/seven-schema-nodes-named-an-author.md from /blog/seven-schema-nodes-named-an-author

A single-author paper from a Cisco DevNet engineer logged 15 AI clients fetching one documentation page across three trials in February and March 2026. Most of the coding agents identified themselves as an HTTP library, and none of the 15 asked for llms.txt.

/blog/coding-agents-arrive-as-curl-and-axios.md from /blog/coding-agents-arrive-as-curl-and-axios

Lantad requested /products.json from the 49 hostnames in a 123 host sample whose home pages loaded Shopify storefront assets, on 9 September 2026. 39 answered HTTP 200, 38 of those parsed as a product array holding 1,112 products, and not one of the 49 robots.txt files mentions the path.

/blog/shopify-ai-crawlers-1112-products-as-open-json.md from /blog/shopify-ai-crawlers-1112-products-as-open-json

The standard advice is to disallow training crawlers and allow search crawlers. That instruction needs two published names to write, and Lantad's registry records six of nine vendors publishing exactly one, while Cloudflare reported on 1 July 2026 that mixed-use crawlers are over 36 percent of crawler activity on its network.

/blog/six-vendors-publish-one-crawler-token.md from /blog/six-vendors-publish-one-crawler-token

Lantad fetched the robots.txt of 72 Squarespace served hostnames on 10 September 2026. 66 answered HTTP 200, and 64 of those carried the platform's generated file naming 26 AI crawler tokens. On 61 of the 66 those 26 names sat in the same group as the wildcard, which under RFC 9309 gives them exactly the access every other crawler gets.

/blog/squarespace-ai-crawlers-61-of-66-named-not-blocked.md from /blog/squarespace-ai-crawlers-61-of-66-named-not-blocked

A paper posted to arXiv on 13 September 2025 scored 1,100 pages that Brave Summary, Google AI Overviews and Perplexity had cited across 70 prompts, using a sixteen pillar framework it calls GEO-16. Structured data placed third of the six pillars the paper reports, at r = 0.63, behind Metadata Freshness at 0.68 and Semantic HTML at 0.65.

/blog/structured-data-came-third-in-a-citation-audit.md from /blog/structured-data-came-third-in-a-citation-audit

Parsed with an HTML parser on 26 August 2026 out of five real pages this scanner captured and stored on 15 July 2026, the four JSON-LD blocks in the set carry 146 distinct human readable text values. 76 of those values appear somewhere in the visible text of the page they sit on. 70 do not. One page contributes 140 of the 146.

/blog/structured-data-for-ai-70-of-146-values-off-page.md from /blog/structured-data-for-ai-70-of-146-values-off-page

Lantad requested the home page of 392 hostnames on 11 September 2026 and parsed the bytes that came back with no JavaScript executed. 382 answered HTTP 200, 241 of those carried at least one JSON-LD block and 141 carried none. The 241 shipped 442 blocks between them, and eight of the 442 do not parse as JSON at all.

/blog/structured-data-141-of-382-home-pages-carried-none.md from /blog/structured-data-141-of-382-home-pages-carried-none

Dries Buytaert published a month of Cloudflare logs from his own site on 5 March 2026 covering ten AI crawlers and 58,279 requests, of which 4,967 fetched a markdown file and zero used content negotiation, three weeks after Cloudflare shipped a feature that converts pages to markdown only when a request carries an Accept header naming text/markdown.

/blog/no-ai-crawler-used-content-negotiation.md from /blog/no-ai-crawler-used-content-negotiation

Ten products sold into AI visibility were read at their own published pages on 17 August 2026. Three of them examine the customer's own pages, four read a record of requests that already arrived, and six put prompts to models and read the answers back. The crawler half of the category is bundled free or sold as an annual licence, and the answer half is sold monthly by prompt volume.

/blog/ten-ai-visibility-products-three-fetch-your-pages.md from /blog/ten-ai-visibility-products-three-fetch-your-pages

Two golden fixtures stored in this repository produce an identical composite of 34.4, an identical grade of F, and identical access, structure and schema sub-scores. One of them ships zero words in its initial HTML. The other ships all 485 of them, inside a JSON payload. The only stored number that separates the two is retrievability, and retrievability is worth nothing in the grade.

/blog/hydration-json-scored-the-same-as-no-text.md from /blog/hydration-json-scored-the-same-as-no-text

Declarative shadow DOM ships component text in the HTML, inside a template element. Lantad's extraction ruleset drops template subtrees, the default DOM serialization omits shadow roots, and the shadow piercing fallback that recovers the text runs in one of our two renderers.

/blog/shadow-dom-text-and-the-extractor.md from /blog/shadow-dom-text-and-the-extractor

Article 50 of Regulation (EU) 2024/1689 has applied since 2 August 2026 and obliges providers of AI systems that generate synthetic text to mark the output in a machine-readable format. The Regulation names no format. Every technique it mentions sits in one sentence of one recital, inside a list introduced by the words such as, and the string HTML appears in the published text zero times.

/blog/ai-act-marks-ai-text-and-names-no-format.md from /blog/ai-act-marks-ai-text-and-names-no-format

Google began testing a Search Console setting on 3 June 2026 that excludes a site from AI Overviews and AI Mode. It is an account setting rather than a file, so no external scan of your site can tell you whether it is set.

/blog/ai-overviews-opt-out-does-not-live-on-your-site.md from /blog/ai-overviews-opt-out-does-not-live-on-your-site

The IETF vocabulary for expressing AI usage preferences reached its sixth draft on 28 April 2026. The companion draft that defines where to write one down expired on 1 May 2026, so there is still nothing on your site for a crawler to read.

/blog/the-ai-preferences-standard-you-cannot-deploy-yet.md from /blog/the-ai-preferences-standard-you-cannot-deploy-yet

A live index built from real scans, showing how readable the measured web is to AI crawlers, with the sample size stated beside every figure.

/research.md from /research

Regulation (EU) 2024/1689 obliges general-purpose AI model providers to identify and comply with machine-readable rights reservations, and the phrase robots.txt appears nowhere in it. The Commission's Code of Practice names the file and the RFC that defines it, and the Commission's power to fine under Article 101 applies from 2 August 2026.

/blog/eu-ai-act-never-writes-robots-txt.md from /blog/eu-ai-act-never-writes-robots-txt

Dated, argued posts built on measured scan data, plus guides and product notes. Every figure carries its measurement date.

/blog.md from /blog

Every rival name on a Lantad report is still extracted from the answer text, through screens we built after shipping two pieces of output that should never have passed. Since 2026-08-07 you can also pin competitors to a roster, which reverses a position this post originally presented as a design choice, so the post now narrates that change instead of quietly absorbing it.

/blog/rival-names-are-extracted-not-entered.md from /blog/rival-names-are-extracted-not-entered

A robots.txt study posted to arXiv in October 2025 reports 60.0 percent of reputable news sites disallowing at least one AI crawler against 9.1 percent of misinformation sites, and an average of 15.5 named AI user agents against 0.77.

/blog/who-blocks-ai-crawlers-splits-by-credibility.md from /blog/who-blocks-ai-crawlers-splits-by-credibility

Google-Extended, Applebot-Extended and anthropic-ai are robots.txt product tokens that send no user agent of their own, so a log search for them returns zero forever, and that zero does not mean what it looks like.

/blog/the-crawler-tokens-that-never-appear-in-your-logs.md from /blog/the-crawler-tokens-that-never-appear-in-your-logs

Parsed with an HTML parser out of five real pages this scanner captured and stored on 15 July 2026, the whole set holds one time element and one JSON-LD block carrying a date. Three of the five score a perfect 1.0 on our schema sub-score, and only one of those three says anywhere in its markup when it was written or last changed.

/blog/full-schema-marks-and-one-page-carried-a-date.md from /blog/full-schema-marks-and-one-page-carried-a-date

The W3C TDM Reservation Protocol defines four places a site can record that AI mining rights are reserved. Read on 4 August 2026, the crawler documentation published by OpenAI, Anthropic and Google names robots.txt in every case and mentions none of the four.

/blog/tdmrep-not-named-in-crawler-documentation.md from /blog/tdmrep-not-named-in-crawler-documentation

Parsed out of five real pages this scanner captured and stored on 15 July 2026, four JSON-LD blocks hold 26,299 bytes and 102 typed nodes. One page accounts for 24,865 of those bytes and 96 of those nodes, 39 of them describing a navigation menu.

/blog/three-of-five-pages-carried-json-ld.md from /blog/three-of-five-pages-carried-json-ld

Lantad's own component library states that a table is the shape an AI crawler extracts most reliably, so the claim was checked against the pages this scanner has actually stored. Counted with an HTML parser on 21 August 2026 over five real pages captured on 15 July 2026, the whole set holds three table elements, all three on one page, all three navigation boxes, and none of the five pages carries a single caption element.

/blog/three-tables-on-five-pages-all-navigation.md from /blog/three-tables-on-five-pages-all-navigation

Digiday reported on 30 July 2026 that Time has converted its pages to markdown for AI systems and is selling FAQ shaped sponsored content inside those files, which turns the machine copy of a page into something that can differ from the human one by design.

/blog/ads-inside-the-markdown-ai-crawlers-read.md from /blog/ads-inside-the-markdown-ai-crawlers-read

SAGEO Arena ran ten published generative engine optimization strategies through retrieval, reranking and generation over a corpus of 171,003 crawled web documents. We relayed its body-text result here on 31 July 2026, at second hand, from a survey that cited it. The primary source, revised on 7 August 2026, also holds the scope the survey did not relay: rewriting the title, meta description, headings and JSON-LD instead moved the average retrieval hit rate from 0.58 to 0.71.

/blog/metadata-rewrites-gained-22-percent-of-retrievals.md from /blog/metadata-rewrites-gained-22-percent-of-retrievals

Cloudflare said on 1 July 2026 that from 15 September newly onboarded domains will block Training and Agent bots by default on pages that display ads. Your robots.txt will not mention it, because robots.txt was never the layer that answers.

/blog/two-layers-decide-if-ai-can-read-your-site.md from /blog/two-layers-decide-if-ai-can-read-your-site

Lantad fetched the robots.txt of 52 Webflow served hostnames on 11 September 2026. Of the 40 that answered on the owner's own domain, 7 returned HTTP 200 with zero bytes and 13 held a Sitemap line and no User-agent line, so 20 carried no rule of any kind. Six named GPTBot, and two of those six files were written by Cloudflare rather than by anyone at the site.

/blog/webflow-ai-crawlers-20-of-40-held-no-rule.md from /blog/webflow-ai-crawlers-20-of-40-held-no-rule

Six real captures, measured on one day in July. Two stores on the same platform produced opposite outcomes, and the difference was not their content.

/blog/what-a-crawler-meets-on-a-real-storefront.md from /blog/what-a-crawler-meets-on-a-real-storefront

Ahrefs measured 137,210 domains in May 2026 and found 97 percent of valid llms.txt files were never fetched at all. We ship an llms.txt tool, and we think you should know this.

/blog/what-the-evidence-says-about-llms-txt.md from /blog/what-the-evidence-says-about-llms-txt

The IETF working group on crawler authentication listed ten active Internet-Drafts on 29 July 2026, two of them filed on 19 July. The core protocol draft carries intended status Informational, and every mechanism it describes sits in places an external scan of your site cannot reach.

/blog/what-web-bot-auth-actually-specifies.md from /blog/what-web-bot-auth-actually-specifies

Sites sampled by the platform they are built on, from Framer and Webflow to Shopify and WordPress, each fetched as an AI crawler sees it and again in a browser, then compared. Per-platform figures, each with its sample size.

/research/crawlability-study.md from /research/crawlability-study

Every AI visibility tool prints a number. When the measurement failed, that number is a guess wearing a lab coat, so ours refuses to appear.

/blog/why-we-withhold-a-grade.md from /blog/why-we-withhold-a-grade

Run against Lantad's shipped robots.txt matcher on 27 August 2026, appending a trailing asterisk to six rule patterns changed the verdict on none of 168 path comparisons. Appending a trailing dollar sign to four patterns changed 36 of 112.

/blog/wildcard-robots-txt-trailing-star-changed-nothing.md from /blog/wildcard-robots-txt-trailing-star-changed-nothing

Lantad requested the robots.txt of 71 Wix served hostnames on 10 September 2026. 69 answered HTTP 200 and not one of them named an AI crawler token. Wix's help centre tells owners to paste four user agents into that file, and the fourth, BingAI, appears in no crawler documentation OpenAI or Microsoft publishes.

/blog/wix-ai-crawlers-zero-of-69-named-one.md from /blog/wix-ai-crawlers-zero-of-69-named-one

Lantad requested the home page and the robots.txt of 120 hostnames on 8 September 2026. 91 home pages answered HTTP 200, and 24 of those carried a WordPress signal. All 24 served a parseable robots.txt, and 16 of the 24 name none of the 15 AI crawler tokens this scanner evaluates.

/blog/wordpress-ai-crawlers-16-of-24-named-none.md from /blog/wordpress-ai-crawlers-16-of-24-named-none

Lantad requested the home page of 392 hostnames on 9 September 2026 and recorded the response headers. Seven responses carried an X-Robots-Tag. The five that said noindex all arrived as an HTTP 202 of 168 or 169 bytes whose body was a redirect to a captcha, and the same response comes back from the hosting company's own domain.

/blog/x-robots-tag-noindex-five-of-391-were-a-captcha.md from /blog/x-robots-tag-noindex-five-of-391-were-a-captcha

What Lantad stores, for how long, and the rights that apply. Written from what the code does rather than from a template.

/privacy.md from /privacy

The terms that apply to using Lantad, including acceptable use of the scanner and the limits of what a report asserts.

/terms.md from /terms

Every public page on lantad.co in markdown, grouped and linked, so a crawler can take the content without parsing HTML.

/md.md from /md