Home / Blog
The Lantad blog
What we find when we measure whether AI crawlers can actually read a site, what to do about it, and what we are changing in the product. Definitions and per-stack fixes live in the glossary and the fix guides; this is the dated, argued half. Also available as an RSS feed.
Latest
234 of 592 sites that ban GPTBot in robots.txt served it a 200
HasData fetched as well as read. Across a 2,096 domain enforcement subset measured in July 2026, 39.5 percent of the sites disallowing GPTBot returned it a live 200 when a request carrying that name actually knocked.
What the file said
- 592 domains disallow GPTBot in robots.txt
- A published, machine readable instruction
- Costs one line to write
- Read without fetching any other page
What the server did
- 234 of those 592 returned a live 200
- 39.5 percent of the declared bans
- Measured with a second request
- The instruction was published, not applied
Findings 21
- Common Crawl's own researchers put its persistent core near 40 percent A paper from three Common Crawl Foundation authors, posted to arXiv on 15 July 2026, fits 51 monthly crawls from 2020 to 2025 and reports a persistent core fraction of about 0.4 at domain granularity, with the rest of the domain population churning through a shell.
- Google's generative AI guide names five GEO tactics you can ignore Google Search Central publishes an official guide to appearing in AI Overviews and AI Mode, and a section of it headed Mythbusting lists llms.txt, chunking, rewriting for AI, inauthentic mentions and structured data among the things you do not need to do.
- More AI citations did not mean more of your page in the answer A measurement framework posted to arXiv on 29 April 2026 scores how much of each cited page reaches the answer text. Across 21,143 valid search-layer citations, Perplexity averaged 16.35 citations per prompt to ChatGPT's 6.88, and mean influence among fetched pages ran the other way, 0.0646 against 0.2713.
- Six of the nine AI vendors we track publish one crawler token The standard advice is to disallow training crawlers and allow search crawlers. That instruction needs two published names to write, and Lantad's registry records six of nine vendors publishing exactly one, while Cloudflare reported on 1 July 2026 that mixed-use crawlers are over 36 percent of crawler activity on its network.
- Time is selling sponsored blocks inside the markdown AI crawlers read Digiday reported on 30 July 2026 that Time has converted its pages to markdown for AI systems and is selling FAQ shaped sponsored content inside those files, which turns the machine copy of a page into something that can differ from the human one by design.
- A survey of 45 GEO studies puts crawling second and says few observe it A critical survey posted to arXiv on 15 July 2026 reviewed 45 generative engine optimization studies and reported that none of them shows a stable, cross-platform causal effect on organic discoverability. Its own pipeline diagram puts crawling and indexing at stage two of seven, and its caption says far fewer studies look there.
- Fourteen Google agents that a robots.txt wildcard does not stop Google's crawling documentation sorts everything it sends at a website into three classes. Nine user-triggered fetchers generally ignore robots.txt, and five special-case crawlers ignore the global user agent group, so fourteen named Google agents are not decided by the group you head with an asterisk.
- Only 16 schema.org types reach 10 million domains Schema.org began publishing domain counts for its whole vocabulary on 4 June 2026. Counting the July 2026 file on 31 July 2026 gives 958 types and 4,587 properties, of which 4,217 sit in the lowest bucket of fewer than 1,000 domains and 16 types reach the top bucket of more than 10 million.
- Search Console added four platforms, and three of them block AI crawlers Google made platform properties globally available on 29 July 2026, so Instagram, TikTok, X and YouTube content now reports inside Search Console. We fetched all four robots.txt files on 31 July 2026 and parsed them with our own parser: Instagram and X blocked 15 of our 15 crawler tokens, TikTok blocked 14, and YouTube blocked none.
- Cloudflare's Content-signal line asks, and the Disallow lines under it block Cloudflare's managed robots.txt writes one Content-signal line and eight Disallow rules into a site's file. We parsed the published example on 30 July 2026: the signal was recorded as an unknown field and ignored, and the Disallow rules blocked 8 of the 15 crawler tokens in our registry.
- Common Crawl archived 2.14 billion pages in July without running JavaScript Common Crawl published CC-MAIN-2026-30 on 28 July 2026, 2.14 billion pages collected between 7 and 25 July. Its own FAQ states the crawler executes no JavaScript, so what a client rendered site contributed to that archive is whatever its server returned.
- How many prompts an AI visibility measurement needs A study of 374,052 citations across Perplexity, OpenAI SearchGPT and Google Gemini found repeated runs of the same query share as little as 0.29 of their cited domains, and that a confidence interval five percentage points wide on citation share needs roughly 40 queries on one platform and 150 or more on another.
- Microsoft's AI opt out is a meta tag, not a robots.txt token Every other major vendor gave site owners a robots.txt name to disallow. Microsoft's published guidance for keeping a page out of its AI answers is NOCACHE or NOARCHIVE in a meta robots tag, which is why none of the 15 tokens in our crawler registry belongs to Microsoft.
- A 404 on robots.txt allows every crawler, a 503 blocks them all RFC 9309 gives a missing robots.txt and a broken robots.txt opposite meanings: a 4xx lets a crawler access any resource on the server, a 5xx obliges it to assume complete disallow. Google's documentation carves 429 out of that rule. Lantad's code does not.
- Four OpenAI crawler IP lists, opened and counted OpenAI publishes a JSON file of IP address ranges for each of its four crawlers. Read on 29 July 2026, the ChatGPT-User file covered 37,328 IPv4 addresses against GPTBot's 3,472, and it is the one crawler OpenAI says robots.txt rules may not apply to.
- The EU AI Act never writes robots.txt, and its Code of Practice does twice Regulation (EU) 2024/1689 obliges general-purpose AI model providers to identify and comply with machine-readable rights reservations, and the phrase robots.txt appears nowhere in it. The Commission's Code of Practice names the file and the RFC that defines it, and the Commission's power to fine under Article 101 applies from 2 August 2026.
- A perfect schema score and a zero, measured on the same page Lantad reads the same JSON-LD twice with two different traversals, and on our stored Wikipedia capture one reading returns a perfect 1.0 while the other returns 0 out of 100. Both are right, and the gap is the honest answer to what structured data does for AI.
- AI Overviews and the one crawler our scanner does not model Getting quoted in an AI Overview starts with Googlebot, and Googlebot is not one of the fifteen tokens in our crawler registry. An audit of our own blind spot, read line by line against our own source.
- Nearly a third of AI Overview citations are not on page one A Washington University in St. Louis team issued 55,393 trending queries over 40 days and found that 29.8 percent of the domains an AI Overview cites appear nowhere on the first page of results for the same query.
- What the evidence actually says about llms.txt Ahrefs measured 137,210 domains in May 2026 and found 97 percent of valid llms.txt files were never fetched at all. We ship an llms.txt tool, and we think you should know this.
- What a crawler actually meets on a real storefront Six real captures, measured on one day in July. Two stores on the same platform produced opposite outcomes, and the difference was not their content.
Guides 7
- What Web Bot Auth actually specifies, and what it does not The IETF working group on crawler authentication listed ten active Internet-Drafts on 29 July 2026, two of them filed on 19 July. The core protocol draft carries intended status Informational, and every mechanism it describes sits in places an external scan of your site cannot reach.
- A user agent is a claim, not an identity Every per-crawler number you have ever seen, including ours, counts requests that said they were a crawler. Verifying the claim needs an IP check, and most tools do not do one.
- The AI Overviews opt out does not live on your site Google began testing a Search Console setting on 3 June 2026 that excludes a site from AI Overviews and AI Mode. It is an account setting rather than a file, so no external scan of your site can tell you whether it is set.
- The AI preferences standard you cannot deploy yet The IETF vocabulary for expressing AI usage preferences reached its sixth draft on 28 April 2026. The companion draft that defines where to write one down expired on 1 May 2026, so there is still nothing on your site for a crawler to read.
- Three AI crawler tokens that will never appear in your logs Google-Extended, Applebot-Extended and anthropic-ai are robots.txt product tokens that send no user agent of their own, so a log search for them returns zero forever, and that zero does not mean what it looks like.
- Two layers decide whether AI can read your site, and robots.txt is the weaker one Cloudflare said on 1 July 2026 that from 15 September newly onboarded domains will block Training and Agent bots by default on pages that display ads. Your robots.txt will not mention it, because robots.txt was never the layer that answers.
- Five structural signals that tell an AI who you are We re-measured the five structured-data signals behind Entity Confidence against 30 pages fetched in one day, removed two of them, and added two nobody was checking.
Product 4
- A Gemini citation tool is not an AI Overview tool What actually comes back when a tool asks Gemini a grounded question, why the cited URI can be a Google redirect rather than a publisher, and which parts of a citation table survive that.
- AI crawler detection is an ordering problem Turning a user agent string into an AI crawler label is a matching order with named failure modes, and every one of them produces a confident wrong answer rather than an error.
- The rival names on your report were never typed in Lantad does produce a comparative view of who answer engines name instead of you, and it does it with no competitor field at all: every rival name is extracted from the answer text, through screens we built after shipping two pieces of output that should never have passed, one a name and one a question.
- Why we will not grade a page we could not measure Every AI visibility tool prints a number. When the measurement failed, that number is a guess wearing a lab coat, so ours refuses to appear.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.