# Search Console added four platforms, and three of them block AI crawlers

> Google made platform properties globally available on 29 July 2026, so Instagram, TikTok, X and YouTube content now reports inside Search Console. We fetched all four robots.txt files on 31 July 2026 and parsed them with our own parser: Instagram and X blocked 15 of our 15 crawler tokens, TikTok blocked 14, and YouTube blocked none.

- Canonical page: https://lantad.co/blog/search-console-platforms-block-ai-crawlers
- This file: https://lantad.co/blog/search-console-platforms-block-ai-crawlers.md
- Last substantive update: 2026-07-31

## Key facts

- **Published:** 2026-07-31
- **Category:** Findings
- **Author:** Lantad
- **Length:** 3642 words
- **Takeaway 1:** Google's Search relations team posted on Wednesday 29 July 2026 that Search Console platform properties are globally available, covering content published on Instagram, TikTok, X and YouTube.
- **Takeaway 2:** Lantad fetched the four platforms' robots.txt files on 31 July 2026 and evaluated them against the 15 crawler tokens in its registry: Instagram blocked 15 of 15 on a profile path, X blocked 15 of 15 on a status path, TikTok blocked 14 of 15 on a profile path, and YouTube blocked 0 of 15 on a watch path.
- **Takeaway 3:** TikTok's robots.txt stacks 25 user agent lines above a single Disallow: / at line 26 and names 14 of Lantad's 15 registry tokens explicitly, with Amazonbot the only one it does not name.
- **Takeaway 4:** X's robots.txt names exactly one of Lantad's 15 registry tokens, Google-Extended, and blocks the other 14 through the catch-all group User-agent: * followed by Disallow: / at line 117.
- **Takeaway 5:** A platform property reports Google Search performance for content hosted on somebody else's domain, and reading a robots.txt file, including the four readings in this post, measures what the file says rather than whether any crawler obeys it.

## Summary

On Wednesday 29 July 2026 Google's Search relations team posted that Search Console platform properties are globally available. A platform property covers content you publish on Instagram, TikTok, X or YouTube, and Google describes it as the first Search Console property you can verify without owning a domain at all. It reports clicks, impressions and query groups for that content in Google Search, Discover and Google News.

It reports nothing about whether an [AI crawler](https://lantad.co/glossary/ai-crawler) can fetch the same post, and it was never built to. That question is settled somewhere else entirely, in a robots.txt file at the root of a domain you do not own and cannot edit. So we read all four. On 31 July 2026 we fetched the files at www.instagram.com/robots.txt, www.tiktok.com/robots.txt, x.com/robots.txt and www.youtube.com/robots.txt, then ran each through parseRobotsTxt and evaluateRobots in core/src/robots.ts, the same two functions every Lantad scan uses. Three of the four block almost every crawler in our registry. The fourth blocks none of them. That is a far wider spread than we expected across four platforms Google has just placed side by side inside one reporting product.

## What Google shipped on 29 July 2026

[Google's announcement](https://developers.google.com/search/blog/2026/07/platform-properties-social-video-guide), posted by Daniel Waisberg and Lizzi Sassman of the Search relations team on Wednesday 29 July 2026, says two things. Platform properties, announced earlier the same month, are now globally available to everyone. And there is a new guide on analysing social and video content performance to go with them.

The feature itself is narrow and clearly described. You connect an account on Instagram, TikTok, X or YouTube, and Search Console gives you Performance, Insights and Achievements reports for how that content does in Google Search, Discover and Google News. The announcement lists the use cases it has in mind: identifying trending query groups, using a 24 hour filter to catch sudden traffic spikes, exporting several properties into one spreadsheet to compare platforms, and using Search Console annotations to track whether rewriting a YouTube title or a TikTok caption changed anything.

Every one of those is a Google Search measurement. None of them is an [AI visibility](https://lantad.co/glossary/ai-visibility) measurement, and Google does not claim otherwise. The announcement does not mention AI Overviews, AI Mode, or any crawler at all. We are flagging the gap because the two get conflated constantly in practice, not because Google has overstated anything here.

The interesting part for anyone who works on discoverability is the premise underneath the feature. Google's own framing is that publishers now reach audiences through a diverse range of channels beyond websites, which is true, and which is why a Search Console property that needs no domain makes sense. But a channel beyond your website is a channel beyond your robots.txt, your server headers and your HTML. Everything we normally look at when we ask whether a machine can read a page belongs to whoever runs the domain. Once your content is on somebody else's, the whole [research](https://lantad.co/research) question changes shape, and the answer stops being yours to change.

## Whose robots.txt decides whether an AI crawler reads your post

[RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), the Robots Exclusion Protocol, puts the file at a fixed place: the path /robots.txt on the authority serving the content. There is one file per host, it is written by whoever controls that host, and a crawler fetching a URL on that host evaluates that host's file. Nothing in the protocol offers a per-account, per-directory or per-creator override. If your post lives at a URL on tiktok.com, the file that governs it is TikTok's.

That is not a loophole, it is the design. But it produces a situation worth naming plainly for anyone treating a social account as a distribution channel for [generative engine optimization](https://lantad.co/glossary/geo). On your own domain you decide crawler access, and if you get it wrong you can fix it in a text file this afternoon. On a platform you have exactly the access the platform grants, and no amount of caption rewriting changes a Disallow rule.

The protocol also fixes what happens when a group matches. RFC 9309 section 2.2.1 states that if no matching group exists, crawlers MUST obey the group with a user-agent line carrying the * value if one is present, and that if no group matches the product token and no wildcard group exists, no rules apply. That single sentence turns out to decide most of what follows, because three of these four files carry a wildcard group and one does not. The protocol's defaults are load bearing in ways site owners rarely think about, which is also why [a 404 on robots.txt allows every crawler while a 503 blocks them all](https://lantad.co/blog/robots-txt-404-and-503-are-opposites).

None of this requires you to take our word for how a given file resolves. Our [robots.txt tester](https://lantad.co/tools/robots-txt-tester) runs any live file through the same code path and prints which group matched and which rule decided, so you can point it at a platform host and read the verdict yourself rather than reading ours.

## What we measured, and on which four paths

On 31 July 2026 we fetched each platform's robots.txt over HTTPS, saved the bytes, and parsed them offline. All four returned 200. We then called evaluateRobots once per file per token, for each of the 15 tokens in the bot registry at core/src/bots.ts, against one path chosen to represent the kind of URL a platform property would actually be reporting on.

The paths matter, so here they are in full. For Instagram we used /creatorhandle/, a profile page. For TikTok, /@creatorhandle. For X, /creatorhandle/status/1234567890, a single post. For YouTube, /watch?v=abcdefghijk. Those are shapes, not real accounts, and a different path can produce a different verdict on the same file: Instagram, X and TikTok all carry path-specific rules for some crawlers, so this is a reading of four paths rather than a reading of four sites.

The results: Instagram blocked all 15, X blocked all 15, TikTok blocked 14, and YouTube blocked none. The parser returned zero parse notes on all four files, which means it found no unrecognised fields to report, unlike the Content-signal line we met when we ran [Cloudflare's managed robots.txt through the same parser](https://lantad.co/blog/content-signal-asks-disallow-blocks) the day before.

Two things about the number 15 are worth stating before anyone quotes it. The registry is a list we maintain, not a census of every crawler on the web, and its contents are a decision rather than a finding. It holds GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, CCBot and Amazonbot, which is nine vendors, and you can read the same list rendered on our [AI crawlers reference](https://lantad.co/tools/ai-crawlers). A platform that blocks 15 of 15 has not blocked every AI crawler in existence; it has blocked every one we check.

What this reading covers is access. It says nothing about what a crawler would find if it were allowed in, which is a separate question about rendering, [structured data](https://lantad.co/glossary/structured-data) and text, and one our [methodology](https://lantad.co/methodology) treats as a distinct layer for exactly this reason.

## Four files, four different ways of saying no

The counts hide how differently these files are written, and the differences say something about each platform's intent.

TikTok's file is the most deliberate document of the four. One group is headed by 25 consecutive user agent lines and closed by a single Disallow: / at line 26. The names in that stack include GPTBot, OAI-SearchBot, anthropic-ai, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, ChatGPT-User, CCBot, Google-Extended, Applebot-Extended, meta-externalagent, Bytespider, DuckAssistBot, AI2Bot, MistralAI-User, Gemini-Deep-Research, GoogleAgent-Mariner and Google-NotebookLM. That is a maintained list. Somebody has read the vendor documentation, including [OpenAI's crawler page](https://developers.openai.com/api/docs/bots) and [Anthropic's crawler article](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), and kept adding tokens as they were published. Fourteen of our fifteen appear in it. Amazonbot is the omission, and it falls through to a wildcard group whose rules did not match the profile path, so it came back allowed.

Instagram's file names 14 crawlers with Disallow: / and then closes with User-agent: * and Disallow: / at line 266, so the naming is emphasis rather than mechanism. Six of the named ones are in our registry: Amazonbot, Applebot-Extended, ClaudeBot, Google-Extended, GPTBot and PerplexityBot. The other nine registry tokens land on the wildcard rule. Between those two blocks sit path-level groups for Googlebot, Bingbot, Applebot, DuckDuckBot, LinkedInBot, Twitterbot and others, which are allowed most of the site and blocked from things like /direct/ and /ajax/. The shape of the file is a short allow list for named search and preview crawlers, and a closed door for everything else.

X takes the opposite approach to TikTok. Of our 15 tokens it names exactly one, Google-Extended, and the block that catches the other 14 is the wildcard group at lines 116 and 117 under a comment reading, verbatim, every bot that might possibly read and respect this file. That is a default-deny posture: a crawler token published next week is blocked on the day it ships, with no file edit required. It is also why the count is 15 rather than 1.

YouTube's file is the outlier and the shortest. It names no crawler in our registry at all. Its wildcard group carries 21 Disallow paths, covering things like /api/, /login, /results, /live_chat and /youtubei/, and a watch URL is not among them. On robots.txt grounds alone, every token we check may fetch a YouTube watch page. Whether one would get anything useful out of it is a different question entirely, and one we did not measure here: [what a crawler receives without JavaScript](https://lantad.co/tools/what-gptbot-sees) and whether that text matches what a person sees is a [prose parity](https://lantad.co/glossary/prose-parity) problem, and a video page is close to the hardest case there is. Robots access is a permission, not a payload.

## X writes Disallow: * where RFC 9309 writes a leading slash

One detail in X's file is worth pulling out, because it is the kind of thing that makes two tools disagree about the same site.

The three named non-search groups in x.com/robots.txt, for Google-Extended at line 105, FacebookBot at line 108 and Discordbot at line 111, each carry the rule Disallow: * rather than Disallow: /. Those look interchangeable and they are not the same string. RFC 9309 gives the grammar for a rule's value as path-pattern, defined in its ABNF as a forward slash followed by any number of characters. A value of * has no leading slash and does not fit that production. The specification does not then say what a crawler must do with a value that fails the grammar, which is the gap.

Our parser treats the value as a pattern, matches * against the path as a wildcard, and returns blocked. A parser that instead rejected the malformed value and ignored the rule would find no other group matching Google-Extended, fall through to the wildcard group at line 117, and return blocked as well. On this file the two readings agree by luck, because the catch-all is Disallow: / and closes the gap. On a file whose wildcard group were permissive they would not agree, and the site owner would have no way of knowing which behaviour a given crawler implements.

We are not asserting that X made a mistake. The file is careful in other places, including two comments that cite RFC 9309 by name to explain tie-breaking between rules of equal length and to note that only * and $ are metacharacters. Somebody there knows the specification well. We are pointing at an ambiguity in the specification that a real file exercises, and noting that we resolved it one way. [Google's robots.txt documentation](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) is the place to check how the largest crawler resolves cases like this, and it is worth reading before assuming any parser's answer is the answer.

This is the same class of problem as trusting the name in a request. A user agent string is [a claim rather than an identity](https://lantad.co/blog/a-user-agent-is-a-claim-not-an-identity), and a robots.txt rule is a request rather than an enforcement. Both are worth measuring precisely, and neither is worth over-reading. Our own crawler's behaviour and how to exclude it are documented on our [bot page](https://lantad.co/bot) for the same reason.

## What this measurement does not tell you

Four files read on one day is a small measurement, and being clear about its edges is more useful than the headline number.

It does not measure crawler behaviour. A Disallow rule is a request that a well-behaved crawler honours, and this reading contains no evidence about whether any of the 15 tokens actually stayed out. Vendors document their own compliance, and some vendors explicitly carve out user-initiated fetches from robots.txt handling, which is a distinction we have written about before in the context of [tokens that never appear in your logs](https://lantad.co/blog/the-crawler-tokens-that-never-appear-in-your-logs). Reading a file tells you what the file asks for.

It does not measure whether an allowed page is readable. YouTube's 0 of 15 is the clearest case: robots.txt permits the fetch, and everything after that depends on what the server returns to a client that runs no JavaScript. Common Crawl's July archive is a useful scale reference for how much of the web is collected by exactly that kind of client, and we went through [what a non-JavaScript crawler actually archived](https://lantad.co/blog/common-crawl-july-archive-no-javascript) earlier this week. Access and payload are two different measurements and conflating them produces confident wrong answers, which is why we [withhold a grade rather than guess](https://lantad.co/blog/why-we-withhold-a-grade) when only one of the two succeeded.

It does not measure a moment beyond 31 July 2026. These files change. TikTok's list in particular is the kind of list that grows, and any of the four could add or drop a token tomorrow without announcing it. Treat every figure here as a reading with a date attached rather than a property of the platform.

And it does not tell you what to do, because that depends on things we cannot see from outside. What it does establish is the shape of the decision. If a meaningful share of your content lives on Instagram, TikTok or X, then as those files stood on 31 July 2026 the crawlers behind ChatGPT, Claude and Perplexity were not permitted to fetch it, and no Search Console report would have told you so. Content on a domain you control is the part where access is yours to decide, and it is the only part a scan of your own URL can measure or a change on your side can fix. If being quoted in an assistant's answer matters to you, the practical reading is that platform reach and [citation in ChatGPT](https://lantad.co/how-to-get-cited/chatgpt) are separate distribution problems that happen to share a piece of content.

## Questions and answers

**Does a Search Console platform property tell me if AI crawlers can read my social posts?**

No. A platform property reports clicks, impressions and query groups for your Instagram, TikTok, X or YouTube content in Google Search, Discover and Google News. Google's announcement of 29 July 2026 does not mention crawlers, AI Overviews or AI Mode, and none of the three reports it names carries a crawler access field. Whether an AI crawler may fetch the post is decided by the platform's robots.txt.

**Which social platform blocks the most AI crawlers?**

On the four paths we evaluated on 31 July 2026, Instagram and X each blocked all 15 crawler tokens in Lantad's registry, TikTok blocked 14 of 15, and YouTube blocked 0 of 15. Instagram and X reach 15 through a wildcard group carrying Disallow: /, while TikTok reaches 14 by naming each token explicitly above a single Disallow: / rule.

**Can I change the robots.txt rules that apply to my own social posts?**

No. RFC 9309 places one robots.txt file per host, at the path /robots.txt, written by whoever controls that host. A post published on a platform is served from the platform's host, so the platform's file governs it. There is no per-account or per-creator override in the protocol.

**Does YouTube blocking 0 of 15 mean AI crawlers can read YouTube videos?**

It means robots.txt does not stop them fetching a watch URL. It does not mean the fetch returns anything useful. We measured access only, not what the server returns to a client that runs no JavaScript, and a video page is among the hardest cases for text extraction. Those are two separate measurements and this post ran one of them.

---

Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering
crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup.

Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot