Search Console added four platforms, and three of them block AI crawlers

What Google shipped on 29 July 2026

Google's announcement, posted by Daniel Waisberg and Lizzi Sassman of the Search relations team on Wednesday 29 July 2026, says two things. Platform properties, announced earlier the same month, are now globally available to everyone. And there is a new guide on analysing social and video content performance to go with them.

The feature itself is narrow and clearly described. You connect an account on Instagram, TikTok, X or YouTube, and Search Console gives you Performance, Insights and Achievements reports for how that content does in Google Search, Discover and Google News. The announcement lists the use cases it has in mind: identifying trending query groups, using a 24 hour filter to catch sudden traffic spikes, exporting several properties into one spreadsheet to compare platforms, and using Search Console annotations to track whether rewriting a YouTube title or a TikTok caption changed anything.

Every one of those is a Google Search measurement. None of them is an AI visibility measurement, and Google does not claim otherwise. The announcement does not mention AI Overviews, AI Mode, or any crawler at all. We are flagging the gap because the two get conflated constantly in practice, not because Google has overstated anything here.

The interesting part for anyone who works on discoverability is the premise underneath the feature. Google's own framing is that publishers now reach audiences through a diverse range of channels beyond websites, which is true, and which is why a Search Console property that needs no domain makes sense. But a channel beyond your website is a channel beyond your robots.txt, your server headers and your HTML. Everything we normally look at when we ask whether a machine can read a page belongs to whoever runs the domain. Once your content is on somebody else's, the whole research question changes shape, and the answer stops being yours to change.

Whose robots.txt decides whether an AI crawler reads your post

RFC 9309, the Robots Exclusion Protocol, puts the file at a fixed place: the path /robots.txt on the authority serving the content. There is one file per host, it is written by whoever controls that host, and a crawler fetching a URL on that host evaluates that host's file. Nothing in the protocol offers a per-account, per-directory or per-creator override. If your post lives at a URL on tiktok.com, the file that governs it is TikTok's.

That is not a loophole, it is the design. But it produces a situation worth naming plainly for anyone treating a social account as a distribution channel for generative engine optimization. On your own domain you decide crawler access, and if you get it wrong you can fix it in a text file this afternoon. On a platform you have exactly the access the platform grants, and no amount of caption rewriting changes a Disallow rule.

The protocol also fixes what happens when a group matches. RFC 9309 section 2.2.1 states that if no matching group exists, crawlers MUST obey the group with a user-agent line carrying the * value if one is present, and that if no group matches the product token and no wildcard group exists, no rules apply. That single sentence turns out to decide most of what follows, because three of these four files carry a wildcard group and one does not. The protocol's defaults are load bearing in ways site owners rarely think about, which is also why a 404 on robots.txt allows every crawler while a 503 blocks them all.

None of this requires you to take our word for how a given file resolves. Our robots.txt tester runs any live file through the same code path and prints which group matched and which rule decided, so you can point it at a platform host and read the verdict yourself rather than reading ours.

What we measured, and on which four paths

On 31 July 2026 we fetched each platform's robots.txt over HTTPS, saved the bytes, and parsed them offline. All four returned 200. We then called evaluateRobots once per file per token, for each of the 15 tokens in the bot registry at core/src/bots.ts, against one path chosen to represent the kind of URL a platform property would actually be reporting on.

The paths matter, so here they are in full. For Instagram we used /creatorhandle/, a profile page. For TikTok, /@creatorhandle. For X, /creatorhandle/status/1234567890, a single post. For YouTube, /watch?v=abcdefghijk. Those are shapes, not real accounts, and a different path can produce a different verdict on the same file: Instagram, X and TikTok all carry path-specific rules for some crawlers, so this is a reading of four paths rather than a reading of four sites.

The results: Instagram blocked all 15, X blocked all 15, TikTok blocked 14, and YouTube blocked none. The parser returned zero parse notes on all four files, which means it found no unrecognised fields to report, unlike the Content-signal line we met when we ran Cloudflare's managed robots.txt through the same parser the day before.

Two things about the number 15 are worth stating before anyone quotes it. The registry is a list we maintain, not a census of every crawler on the web, and its contents are a decision rather than a finding. It holds GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, CCBot and Amazonbot, which is nine vendors, and you can read the same list rendered on our AI crawlers reference. A platform that blocks 15 of 15 has not blocked every AI crawler in existence; it has blocked every one we check.

What this reading covers is access. It says nothing about what a crawler would find if it were allowed in, which is a separate question about rendering, structured data and text, and one our methodology treats as a distinct layer for exactly this reason.

Four files, four different ways of saying no

The counts hide how differently these files are written, and the differences say something about each platform's intent.

TikTok's file is the most deliberate document of the four. One group is headed by 25 consecutive user agent lines and closed by a single Disallow: / at line 26. The names in that stack include GPTBot, OAI-SearchBot, anthropic-ai, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, ChatGPT-User, CCBot, Google-Extended, Applebot-Extended, meta-externalagent, Bytespider, DuckAssistBot, AI2Bot, MistralAI-User, Gemini-Deep-Research, GoogleAgent-Mariner and Google-NotebookLM. That is a maintained list. Somebody has read the vendor documentation, including OpenAI's crawler page and Anthropic's crawler article, and kept adding tokens as they were published. Fourteen of our fifteen appear in it. Amazonbot is the omission, and it falls through to a wildcard group whose rules did not match the profile path, so it came back allowed.

Instagram's file names 14 crawlers with Disallow: / and then closes with User-agent: * and Disallow: / at line 266, so the naming is emphasis rather than mechanism. Six of the named ones are in our registry: Amazonbot, Applebot-Extended, ClaudeBot, Google-Extended, GPTBot and PerplexityBot. The other nine registry tokens land on the wildcard rule. Between those two blocks sit path-level groups for Googlebot, Bingbot, Applebot, DuckDuckBot, LinkedInBot, Twitterbot and others, which are allowed most of the site and blocked from things like /direct/ and /ajax/. The shape of the file is a short allow list for named search and preview crawlers, and a closed door for everything else.

X takes the opposite approach to TikTok. Of our 15 tokens it names exactly one, Google-Extended, and the block that catches the other 14 is the wildcard group at lines 116 and 117 under a comment reading, verbatim, every bot that might possibly read and respect this file. That is a default-deny posture: a crawler token published next week is blocked on the day it ships, with no file edit required. It is also why the count is 15 rather than 1.

YouTube's file is the outlier and the shortest. It names no crawler in our registry at all. Its wildcard group carries 21 Disallow paths, covering things like /api/, /login, /results, /live_chat and /youtubei/, and a watch URL is not among them. On robots.txt grounds alone, every token we check may fetch a YouTube watch page. Whether one would get anything useful out of it is a different question entirely, and one we did not measure here: what a crawler receives without JavaScript and whether that text matches what a person sees is a prose parity problem, and a video page is close to the hardest case there is. Robots access is a permission, not a payload.

X writes Disallow: * where RFC 9309 writes a leading slash

One detail in X's file is worth pulling out, because it is the kind of thing that makes two tools disagree about the same site.

The three named non-search groups in x.com/robots.txt, for Google-Extended at line 105, FacebookBot at line 108 and Discordbot at line 111, each carry the rule Disallow: * rather than Disallow: /. Those look interchangeable and they are not the same string. RFC 9309 gives the grammar for a rule's value as path-pattern, defined in its ABNF as a forward slash followed by any number of characters. A value of * has no leading slash and does not fit that production. The specification does not then say what a crawler must do with a value that fails the grammar, which is the gap.

Our parser treats the value as a pattern, matches * against the path as a wildcard, and returns blocked. A parser that instead rejected the malformed value and ignored the rule would find no other group matching Google-Extended, fall through to the wildcard group at line 117, and return blocked as well. On this file the two readings agree by luck, because the catch-all is Disallow: / and closes the gap. On a file whose wildcard group were permissive they would not agree, and the site owner would have no way of knowing which behaviour a given crawler implements.

We are not asserting that X made a mistake. The file is careful in other places, including two comments that cite RFC 9309 by name to explain tie-breaking between rules of equal length and to note that only * and $ are metacharacters. Somebody there knows the specification well. We are pointing at an ambiguity in the specification that a real file exercises, and noting that we resolved it one way. Google's robots.txt documentation is the place to check how the largest crawler resolves cases like this, and it is worth reading before assuming any parser's answer is the answer.

This is the same class of problem as trusting the name in a request. A user agent string is a claim rather than an identity, and a robots.txt rule is a request rather than an enforcement. Both are worth measuring precisely, and neither is worth over-reading. Our own crawler's behaviour and how to exclude it are documented on our bot page for the same reason.

What this measurement does not tell you

Four files read on one day is a small measurement, and being clear about its edges is more useful than the headline number.

It does not measure crawler behaviour. A Disallow rule is a request that a well-behaved crawler honours, and this reading contains no evidence about whether any of the 15 tokens actually stayed out. Vendors document their own compliance, and some vendors explicitly carve out user-initiated fetches from robots.txt handling, which is a distinction we have written about before in the context of tokens that never appear in your logs. Reading a file tells you what the file asks for.

It does not measure whether an allowed page is readable. YouTube's 0 of 15 is the clearest case: robots.txt permits the fetch, and everything after that depends on what the server returns to a client that runs no JavaScript. Common Crawl's July archive is a useful scale reference for how much of the web is collected by exactly that kind of client, and we went through what a non-JavaScript crawler actually archived earlier this week. Access and payload are two different measurements and conflating them produces confident wrong answers, which is why we withhold a grade rather than guess when only one of the two succeeded.

It does not measure a moment beyond 31 July 2026. These files change. TikTok's list in particular is the kind of list that grows, and any of the four could add or drop a token tomorrow without announcing it. Treat every figure here as a reading with a date attached rather than a property of the platform.

And it does not tell you what to do, because that depends on things we cannot see from outside. What it does establish is the shape of the decision. If a meaningful share of your content lives on Instagram, TikTok or X, then as those files stood on 31 July 2026 the crawlers behind ChatGPT, Claude and Perplexity were not permitted to fetch it, and no Search Console report would have told you so. Content on a domain you control is the part where access is yours to decide, and it is the only part a scan of your own URL can measure or a change on your side can fix. If being quoted in an assistant's answer matters to you, the practical reading is that platform reach and citation in ChatGPT are separate distribution problems that happen to share a piece of content.