Google indexes 20 document file types, and three AI crawler vendors name none

What Google documents it can index

The page is short and it is a list. Under supported flat file types, meaning files whose content is stored as plain unencoded text, it names nine: comma-separated values, Google Earth in both .kml and .kmz, GPS eXchange Format, HTML, Scalable Vector Graphics, TeX and LaTeX, plain text, Wireless Markup Language in .wml and .wap, and XML. The plain text entry carries six nested examples of source code that Google treats as text, covering Basic, C and C++, C#, Java, Perl and Python by extension.

Under supported encoded file types, meaning binary files or containers that need a specific parser to get the human-readable text out, it names eleven: Adobe Portable Document Format, Adobe PostScript, Electronic Publication, Hancom Hanword, Microsoft Excel in .xls and .xlsx, Microsoft PowerPoint, Microsoft Word, OpenOffice presentation, OpenOffice spreadsheet, OpenOffice text, and Rich Text Format. Nine plus eleven is the twenty in the title of this post, and the count is of top-level entries on that page rather than of file extensions, which run higher because several entries carry two or three.

Below those, the same page lists media: seven image formats, being BMP, GIF, JPEG, PNG, WebP, SVG and AVIF, and twenty-one video formats from 3GP through XAP. SVG is the one format that appears in both the flat file list and the image list, which is correct rather than sloppy, since an SVG is XML that also renders as a picture. The page closes by describing the filetype: search operator, which is how a person can check from outside whether any of this reached the index for a given site.

Two things about that list matter more than its length. The first is that it exists at all, dated and revisable, so a site owner can plan against it. The second is that it is an indexing list rather than a retrieval promise: it says Google can extract text from a .docx, not that any particular .docx will rank or be quoted. That distinction runs through everything on this site, and it is the same reason the scoring methodology separates what was measured from what was inferred. A format being parseable is upstream of everything else and settles nothing on its own, in the same way that valid structured data is upstream of a rich result without guaranteeing one. What the list does settle is the negative case, which is worth more: a format that is not on it is not being read as text by that crawler, and no amount of work on the file will change that. A related boundary is already published here: Googlebot reads the first two megabytes of a supported file type and stops, which means a long PDF is both a supported format and a truncated one.

What the AI crawler documentation says about file formats

Nothing. That is the whole finding for this section, and the way it was established is worth stating so a reader can repeat it: each page was fetched, stripped of markup, and searched for the strings PDF, file type, filetype, Content-Type and sitemap. On all three, every one of those counts came back zero on 19 August 2026.

OpenAI's crawler documentation describes three agents. GPTBot is described as used to crawl content that may be used in training its generative AI foundation models. OAI-SearchBot is described as used to surface websites in search results in ChatGPT's search features. ChatGPT-User is described as visiting a web page when a user asks ChatGPT or a Custom GPT a question. The page publishes IP address ranges and explains that the robots.txt tokens are independent of one another. It carries no date and says nothing about what those agents do when the thing at the URL is a spreadsheet.

Anthropic's support article on its crawlers is structured as a table of three bots and what happens when you disable each. ClaudeBot collects web content that could contribute to model training. Claude-User accesses websites when an individual asks Claude a question. Claude-SearchBot navigates the web to improve search result quality. The page shows a date of 7 April 2026, which makes it the only one of the three that lets a reader tell how current it is. It, too, describes robots.txt and nothing about formats.

Perplexity's bot documentation names PerplexityBot, described as designed to surface and link websites in search results and explicitly not used to crawl content for AI foundation models, and Perplexity-User, described as visiting a web page when a user asks a question. It carries no date.

None of this is evidence that those crawlers cannot read a PDF. It is evidence that a site owner cannot find out from the vendor, which is a different and more actionable thing, because it tells you where the guessing starts. The pattern is familiar from other checks on this blog: no crawler vendor documents Retry-After either, and the gap between how many tokens a vendor operates and how many it publishes was measured when six vendors published one crawler token each. Crawler documentation from these companies is written to answer one question, which is how to block the crawler, and it answers that question well. It is not written to tell you what the crawler can consume, and reading it as though it were is how a plan gets built on an assumption nobody checked.

Why the Content-Type header is the whole decision

Google's page states the mechanism in one sentence: the file type is determined by the Content-Type HTTP header returned when Google crawls the file, though in some cases Google may use the file extension or re-parse the file using a different parser if the header is missing or incorrect. Read that as an ordering. The header is consulted first. The extension is a fallback. Re-parsing is a second fallback. A misconfigured server is therefore recoverable but not reliably so, and the recovery is undocumented in its particulars.

This puts a lot of weight on a header most site owners never look at and most content management systems set automatically. It is also the header that decides what happens on the ordinary HTML path, which is why it is worth checking rather than assuming. A page served as text/plain is not going to be parsed as a document tree by anything, however good the markup inside it is. A JSON API response served at a URL a person can visit is not a page. The cheapest way to see what a plain fetch actually returns from your own site is to look at the crawler's view of a URL, which shows the response before any browser has touched it.

The header is not the only place a fetch can go wrong on the transport layer rather than the content layer, and this blog has counted a few of them. Ten AI crawlers fetching a site over a month used no content negotiation at all, asking for markdown by URL instead, which means the Accept header is not the lever some advice treats it as. A robots.txt file returning a 404 and returning a 503 mean opposite things to a compliant crawler. These are unglamorous, and they sit upstream of every content decision anyone makes.

Where this bites hardest is on hosted platforms, because the header is set by the platform and not by the person writing the content. If a store or a site builder serves a downloadable asset with a generic octet-stream type, the fallback path is the only thing left, and the fallback path is documented as sometimes rather than always. That is worth checking once on the platform you actually use: the Shopify guide covers the equivalent checks on the HTML side, where the same principle applies, which is that the platform's default is the behaviour unless somebody changed it. The measurement this scanner does of prose parity is a comparison of two views of one HTML document, and it presupposes that the document arrived as HTML in the first place.

A PDF has no head element, so it has no robots meta tag

The second-order consequence of non-HTML content is that the usual controls are not available on it. A robots meta tag lives in the head element of an HTML document. A PDF, a spreadsheet and a JPEG have no head element, so there is nowhere to put one.

Google documents the workaround directly. Google's robots meta tag documentation, carrying Last updated 2026-03-24 UTC, states that to block indexing of non-HTML resources such as PDF files, video files or image files, you use the X-Robots-Tag response header instead, and that you can use X-Robots-Tag for non-HTML files where the use of robots meta tags in HTML is not possible. It gives configuration for both Apache and NGINX matching on a .pdf suffix and setting the header to noindex, nofollow. That is a per-server change rather than a per-file one, which is why it is often the piece that never gets done.

robots.txt has no such gap, and the reason is in the specification. RFC 9309 opens by saying it applies to services that provide resources clients can access through URIs, and the rules it defines are matched against the URI path. A path is a path whatever it serves, so a Disallow line covering /docs/ covers the PDFs under it exactly as it covers the HTML. The same RFC is blunt that these rules are not a form of access authorization, which is the line worth remembering before treating any of this as a control. You can check how a given file is treated per crawler with the robots.txt tester, which evaluates each token separately rather than collapsing them.

So the shape of the control surface is asymmetric, and the asymmetry cuts a specific way. robots.txt reaches every file type and is honoured or not by each crawler on its own terms. The meta tag reaches only HTML. The X-Robots-Tag header reaches everything but requires server configuration. And the AI-specific opt-outs mostly live in one of the two narrow places rather than the broad one: Microsoft's AI opt-out is a meta tag, which by construction cannot be applied to a PDF at all. A company that publishes its substantive material as documents and its marketing as pages therefore has finer control over the marketing than over the material, which is the opposite of what most people would choose.

What this scanner does with a URL that is not HTML

It refuses to score it. That is a design decision in this repository rather than a measurement, and stating it here is the point of this section, because a post criticising three companies for silence should say what its own tool does before a reader has to find out.

The scan pipeline reads the content-type header off the first plain fetch and computes a single boolean from it. A URL counts as HTML when the header contains text/html, or contains xhtml, or is absent entirely. Everything else is false. That boolean gates the browser render, so no rendering is attempted on a non-HTML response, and it gates the verdict, which returns a state called not_html with a fixed headline saying that the URL does not serve HTML, so there is nothing for an AI crawler to parse as a page, and no score. Every sub-score, including access, is set to null rather than to zero, because a zero would read as a measured failure rather than as a measurement that was not taken.

There is one ordering rule worth naming because it was found the hard way and is written into the code as a comment: a failing status beats the content-type test. A URL answering 429 while serving JSON is an access failure, not a not-HTML result, and the code checks the fatal access branch first so the verdict blames the right thing. That is the same instinct behind withholding a grade rather than emitting a confident wrong one.

The honest summary is that this scanner covers one of the twenty document formats on Google's list, and the one it covers is the one that everything else here is built around: fetch it plainly, render it, compare the two, and report the difference. It reads a site's robots.txt against the crawler tokens it evaluates, and those rules do cover a PDF path, so the access half of the answer is available for a document even though the readability half is not. Nothing in the pipeline opens a PDF, extracts its text, or reports whether the text inside it survived. If your most important content is a document rather than a page, this tool measures the route to it and not the thing itself, and we would rather write that down than let a green grade imply otherwise. Our own crawler's behaviour and opt-out are published at our bot policy page on the same principle.