Mistral AI crawler: 1 of 323 robots.txt files named one

What is the Mistral AI crawler, and which token blocks it?

There is no single Mistral AI crawler. The page at docs.mistral.ai/robots, which we read on 12 September 2026, documents three, and the distinction between them decides which rule you should write.

Of the first, Mistral writes that MistralAI-User "is for user actions in Vibe", that when users ask a question "it may visit a web page to help answer and include a link to the source in its response", and that it "is not used for crawling the web in any automatic fashion, nor to crawl content for generative AI training". Of the second, that MistralAI-Index "is for automated crawling of the web for indexing purposes only", that it indexes content for Mistral search, and that what it crawls "is not used for generative AI training of any kind". Of the third, that MistralAI-Training "crawls web content to help build datasets for training Mistral generative AI models" and that webmasters can disallow that user agent in their robots.txt file.

Read in order those three sentences answer the question most site owners are actually asking. If you want to stay out of training data and stay in answers, the token you disallow is MistralAI-Training and the two you leave alone are the other two. Disallowing MistralAI-User does the opposite of what most people reach for robots.txt to do: it refuses the fetch that produces a cited link. That asymmetry between a training crawler and a retrieval crawler is the same one that makes the tokens that never appear in your logs so easy to get backwards, and it is why what an AI crawler is is worth being precise about before writing a rule.

Each of the three publishes a full user-agent string, and all three point back at the same documentation URL. That is more than several larger operators manage. OpenAI's crawler documentation names its agents with full strings, Anthropic's help centre article publishes tokens but no full user-agent string for them, and Google's crawler overview documents eleven common crawlers of which none is a Gemini crawler, leaving one token covering training and grounding together. Against that field, three purposes and three names is the clearer contract.

One absence is worth naming. Mistral's page carries no published or last updated date anywhere on it, so a reader has no way to tell whether the three tokens are this month's list or last year's. We have measured before that crawler IP files carry two dates that disagree, and an undated page is the same problem with the evidence removed: a user agent is a claim, not an identity, and an undated claim cannot even be checked for staleness.

How many robots.txt files name a Mistral token?

One. Here is how that was counted, because the frame matters more than the number.

On 12 September 2026 we requested https://HOST/robots.txt once for each of the 392 hostnames in the committed platform corpus frame, as LantadBot/1.0, with redirects followed, a single attempt each, a twenty second timeout, from one network location. Every one of the 392 returned something: 359 answered HTTP 200, 26 answered HTTP 404, four answered HTTP 202 and three answered HTTP 403. A 404 is not a neutral outcome here, since under RFC 9309 an absent file permits access to everything. Of the 359 that answered 200, two returned an HTML document rather than a text file and 34 carried neither a user-agent line nor an allow or disallow line, leaving 323 files that parse as robots.txt with at least one group.

Those 323 files carry 2,382 user-agent lines between them, naming 255 distinct values once the wildcard is counted as one. Exactly one of the 323 names a Mistral token. It is sammylabs.com, whose file runs to 236 lines and names 25 distinct values, and the relevant three lines sit under a comment reading "# Mistral": a group for MistralAI-User, then Allow: / and Disallow: /api/. So the single site in this corpus that has heard of Mistral's crawlers is inviting one of them in, not keeping it out. Its wildcard group allows the root as well, and carries a Content-Signal line declaring search=yes, ai-input=yes and ai-train=yes, which is a separate mechanism we have measured where the signal line asks and the disallow lines block. MistralAI-Index and MistralAI-Training are not named anywhere in that file, so they fall to the wildcard group and are allowed too, by default rather than by decision.

One other file in the 392 mentions Mistral at all. botcity.dev writes User-agent: MistralBot followed by Disallow: /, in a block that reads unmistakably as an attempt to refuse the company. MistralBot appears nowhere in the documentation Mistral publishes. RFC 9309 requires crawlers to use case-insensitive matching to find the group that matches their product token, and if no group matches, to obey the group with the "*" value instead. MistralAI-Training does not match a group named MistralBot. The line is inert. What actually refuses Mistral at botcity.dev is the wildcard group 43 lines above it, which carries its own Disallow: /, so the site gets the outcome it wanted for a reason unconnected to the rule it wrote. That is the same failure mode as a renamed token leaving a stale rule matching nothing and as the file we found where ChatGPT disallows a token no crawler can match. A token is not a brand name. It is a string, and it either matches or it does nothing, which is what our robots.txt tester exists to show before you publish the file.

Why the long blocklists do not name Mistral

The shape of these 323 files explains the result better than any claim about what site owners believe. Most of them are not blocklists at all: 180 of the 323 name exactly one user-agent value, and for 177 of those 180 that single value is the wildcard. Another 73 name between two and nine, 18 name between ten and nineteen, 49 name between twenty and thirty-nine, and three name forty or more. The largest is distefanolandscaping.com, whose 5,776 byte file names 138 distinct values.

So the corpus splits into a large majority that has written nothing about AI crawlers specifically and a visible minority that has written a great deal. The interesting thing about that minority is how little of it is original. Among the 52 files naming twenty or more values there are only eleven distinct sets of names, and the largest single set is shared, value for value, by 40 different sites: the wildcard plus 29 named tokens, running from adsbot-google through to YouBot and taking in AI2Bot, AI2Bot-Dolma, aiHitBot, Cohere-ai, Cohere-training-data-crawler, DuckAssistBot, img2dataset, MyCentralAIScraperBot, Omgili, omgilibot, Quora-Bot and TikTokSpider along the way. Forty sites, one list, no Mistral token in it.

That is not a criticism of the 40 sites. Pasting a maintained list is a reasonable thing for a small business to do, and the alternative is reading nine vendor documentation pages a quarter. It is a statement about what a pasted list is: a snapshot of the day somebody wrote it, propagated unchanged. MyCentralAIScraperBot is named by 40 files in this corpus and MistralAI-Training by none, which tells you about the vintage of the list rather than the relative importance of the two crawlers. We have watched the same mechanism from the platform side, where Wix told owners to paste four user agents and one of them was BingAI, where Squarespace generated 26 token names and put them in the wildcard group, and where Cloudflare will generate the file for you and change its token list without telling you.

Set against the pasted list, the named tokens in the corpus track roughly how long each crawler has existed rather than how much traffic it represents. GPTBot appears in 75 of the 323 files, ClaudeBot in 74, Google-Extended in 69, Bytespider in 64, Applebot-Extended in 63, CCBot in 62, Meta-ExternalAgent in 60 and Amazonbot in 57. The newer retrieval tokens are much thinner: PerplexityBot and ChatGPT-User in 22 each, OAI-SearchBot in 18, Perplexity-User in 13, Claude-User in 6 and Claude-SearchBot in 5. We measured the same ranking from a different angle when 16 of 140 files named OAI-SearchBot and 25 named GPTBot. A robots.txt file is a record of what its author had read about, and on this evidence most authors stopped reading some time ago.

What Mistral publishes so you can verify the crawler

A robots.txt token asks a crawler to identify itself honestly. Verifying that it did means checking the address the request came from against a list the operator publishes, which is the only half of crawler identity that cannot be forged by setting a header.

Mistral's documentation links an address list for two of its three crawlers, and we requested all three on 12 September 2026. The file at mistral.ai/mistralai-user-ips.json returned HTTP 200 and 279 bytes: four prefixes, every one of them a /32, so four single IPv4 addresses, under a stated creationTime of 2025-02-19. The file at mistral.ai/mistralai-index-ips.json returned HTTP 200 and 176 bytes: two prefixes, both /32, so two addresses, under a stated creationTime of 2026-04-19. Following the same naming pattern to mistral.ai/mistralai-training-ips.json returned HTTP 404, and the documentation page lists no address file for the training crawler at all, publishing only its user-agent string.

The distribution of what is published is therefore the inverse of what a site owner most needs. The crawler whose behaviour people care most about controlling is the training crawler, and it is the one you cannot verify. If a request arrives claiming to be MistralAI-Training, there is no published list to check it against, which means there is no way to distinguish it from anything else that has typed that string into a header. We have described that gap in general terms before, where a bot declares three things and proves one and where 6 of 15 crawler tokens publish a user-agent string to match at all. This is a clean instance of it.

Six published addresses across two lists is also small. For scale, we read Anthropic's list on 7 September 2026 and found 26 prefixes covering 1,094 IPv4 addresses, a single list shared by all three of its bots, and OpenAI publishes separate files per agent, which we opened and counted rather than taking on trust. Mistral's two files are narrower and more specific than either, which is useful if they are current and misleading if they are not. The creationTime in each file is a string the file asserts about itself. It is not evidence that the set is complete today, and the user list asserting February 2025 while the index list asserts April 2026 is exactly the kind of spread that makes a stated date worth quoting rather than relying on.

What Lantad does not measure about Mistral

The uncomfortable half of this post is about our own scanner rather than about anybody's robots.txt file.

Lantad's crawler registry in core/src/bots.ts holds 15 product tokens across nine vendors: OpenAI, Anthropic, Perplexity, Google, Apple, Meta, ByteDance, Common Crawl and Amazon. None of those 15 tokens is a Mistral token. That is a configuration choice rather than a measurement, and the consequence is concrete and current: when a Lantad scan evaluates a robots.txt file today, it does not ask whether MistralAI-User, MistralAI-Index or MistralAI-Training may fetch the page, so no report we produce answers that question for any site. The one file in this corpus that names MistralAI-User names 25 distinct user-agent values, and only 13 of those are tokens our registry evaluates, so our own output would pass over the Mistral group without comment.

The gap runs one layer further. The answer engine registry in core/src/engines.ts holds eight engines, and they are Workers AI, Gemini, DeepSeek, Grok, Perplexity, OpenAI, Anthropic and Google AI Overviews via a search results capture. Mistral's assistant is not among them, so nothing in our visibility measurement asks a Mistral model anything or reads a citation back from one. Both registries are lists somebody maintains, and both are currently one company short. We have published this shape of finding before, when Meta documented five crawler tokens and we had published that it documents one, when six of nine vendors we track published exactly one token, and when we named the AI Overviews crawler our scanner does not model. Recording it is the only mechanism that has ever made us fix it.

Until the registry carries the tokens, the honest answer for a reader who wants to know where they stand with Mistral is to check it directly, and it takes one request and one read. Fetch your own /robots.txt, search it for the string Mistral, and note whether what you find is one of the three documented tokens or a name like MistralBot that matches nothing. Decide deliberately which of the three purposes you are refusing, since the retrieval crawler and the training crawler pull in opposite directions for anyone who wants to be cited. Then check the address lists yourself if you care about verification, and treat the training crawler as unverifiable until a list exists. That is the same order of operations our methodology applies to the nine vendors it does cover, our crawler directory publishes token by token, and our own crawler is held to.