BlogFindings
Grok SEO: xAI documents no crawler, and 10 robots.txt files guessed 8 names
Lantad read every page of xAI's published documentation on 18 September 2026, 182 pages and 200,241 words, and the strings robots, crawler, crawl and user agent appear in it zero times each. In the same day's sweep of 1,419 hostnames, ten robots.txt files tried to write a rule for an xAI crawler anyway, between them spelling its name eight different ways.
On 18 September 2026 we went looking for the same page at xAI and read the whole of what the company publishes to developers instead: the documentation index at docs.x.ai, the sitemap behind it, and the single concatenated file that carries every page's full text. That is 182 pages and roughly 200,241 words. The word robots does not appear in it. Neither does crawler, crawl, user agent or spider. What does appear, 151 times, is web_search, the name of the API tool that sends Grok to fetch web pages on a caller's behalf, and 179 times the word citation, which is what the same API hands back afterwards.
In short
- Grok SEO has no robots.txt control surface to configure: xAI's complete published documentation, 182 pages and 200,241 words read on 18 September 2026, contains the strings robots, crawler, crawl, user agent and spider zero times each, while web_search appears 151 times and citation 179 times.
- Lantad requested /robots.txt from 1,419 hostnames on 18 September 2026. 1,000 of the files returned parsed into at least one group and named 2,240 distinct product tokens between them. Ten of those files name a token containing grok or xai, and those ten spell it eight different ways.
- Eight of those ten files disallow the site root to the xAI token they guessed at, and on seven of the eight the wildcard group in the same file allows the site root, so each refusal holds only if the spelling guessed is the one xAI actually sends.
- xAI's two own robots.txt files disagree about training on 18 September 2026: x.ai declares Content-Signal ai-train=yes and docs.x.ai declares ai-train=no, and neither file names a crawler belonging to xAI.
- Lantad's own bot registry evaluates 15 tokens across nine vendors and holds no xAI entry, which is a gap in this scanner rather than a finding about the web, and it stays a gap until xAI publishes a token we can source.
| String | Occurrences | What a hit would have documented |
|---|---|---|
| web_search | 151 | The API tool that fetches web pages for Grok |
| citation | 179 | The source URLs the tool returns with an answer |
| allowed_domains | 10 | A domain filter, set by the API caller |
| excluded_domains | 7 | A domain exclusion, set by the API caller |
| robots | 0 | Any reference to the robots exclusion protocol |
| crawler | 0 | A named crawler a site owner could address |
| crawl | 0 | Fetching behaviour described as crawling at all |
| user agent | 0 | A request header string to match or verify |
| spider | 0 | The older word for the same thing |
What does Grok SEO actually control?
Every other engine in this category gives a site owner one lever that costs nothing and works without an account: a product token to name in robots.txt. The lever is weak, and this blog has spent two months measuring how weak, but it exists, it is documented, and it is checkable. How to get cited by ChatGPT turns on tokens OpenAI publishes. How to get cited by Claude turns on tokens Anthropic publishes. How to get cited by Perplexity turns on tokens Perplexity publishes. The same page shape cannot be written for Grok, because the first ingredient is missing.
That matters more than the size of xAI would suggest, because Grok is not a minor destination for citations. An earlier post here reports a study by the AI visibility vendor Goodie that placed Grok first among engines for news citations at 23 percent. That figure is Goodie's and is repeated from our own post rather than re-verified today, and it is the reason this gap is worth a measurement: the engine with the largest reported share of news citations is the one engine a site cannot address by name.
So the honest answer to what Grok SEO controls, on the evidence below, is the same thing that governs any unnamed fetcher: the wildcard group, the server's own response, and whatever the page itself contains. Nothing engine specific. Everything a site does for generative engine optimization still applies, and none of the per crawler configuration that the rest of this category assumes has anything to attach to. A reader who came here to add three lines to a file is going to leave with one line at most, and it is a line about AI crawlers in general rather than about Grok.
| Vendor | Tokens in our registry | Vendor documentation | URL answered on 18 Sep 2026 |
|---|---|---|---|
| OpenAI | 3 | developers.openai.com | HTTP 200 |
| Anthropic | 4 | support.claude.com | HTTP 200 |
| Perplexity | 2 | docs.perplexity.ai | HTTP 200 |
| 1 | developers.google.com | HTTP 200 | |
| Apple, Meta, ByteDance, Amazon, Common Crawl | 5 | One page each, all recorded | Not re-requested today |
| xAI | 0 | None found | No page to request |
The documentation never writes the word robots
The method here is deliberately one a reader can repeat in three requests, because a claim that a company has not published something is the kind of claim that deserves to be checkable. xAI's documentation site publishes an index at docs.x.ai/llms.txt, which lists 175 page links, and a sitemap at docs.x.ai/sitemap.xml carrying 184 URLs. It also publishes the concatenated form at docs.x.ai/llms-full.txt, 1,575,515 characters covering 182 pages, which is the file we searched. Those three URLs are written here as plain text rather than linked, because this site links only hosts registered in its own outbound policy and x.ai is not one of them.
Searching that text case insensitively for robots returns nothing. Crawler, crawl, spider, user agent and user-agent return nothing. GrokBot and Grok-DeepSearch, two of the strings that circulate in third party crawler directories, return nothing. The only match for anything resembling xAI-Grok is xai-grok-shell, the package name of the company's command line tool, which is not a crawler. The documentation is not thin: it covers models, pricing, a REST API, SDKs in four languages and an agent product, and it runs to roughly the length of a short book. The absence is specific rather than general.
What the documentation does describe, at length, is retrieval from the other side of the transaction. The web_search tool sends Grok to fetch and browse pages, and its documented parameters include allowed_domains and excluded_domains, each capped at five. Those are filters the API caller sets. A site owner cannot reach them. Read next to the way the rest of this field works, that is a design in which the scope of a fetch is decided by the person asking the question, and the llms.txt convention and robots.txt both sit outside the loop. Our own scan methodology assumes the opposite arrangement, as does every tool on this site.
Reading xAI's published documentation
- GET https://docs.x.ai/llms.txt 200, 17,023 bytes, 175 page links
- GET https://docs.x.ai/sitemap.xml 200, 184 URLs listed
- GET https://docs.x.ai/llms-full.txt 200, 1,575,515 chars, 182 pages
- grep -i robots|crawler|crawl|user agent|spider 0 matches in 200,241 words
What xAI publishes in its own robots.txt files
A company that documents no crawler still has to serve robots.txt on its own hosts, and those files turned out to say something. We requested three of them on 18 September 2026. The file at x.ai is 225 bytes: a wildcard group allowing everything except /tools/, a sitemap line, and a Content-Signal line declaring ai-train=yes, search=yes, ai-input=yes. The file at docs.x.ai is 218 bytes, a wildcard group allowing everything, and a Content-Signal line declaring ai-train=no. Two hosts of one company, published the same day, giving opposite answers on training.
Content-Signal is Cloudflare's syntax, and this site has already measured that the signal asks while the Disallow lines decide. Neither xAI file carries a Disallow that would enforce either declaration against a named AI crawler, so the difference between the two is a difference in stated preference and nothing else. It is still worth recording, because a vendor's own files are the closest thing available to a statement of what that vendor thinks these mechanisms are for.
The third file is the one that makes the point. grok.com serves 552 bytes, and inside it there is a single group naming six user agents: GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended and Applebot-Extended, with a short allow list and then Disallow: /. Every one of those six is a token in this scanner's registry, and every one belongs to a competitor of xAI. The product surface of Grok is configured against other companies' documented crawlers, using exactly the mechanism xAI's own documentation never mentions, and it names no crawler of its own for anyone else to configure against. A reader can verify all three files in a browser; the same request through our crawler page identity returned the same bytes.
x.ai/robots.txt, 225 bytes
- User-agent: *
- Allow: /
- Disallow: /tools/
- Content-Signal: ai-train=yes, search=yes, ai-input=yes
- No xAI crawler token named
docs.x.ai/robots.txt, 218 bytes
- User-agent: *
- Allow: /
- (no path disallowed)
- Content-Signal: ai-train=no, search=yes, ai-input=yes
- No xAI crawler token named
Ten robots.txt files, eight spellings of one guess
The corpus is the two committed seed frames this repository scans from, the same frames behind the crawlability study and every corpus figure published here: 392 hostnames stratified by publishing platform and 1,027 stratified by industry sector, 1,419 distinct hosts between them. Each host was asked once for https://host/robots.txt over HTTPS on 18 September 2026, redirects followed, a fifteen second timeout, no JavaScript executed, from one network location. 1,374 answered without a transport failure, 1,071 answered HTTP 200, and 1,000 of those returned a body that the scanner's own robots parser resolved into at least one group. Those 1,000 files are the base for everything below, and they named 2,240 distinct product tokens between them, a figure consistent with the 2,209 tokens counted the day before on the same corpus.
Ten of the 1,000 files name a token containing grok or xai. For scale, 172 of the same 1,000 name GPTBot. Those ten files do not agree on what the token is. Between them they use eight distinct spellings: GrokBot in five files, Grok-DeepSearch in three, xAI-Grok in three, Grok in three, and one file each for GrokAI, xAI, xai-crawler and xAI-Web-Crawler. Four of the ten hedge by writing more than one spelling in the same file. amazon.com, rappler.com and svt.se each write the same three, GrokBot, xAI-Grok and Grok-DeepSearch, which is the signature of a shared list rather than three independent decisions. yle.fi writes a different three: Grok, GrokAI and xAI.
This is what a missing specification looks like at the far end. When a vendor publishes a token the files converge on it, which is why 172 files can agree on the five characters in GPTBot. When no vendor publishes one, the files diverge, and the divergence is not evidence that site owners are careless. It is evidence that eight reasonable people read the same absence and filled it in eight different ways. Our own AI crawler directory lists what each vendor publishes precisely so that a reader does not have to guess, and it has no xAI row for the same reason.
| Spelling as written | Files | One site that writes it | Verdict at the site root |
|---|---|---|---|
| GrokBot | 5 | theregister.com | Disallowed |
| Grok-DeepSearch | 3 | amazon.com | Disallowed |
| xAI-Grok | 3 | rappler.com | Disallowed |
| Grok | 3 | france24.com | Disallowed |
| GrokAI | 1 | yle.fi | Disallowed |
| xAI | 1 | yle.fi | Disallowed |
| xAI-Web-Crawler | 1 | drugs.com | Disallowed |
| xai-crawler | 1 | capitecbank.co.za | Allowed |
Where a rule with the wrong name actually lands
A user-agent line in robots.txt is an exact product token, not a description. RFC 9309 defines the matching as case insensitive on the token, and it gives a crawler no obligation to recognise a name it does not use. A group headed by a token no crawler sends is a group that matches nothing, and the crawler falls through to the wildcard. That is the mechanism, and it is the same one that made 559 of 581 disallowed paths a wildcard decision on this corpus yesterday.
Applied to these ten files, the result is uncomfortable. Eight of the ten disallow the site root to at least one of their xAI spellings. On seven of those eight, the wildcard group in the same file allows the site root. theregister.com is the exception: its wildcard also refuses, so its refusal stands whatever name the fetcher sends. For the other seven, the refusal is conditional on a guess. If xAI's fetcher sends GrokBot, amazon.com refuses it and yle.fi does not. If it sends Grok, yle.fi refuses it and amazon.com does not. If it sends anything else, or no product token at all, all seven allow it everything.
Across the full base the wildcard is permissive almost everywhere: an unnamed crawler evaluated against the root is allowed by 981 of the 1,000 files, refused by 11, and meets no wildcard group at all in 8. So the default a nameless fetcher receives on this corpus is access. None of this says anything about what xAI's infrastructure does, which is a separate question that a robots.txt sweep cannot answer and that this post does not claim to. A user agent is a claim rather than an identity, and without a published token or a published address list there is nothing to check a claim against. If you want to see how your own file resolves for a token, real or guessed, the robots.txt tester evaluates the file we fetch rather than the one you paste.
Flow: Fetcher requests a page to Token you guessed?; Token you guessed? (exact match) to Your xAI rule applies; Token you guessed? (no match) to Wildcard group applies; Wildcard group applies to 981 of 1,000 allow root.
What a site owner can do about a crawler with no name
The useful version of this post is short, and it is shorter than the advice currently ranking for the search. Writing GrokBot into robots.txt costs nothing and may do something if that is the string xAI sends, and nobody outside xAI can tell you whether it is. Writing all eight spellings costs nothing either and raises the odds without settling them. What it must not be mistaken for is a control, and it should not be recorded anywhere as a site that has opted out of Grok, because on seven of the eight files measured here the wildcard underneath it says the opposite.
The lever that does work is the wildcard group, because that is the rule an unnamed fetcher actually receives. If the intent is to refuse fetchers that will not identify themselves, the rule has to be written at the wildcard and it will apply to everything else that falls through as well, which is a genuine cost rather than a technicality. If the intent is the reverse, which for most commercial sites it is, then the work that matters is the work that has nothing to do with tokens: pages that carry their prose in the HTML, a server that answers an unfamiliar user agent with a 200, and markup a retrieval system can lift. What GPTBot sees shows the first of those for one page, and the same reading applies to any fetcher, named or not.
We should be straight about our own position in this. This scanner evaluates 15 crawler tokens across nine vendors and has no xAI entry, so a Lantad report today says nothing about Grok. That is a gap in the product, and it is not one we can close by inventing a token: adding GrokBot to the registry on the strength of a third party directory would put a guess inside a measurement, which is the failure this blog exists to avoid. It closes when xAI publishes a page, and until it does, any AI visibility product claiming per crawler Grok coverage is either working from a private agreement or guessing the same eight strings. The rest of the field is documented: Google's crawler list and Anthropic's crawler page both name their tokens in public, and OpenAI's bot documentation has carried its three since before this blog started. One vendor out of ten is the outlier here, not the norm.
- A vendor published product token Not found in 182 pages of xAI documentation. Eight spellings circulate; none is sourced to xAI.
- A vendor published IP range file No address list was found alongside the documentation, so a claimed Grok request cannot be verified the way an OpenAI or Anthropic one can.
- A wildcard robots.txt group The rule an unnamed fetcher receives. Allows the root on 981 of the 1,000 files measured here.
- Server side response control Your origin decides what an unfamiliar user agent gets, independently of any file.
- Readable prose and markup in the HTML Applies to every retrieval system equally, and is the only item here that improves an answer rather than restricting one.
- A domain filter you can set allowed_domains and excluded_domains are documented parameters of the web_search tool, capped at five, and set by the API caller rather than by the site.
Lantad
Published .
Grok SEO is a search people type, and the tools it returns answer it the way the other engines are answered: find the vendor's crawler page, learn the product token, write the rule. That sequence works for four of the engines this site writes about. OpenAI publishes GPTBot, OAI-SearchBot and ChatGPT-User. Anthropic publishes ClaudeBot, Claude-SearchBot and Claude-User. Perplexity publishes PerplexityBot and Perplexity-User. Google publishes Google-Extended. Each of those names came from a page the vendor maintains, and each is a string a site owner can put in a file and expect to match something.
Common questions
What is the robots.txt user agent for Grok?
xAI publishes none. Lantad searched the whole of xAI's developer documentation on 18 September 2026, 182 pages and 200,241 words, and the strings robots, crawler, crawl, user agent and spider appear zero times each. The names in circulation, including GrokBot, xAI-Grok and Grok-DeepSearch, come from third party crawler directories rather than from xAI.
Does writing GrokBot in robots.txt block Grok?
Only if GrokBot is the token xAI sends, which no published source establishes. Of ten sites measured writing such a rule on 18 September 2026, eight disallowed the root to their chosen spelling, and on seven of those eight the wildcard group in the same file allowed the root, so a fetcher sending any other name was allowed everything.
Is Grok SEO different from optimising for ChatGPT or Perplexity?
The page level work is the same: prose in the HTML, a server that answers an unfamiliar user agent, and markup a retrieval system can extract. The difference is the crawler layer. ChatGPT, Claude, Perplexity and Google each publish tokens a site can name in robots.txt, and xAI publishes none, so there is no per crawler configuration step for Grok.
Does Lantad measure Grok?
Not at the crawler layer. The scanner evaluates 15 tokens across nine vendors and holds no xAI entry, so a report says nothing about whether Grok can reach a page. Adding a token sourced to a third party directory rather than to the vendor would place a guess inside a measurement, so the gap stays until xAI publishes one.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
Generative Engine Optimization (GEO)
Generative Engine Optimization (GEO) is the practice of structuring and publishing content so that generative AI engines can fetch it, read it in full, and cite it when they answer a question.
AI visibility
AI visibility is whether AI systems can access, read, and faithfully represent your web pages, the prerequisite for being cited in an AI answer.
Crawler user agent: 1,004 robots.txt files named 2,209 tokens, and 120 name one Anthropic no longer documents
A user-agent line in robots.txt is an exact product token, and a token that matches nothing is a rule that does nothing. Lantad requested /robots.txt once from each of 1,419 hostnames on 17 September 2026. 1,004 files parsed into at least one group and named 2,209 distinct tokens between them, 1,098 of which appear in a single file, 813 of which contain a character RFC 9309 does not let a crawler put in its own name, and 120 of which name anthropic-ai or Claude-Web, neither of which appears on Anthropic's crawler page.
A user agent is a claim, not an identity
Every per-crawler number you have ever seen, including ours, counts requests that said they were a crawler. Verifying the claim needs an IP check, and most tools do not do one.
GEO engine: 23 percent of news citations came from Grok, and our registry reads none of them
A study published on 28 July 2026 by the AI visibility vendor Goodie, covering 31 million AI citations recorded between October 2025 and July 2026, puts the largest share of news citations on Grok at 23 percent, ahead of AI Overviews at 19.5 and Gemini at 13.5. Lantad's own answer engine registry, read on 7 September 2026, asks Grok with no retrieval and reads no citations back from it.
