BlogFindings
MCP labels every request, and the label stops at the MCP server
The Model Context Protocol revision of 28 July 2026 requires every HTTP POST to an MCP endpoint to name its protocol version, its method and the resource it is reading. Those three headers describe one hop, and the site whose content is being read is not on it.
The Model Context Protocol asks a different question of the same plumbing, and it answers it far more rigorously than any crawler ever has. An MCP request over HTTP is required to say what protocol version it speaks, what method it is calling, and which tool, resource or prompt it is naming, and a server has to reject the request if any of those disagree with the message body. That is more structured self-identification than an AI crawler has ever been asked for. It also arrives at a machine that is usually not yours. This post reads the specification revised on 28 July 2026 and Cloudflare's detection post of 14 August 2026, and separates the hop those headers describe from the hop a site owner actually sees. Lantad has measured nothing here: it observes no MCP traffic, holds no volume figure, and everything below is read from published documents on 15 August 2026.
In short
- The Model Context Protocol specification revised 28 July 2026 requires every HTTP POST to an MCP endpoint to carry an MCP-Protocol-Version header and an Mcp-Method header, and requires a tools/call, resources/read or prompts/get request to also carry an Mcp-Name header naming the tool, resource or prompt.
- Those headers are copies rather than the record: the specification states that all protocol metadata travels in the message body, that the body remains the source of truth, and that a server must reject a header disagreeing with the body using HTTP 400 and JSON-RPC error code -32020, HeaderMismatch.
- Cloudflare published MCP detection built on that header on 14 August 2026 and wrote in the same post that its presence is a strong positive indicator of MCP, and that its absence does not prove that a request is not MCP.
- The three labelled headers describe the hop from an MCP client to an MCP server, so a site read through somebody else's MCP server receives an ordinary HTTP request carrying none of them, and RFC 9309 robots.txt matching is left with no product token to match against.
- Lantad measured none of this. Every quotation here is read from three pages of the MCP specification and from Cloudflare's post on 15 August 2026, and Lantad observes no MCP traffic and holds no figure for its volume on any site.
Flow: AI model or agent to MCP client; AI model or agent to AI crawler; MCP client (labelled POST) to MCP server; MCP server (plain request) to Your web server; AI crawler (User-Agent token) to Your web server.
What an MCP request over HTTP is required to say
The specification defines exactly two standard transport bindings. One is stdio, newline delimited messages over the standard streams of a client launched subprocess, which never touches the network at all. The other is Streamable HTTP, and it is the one a network can see. Its rules are short. The server must provide a single HTTP endpoint path, called the MCP endpoint, that supports POST. Every JSON-RPC message the client sends must be a new HTTP POST to that endpoint. The client must include an Accept header listing both application/json and text/event-stream, because the server chooses per request whether to answer with a single JSON object or a Server-Sent Events stream scoped to that request.
Then come the labels. Every POST to the MCP endpoint must include an MCP-Protocol-Version header, and the specification gives the example value 2026-07-28. Two further headers are set out in a table the specification introduces with the sentence that these headers are required for compliance. Mcp-Method carries the value of the message's method field and is required on all requests. Mcp-Name carries params.name or params.uri and is required on tools/call, resources/read and prompts/get, which is to say on every request that actually reads something. A worked example in the specification shows a weather tool call arriving with Mcp-Method: tools/call and Mcp-Name: get_weather sitting above the JSON body, and a file read arriving with Mcp-Name set to the full file URI being read.
There is a fourth family. A server may mark a tool parameter with an x-mcp-header annotation in the tool's input schema, and a conforming client must then mirror that parameter's value into a header named Mcp-Param followed by the chosen name. The specification's own example annotates the region parameter of a SQL execution tool, producing a request whose headers announce the region before any component reads the body. Values that cannot be carried as plain ASCII are wrapped in a base64 sentinel format, and the same encoding applies to Mcp-Name, so a resource URI containing non-ASCII characters still survives as a header value.
Set against that, the crawler world looks impoverished. A crawler declares one string it chose for itself, and that string is a claim rather than an identity, which is the reason cryptographic schemes exist at all and why Web Bot Auth specifies what a signature does and does not prove. Several tokens that vendors document never appear in a site's logs, and when the client is a coding agent rather than an indexing fleet, seven of nine arrived without a product name at all. MCP has more discipline about who is calling than the crawler ecosystem has ever managed.
| Header | Mirrors this body field | Required on | Arrives at your origin |
|---|---|---|---|
| MCP-Protocol-Version | io.modelcontextprotocol/protocolVersion inside _meta | Every POST to the MCP endpoint | No |
| Mcp-Method | method | All requests | No |
| Mcp-Name | params.name or params.uri | tools/call, resources/read, prompts/get | No |
| Mcp-Param plus a name | A tool parameter annotated x-mcp-header | When the server annotates that parameter | No |
Why the specification calls the body the source of truth
The reason those headers exist is stated plainly, and it is not identification. The transports overview says all protocol metadata travels in the message body, that every request carries its protocol version and client capabilities in fields under _meta.io.modelcontextprotocol, and that a binding may additionally mirror selected body fields into envelope metadata. The Streamable HTTP page gives the purpose: the transport mirrors selected JSON-RPC body fields into HTTP headers so that intermediaries, and it names load balancers, gateways and observability tooling, can route and inspect requests without parsing the body. Then the overview closes the loop with the sentence that decides how to read every header above. The body remains the source of truth.
That is a different design from the one a site owner is used to. In HTTP as crawlers use it, the header is the whole claim, because there is nothing underneath it to check against. In MCP the header is a derived copy, and the specification spends real effort keeping the copy honest. The MCP-Protocol-Version value must match the protocol version field in the body's _meta, and if the values do not match the server must reject the request with 400 Bad Request and a HeaderMismatch error, JSON-RPC code -32020. Any server that processes the body must validate that header values, decoded first if they carry the base64 sentinel, match the corresponding values in the body. The failure conditions are listed: a required standard header missing, a header value that does not match the body, a header value containing invalid characters.
The specification also says why it bothers, and the reason is worth quoting because it describes a failure mode that predates MCP by decades. The validation rule exists, it says, to prevent potential security vulnerabilities when different components in the network rely on different sources of truth, and it gives the example of a load balancer routing on the header value while the MCP server executes based on the body value. Header names are constrained to the HTTP field name token syntax of RFC 9110 section 5.1, must not contain carriage return or line feed, and must be unique case insensitively within one input schema.
There is a matching instruction for anybody enforcing policy on these headers, and it applies directly to a CDN or a gateway. An intermediary that routes or rate limits on the mirrored headers should verify that the MCP-Protocol-Version header indicates a version that requires validation of the header against the body, and should reject the request rather than trust unvalidated header values if the version is older or the header is absent. In other words the specification's own advice to intermediaries is that the label is only trustworthy when a version that polices it says so. That is a more careful position than most crawler control takes, and it is the opposite of the assumption behind an ordering problem in AI crawler detection, where what a request claims is the only thing on offer.
POST /mcp HTTP/1.1
- MCP-Protocol-Version: 2026-07-28 must match the body
- Mcp-Method: tools/call mirrors method
- Mcp-Name: get_weather mirrors params.name
- _meta io.modelcontextprotocol/protocolVersion source of truth
- _meta io.modelcontextprotocol/clientInfo who is calling
- header disagrees with body 400, code -32020
Cloudflare detects MCP on that header, and says what it does not prove
On 14 August 2026 Cloudflare published a post by Kenny Johnson on how it detects MCP traffic and helps secure it. The mechanism is the header above. Cloudflare Gateway identifies MCP requests from protocol level signals, and customers can act on the result with a selector the post gives as experimental.is_mcp equals true. The stated purpose is to find what the post calls shadow MCP traffic inside an organisation, to enforce access through approved server portals, and to block direct connections on managed network paths.
Two sentences in that post are the reason it belongs in a measurement blog rather than a product roundup. The first records what the specification requires, and it names both revisions: the MCP 2025-11-25 specification says clients must include MCP-Protocol-Version on every HTTP request after initialization, and the MCP 2026-07-28 specification goes further and requires it on every POST request. The second is the caveat, and it is unusually direct for a vendor announcing a detection feature. Its presence is a strong positive indicator of MCP, and its absence does not prove that a request is not MCP.
The post then lists the ways absence happens. The initial request from a legacy client may not contain the header. Protocol versions earlier than 2025-06-18 did not define it. Local stdio, custom transport, or nonconforming traffic may never carry it. The specification agrees from the other side: a server that supports clients on versions earlier than 2025-06-18 may treat a request that omits the header as protocol version 2025-03-26, which means an unlabelled request is a legitimate thing for a compliant server to accept.
A vendor publishing the boundary of its own detector alongside the detector is the pattern worth copying, and it is the same discipline as reporting what a scanner refuses to grade rather than inventing a number to fill the gap. It also matters for anyone reasoning about volume. A detector that finds a strong positive signal and admits it misses an unknown quantity cannot produce a denominator, so no percentage of MCP traffic derived from it would be honest, and this post publishes none. That is the same reason an agent readiness score that names four check groups and counts three is worth reading twice before quoting.
-
Header present on a POSTStrong positive Cloudflare states that its presence is a strong positive indicator of MCP. -
Header absentProves nothing The same post states that its absence does not prove that a request is not MCP. -
First request from a legacy clientMay be unlabelled Named in the post as a case where the initial request may not contain the header. -
Protocol earlier than 2025-06-18Header undefined Those revisions did not define the header, and a server may read the omission as 2025-03-26. -
stdio or a custom transportNever carries it The post names local stdio, custom transport and nonconforming traffic as cases that may never carry it.
The hop your origin actually sees, and why it carries no label
Now put the two together, because this is the part that decides what any of it means for a website. Every header in this post travels on a request from an MCP client to an MCP server. That is the only direction the transport defines. The specification is explicit that a binding must deliver client sent requests and notifications to the server and server sent responses and notifications to the client, and that no other message direction exists.
An MCP server is a program that exposes tools. Some of those tools read the web. When a model calls one, the labelled POST ends at that server, and the server then makes its own outbound HTTP request to fetch the page. That second request is not MCP. It is an ordinary GET issued by whatever HTTP client the server was built on, carrying whatever User-Agent that library sets by default, and nothing in the three specification pages read for this post says anything about it, because it is outside the protocol. The site being read is one hop further along than the protocol reaches.
So the asymmetry is complete and it runs the wrong way for a publisher. The request that is meticulously labelled goes to somebody else's server. The request that arrives at your origin carries none of that structure. You cannot see which model triggered the fetch, which tool was called, or that MCP was involved at all, and the specification is not withholding that information so much as never having been in a position to provide it. This is the same shape as an AI browsing agent that presents itself as an ordinary browser, and the same shape as the finding that a robots.txt block did not stop 12 of 18 AI chatbots, where the request reaching the site had no relationship to the name the site had written a rule about.
It is worth being precise about the one case where the headers do arrive. If your own site hosts an MCP endpoint, then you are the MCP server, the POSTs land on you, and every label in the first section is yours to read and enforce. That is a real and growing configuration: Cloudflare's developer preview that gives any site a WebMCP interface inserts a script into HTML responses, and Lighthouse now audits for agentic browsing signals. But hosting an MCP endpoint is a deliberate engineering decision, and the overwhelming majority of sites have not made it. For them the MCP channel is entirely inbound through third parties, and entirely unlabelled at the door.
Client to MCP server
- POST to a single MCP endpoint
- MCP-Protocol-Version on every POST
- Mcp-Method names the call
- Mcp-Name names what is read
- clientInfo in the message body
- Mismatch rejected with code -32020
MCP server to your origin
- An ordinary HTTP request
- No MCP header of any kind
- User-Agent set by an HTTP library
- No record of which tool was called
- No record of which model asked
- Undefined by the specification
Does robots.txt apply to MCP traffic?
The honest answer has two halves, and both come from reading RFC 9309, the Robots Exclusion Protocol published in September 2022 by Martijn Koster, Gary Illyes, Henner Zeller and Lizzi Sassman.
The first half concerns matching. RFC 9309 groups rules by a name, and it says crawlers set their own name, which is called a product token, to find relevant groups, with the token restricted to letters, underscores and hyphens. A robots.txt group therefore needs a token to address. The three specification pages read for this post define no crawler product token, name no User-Agent requirement, and never mention robots.txt. So there is nothing for a group to be headed by. Whether the fetch a given MCP server makes on your behalf is covered by a rule in your file depends entirely on what User-Agent that server's HTTP client happens to send, which is a property of the software rather than the protocol, and is not documented anywhere the protocol points to. Testing your file against named crawler tokens will tell you nothing about it, because the token is unknown.
The second half concerns what a rule means even when it does match. RFC 9309 disposes of that in one sentence: these rules are not a form of access authorization. A Disallow line is a request that a well behaved crawler is asked to honour. It is not enforcement, which is why 234 of 592 sites that ban GPTBot in robots.txt served it a 200 anyway, and why six of the nine vendors Lantad tracks publish exactly one crawler token, leaving the training and search split unexpressible for the other three. The file is a statement of preference addressed to names, and MCP has given it no name to address.
None of that makes robots.txt useless, and this post is not an argument for deleting it. The crawler channel is real, large and still the way most AI systems reach most content, and a robots.txt error there is a live problem regardless of what a second channel is doing. It does mean the file should be described accurately. It governs clients that declare a token and choose to obey. It does not govern an application fetching a URL, it never has, and two separate layers still decide whether an AI system can read your site. What is new is that a second, better specified protocol now sits above the fetch and stops short of it, which makes the boundary easier to see than it used to be.
| Control | Matches on | Reaches a declared AI crawler | Reaches an MCP server's own fetch |
|---|---|---|---|
| robots.txt user-agent group | A self-chosen product token | Yes, if it obeys | Only if that fetcher sends a matching token |
| MCP-Protocol-Version header | The client to server hop | Not applicable | Absent, the hop has ended |
| Server side status codes | The request as it arrives | Yes | Yes, it is an ordinary request |
| Authentication on the origin | Credentials, not names | Yes | Yes |
What Lantad measures here, and what it does not
Lantad scans the crawler channel. It fetches a URL as a named client, reads what robots.txt says to each crawler token, renders the page in a real browser, and compares the prose a crawler receives against the prose a browser receives. That is a bounded and externally observable thing, and everything the scanner reports is drawn from those two responses.
It observes nothing about MCP. It does not sit in the path of an MCP request, holds no sample of MCP traffic on any site, and has no way to tell from an external scan whether a page has ever been read through an MCP server. There is no figure in this post for how much of the web's machine traffic MCP represents, because Cloudflare's own detector reports a signal whose absence proves nothing, and a denominator built on that would be invented. Lantad has also not tested any MCP server implementation, so nothing here describes what any particular server sends when it fetches a page.
What survives the boundary is the part that was already true. Whatever reaches your content, through a crawler or through somebody else's tool call, arrives as an HTTP request and receives whatever your server gives it. If the text of the page only exists after JavaScript executes, a client that does not run JavaScript gets an empty document whether it announced itself as GPTBot or as a Python library. If a CDN rule answers an unfamiliar client with a challenge, it answers this one too. Those conditions are observable from outside, they are what the scanner measures, and they do not depend on knowing which protocol sat above the fetch. A page that is readable to an anonymous non-browser client is readable to both channels, and that is the practical reason the boundary in this post does not change the work.
Three things follow for anyone auditing their own site. Check what an unnamed, non-browser client actually receives, rather than only what a declared crawler receives, since the MCP path arrives as the former. Check your logs for fetches that match no known AI crawler token and are not browsers, and treat that residue as a population rather than noise. And describe your own controls precisely in whatever you publish about them, because the difference between a file that expresses a preference to named clients and a mechanism that enforces access is the difference this whole post turns on, and it is the same distinction an AI visibility measurement has to keep straight to be worth anything.
- What a non-browser client receives Observable from outside. Status code, body and headers returned to an unnamed client.
- Prose present before JavaScript runs Observable from outside. Applies identically to a crawler and to any server fetching the page.
- What robots.txt says to a named token Observable from outside. Only meaningful for clients that declare a token and obey it.
- Whether an MCP server has read the page Not observable. The MCP hop ended before the request reached the origin.
- How much traffic is MCP driven Not measurable here. The one published detector reports a signal whose absence proves nothing.
- Which User-Agent an MCP server sends Not specified. It is a property of the HTTP library the server was built on.
Lantad
Published .
For thirty years the way a website learned that a machine was reading it was the request envelope. A client announced a name in its User-Agent header, the site matched that name against a group in robots.txt, and the whole apparatus of crawler control was built on top of that one convention. It is a weak convention, because the name is self-chosen and unverified, but it is at least present on the request that arrives at the origin.
Common questions
Does robots.txt block MCP traffic?
Not in any reliable sense. RFC 9309 groups rules by a product token that a crawler sets for itself, and the MCP specification pages read on 15 August 2026 define no such token and name no User-Agent requirement. Whether a rule matches the fetch an MCP server makes depends on the HTTP library that server uses, which the protocol does not specify. RFC 9309 also states that its rules are not a form of access authorization, so a matching rule is a request rather than a block.
What headers does an MCP request carry?
Under the specification revised on 28 July 2026, every POST to an MCP endpoint carries MCP-Protocol-Version, and every request carries Mcp-Method. A tools/call, resources/read or prompts/get request also carries Mcp-Name, holding the tool name or resource URI. A server may additionally require tool parameters to be mirrored into headers named with an Mcp-Param prefix. All of these mirror fields in the JSON-RPC body, which the specification calls the source of truth.
Can I see MCP requests in my own server logs?
Only if your site is itself the MCP endpoint. The labelled POST travels from an MCP client to an MCP server. If a third party server reads your page for a model, the request arriving at your origin is that server's own outbound HTTP fetch, which carries no MCP header and no record of the tool call that triggered it.
How much AI traffic is MCP?
There is no honest figure available and this post publishes none. Cloudflare's detection post of 14 August 2026 states that the presence of the MCP-Protocol-Version header is a strong positive indicator of MCP and that its absence does not prove a request is not MCP, naming legacy clients, versions before 2025-06-18, stdio and custom transports as cases it can miss. A percentage computed from a detector with an unknown miss rate would be a guess.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
Cloudflare put a WebMCP interface on any site, and only a browser can find it
Cloudflare opened a developer preview on 6 August 2026 that adds a WebMCP tool interface to any site it fronts, using HTMLRewriter to insert one script reference into each HTML response. Chrome's own WebMCP documentation, last updated 2026-06-09, files tool discoverability under its limitations: clients and browsers must visit a site directly to know if it has callable tools.
A user agent is a claim, not an identity
Every per-crawler number you have ever seen, including ours, counts requests that said they were a crawler. Verifying the claim needs an IP check, and most tools do not do one.
What Web Bot Auth actually specifies, and what it does not
The IETF working group on crawler authentication listed ten active Internet-Drafts on 29 July 2026, two of them filed on 19 July. The core protocol draft carries intended status Informational, and every mechanism it describes sits in places an external scan of your site cannot reach.
Cloudflare shipped an agent browser that is not Chromium
Cloudflare published Kitesurf on 6 August 2026, a browser built for AI agents that runs in V8 isolates on Workers rather than as a Chromium process. Its own documentation reports DOM subtests passing at 97 percent and lists a bot-challenge handshake with real TLS fingerprints among the things it cannot do.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
Generative Engine Optimization (GEO)
Generative Engine Optimization (GEO) is the practice of structuring and publishing content so that generative AI engines can fetch it, read it in full, and cite it when they answer a question.
