BlogFindings
NLWeb: two endpoints, and a server you run rather than a file a crawler finds
Microsoft introduced NLWeb on 19 May 2025 as a way to reuse the Schema.org and RSS a site already publishes, and stated that every instance is also an MCP server. Cloudflare's deployment page for it, last updated 10 July 2026, documents an /ask and an /mcp endpoint and instructs the reader to ingest pages by crawling them.
NLWeb has a different shape, and that difference is why it earns a post here rather than a line in a roundup. It is not a file. It is a server a site owner chooses to deploy, which answers questions about that site's own content over HTTP. Microsoft introduced it as a way of reusing the structured data a site already publishes, which puts it directly next to the thing this blog measures most often, and one large platform now documents a deployment path for it. What follows is a reading of those published pages with the dates they carry, and nothing else. The last section states plainly why Lantad has measured none of it.
In short
- Microsoft introduced NLWeb on its Source blog on 19 May 2025, describing an open project that leverages semi-structured formats like Schema.org, RSS and other data that websites already publish.
- That announcement states that every NLWeb instance is also a Model Context Protocol server, allowing websites to make their content discoverable and accessible to agents and other participants in the MCP ecosystem if they choose.
- Cloudflare's NLWeb page, last updated 10 July 2026 and read on 4 September 2026, documents two endpoints, /ask as the standard conversational endpoint and /mcp as the MCP server endpoint for trusted AI agents, and describes the feature as a public preview.
- The setup instructions on that Cloudflare page tell the reader to use Website as the data source and to crawl and ingest pages with the Rendered Sites option, and name no Schema.org or RSS input anywhere in the procedure.
- Lantad has measured no NLWeb deployment and cannot observe one from outside a site: an /ask endpoint is a server an owner chooses to run at a path they choose, not a file at a fixed path that a scan of a hostname will find.
| Source | What it is | Date the page carries |
|---|---|---|
| Microsoft Source announcement | The introduction of NLWeb, naming its inputs and its MCP relationship | 19 May 2025 |
| nlweb-ai/NLWeb repository README | The reference implementation, Python, MIT licence | No date on the page |
| Cloudflare AI Search NLWeb page | A deployment path, naming two endpoints and a data source procedure | Last updated 10 July 2026 |
What is NLWeb, and what did Microsoft actually announce?
Microsoft introduced NLWeb on its Source corporate blog on 19 May 2025, at news.microsoft.com/source/features/company-news/introducing-nlweb-bringing-conversational-interfaces-directly-to-the-web/. That address is written here as plain text rather than as a link because the host is not registered on this site's outbound list, and a plain URL still lets a reader verify the quotes below.
Two sentences in that announcement carry the technical content and the rest is framing. The first names the inputs: NLWeb leverages semi-structured formats like Schema.org, RSS and other data that websites already publish. The second names the protocol: every NLWeb instance is also a Model Context Protocol server, allowing websites to make their content discoverable and accessible to agents and other participants in the MCP ecosystem if they choose. Read together, those two sentences describe a translation layer. On one side is markup a site already emits for search engines. On the other is an interface an agent already knows how to call. The proposition is that the first can be turned into the second without writing new content.
The reference implementation is a Python project under the MIT licence, and its README describes NLWeb as a collection of open protocols and associated open source tools. Read on 4 September 2026 it sits at github.com/nlweb-ai/NLWeb, under an organisation of that name rather than under Microsoft's own, again plain text for the same reason as above. That is worth one sentence only because a good deal of secondary writing still describes NLWeb as a Microsoft product rather than as an open project Microsoft started.
The MCP half is the part most likely to be misread, and this blog has already published the reason. MCP labels every request, and the label stops at the MCP server recorded that the protocol carries a client identity as far as the server it connects to and no further. An MCP endpoint is something an agent connects to deliberately, having been told where it is. It is not something an AI crawler discovers by walking your links, which is the same constraint that applied to a WebMCP interface, which needs a browser to be found.
-
Reuses existing site dataStated Names semi-structured formats like Schema.org, RSS and other data that websites already publish. -
Every instance is an MCP serverStated Phrased as making content discoverable and accessible to agents in the MCP ecosystem if they choose. -
A crawler will find it on its ownNot stated The announcement names no discovery path, no well-known URI and no directive a crawler would read. -
Changes what an AI crawler reads from your pagesNot stated Nothing in the announcement describes an effect on an ordinary crawl of the site's HTML.
Which endpoints does an NLWeb instance expose?
Cloudflare's NLWeb documentation, last updated 10 July 2026 and read here on 4 September 2026, is the most useful page on the subject for a site owner, because it is a vendor describing how to actually stand one up rather than a project describing what it is for. It calls NLWeb an open project developed by Microsoft that defines a standard protocol for natural language queries on websites, and its purpose making websites as accessible and interactive as a conversational AI app.
That page documents two endpoints. The first is /ask, which it calls NLWeb's standard conversational endpoint, and which it says powers the conversational interface at the root path and the embeddable preview widget at /snippet.html. The second is /mcp, which it calls NLWeb's MCP server endpoint for trusted AI agents. The split is the interesting part. One endpoint is for a person typing a question into a box on your site. The other is for a machine that has been pointed at you. They serve the same content through two different front doors, and only one of those doors is a thing a human visitor ever sees.
The page describes the feature as a public preview and ideal for experimentation, which is the vendor's own framing and worth repeating rather than paraphrasing away. A public preview is not a commitment, and a post that reported this as a settled part of the web would be overstating what the page says.
It is also worth noting where this sits in the same vendor's product line. Cloudflare has been shipping agent-facing infrastructure at a fast rate: this blog covered an agent browser that is not Chromium and, more recently, found that only one of ten Browser Run endpoints reads Content Signals. NLWeb is the same pattern from the other side. Those products are about a machine reading somebody else's site. This one is about your site answering a machine. Whether that trade is worth making is a question about your traffic and your AI visibility, not one this page can settle for you.
Sample Illustrative, not a measurement of any real site.
Flow: Person, question in a box to POST /ask; AI agent, told the address to POST /mcp; POST /ask to NLWeb instance; POST /mcp to NLWeb instance; NLWeb instance (retrieve) to Ingested content; Ingested content to Answer, JSON.
Does NLWeb read the Schema.org markup you already publish?
This is the question a reader of this blog will actually have, because it decides whether NLWeb is a reason to go and mark up your pages. The announcement's answer and the deployment page's answer are not the same answer, and the gap between them is the finding of this post.
The announcement names Schema.org first among the formats NLWeb leverages, and Schema.org is the vocabulary this site scores against. The reference implementation's own documentation describes returning results as Schema.org typed objects, so a recipe comes back as a Recipe and a restaurant as a Restaurant. On that description, an NLWeb instance is a query layer sitting on top of JSON-LD a site is already emitting, and the markup is the product rather than a decoration.
The Cloudflare procedure does not work that way. Its setup instructions tell the reader to use Website as the data source option and to crawl and ingest pages with the Rendered Sites option. Read carefully on 4 September 2026, the instructions on that page name no Schema.org input and no RSS input anywhere in the procedure. The only Schema.org on the page is the TechArticle block in its own head, which is markup describing the documentation rather than guidance about yours. So the largest platform path currently documented for NLWeb ingests rendered HTML, which is the same thing an ordinary crawler takes, rather than the structured data the announcement leads with.
That is not a contradiction and it should not be reported as one. NLWeb is a protocol with more than one implementation, and a vendor is free to fill the index by crawling. It does mean a site owner cannot read the announcement, conclude that marking up pages is the prerequisite, and be confident that is true of the path they will actually use. It is also a familiar shape. This blog has repeatedly found the gap between what markup promises and what a consumer of it does: 70 of 146 markup values were not on the page at all, a valid BreadcrumbList scored zero because of its syntax, and across the whole web only 16 Schema.org types reach 10 million domains. Markup is only ever worth what the thing reading it does with it.
| Input | Named in the announcement | Named in the Cloudflare procedure |
|---|---|---|
| Schema.org markup | Yes, named first | No |
| RSS | Yes | No |
| Other data websites already publish | Yes, unspecified | No |
| Crawled and rendered pages | Not named | Yes, the documented route |
Why an /ask endpoint is not the same kind of control as robots.txt
Every control this blog usually writes about shares one property that is easy to stop noticing: it is passive. You write a file, you put it at the path the specification names, and from then on the work is done by somebody else's crawler arriving and reading it. You do not run anything. You do not scale anything. You do not pay per query. If nobody comes, the file costs you nothing.
An NLWeb instance is the opposite on every one of those axes. It is a running service with a model behind it and a bill attached, it answers only questions that arrive at an address somebody already knows, and it does nothing at all until it is deployed and pointed at. That is not a criticism of the design, because a conversational interface genuinely cannot be a static file. It does mean the two things belong in different mental categories, and a site owner who files NLWeb next to llms.txt in the same drawer marked things to publish for AI will be surprised by the operational difference.
The discovery gap is the sharper half. A crawler finds your robots.txt because RFC 9309 fixes the path. It finds your llms.txt for the same reason, which is precisely why llms.txt v2 spent its effort on link relations and discovery. Nothing in the pages read for this post fixes a path for an /ask endpoint, names a well known URI for it, or defines a directive by which a page could advertise one. An agent reaches your instance because a human told it the address, or because a platform that hosts you routes it there.
Which leads to the uncomfortable comparison this site has made before and should keep making. Lantad ships a tool for llms.txt and has published, more than once, that the evidence for llms.txt doing anything measurable is thin. The honest position on NLWeb today is narrower still, because there is not even a contested measurement to argue about: it is an open protocol with a reference implementation, a vendor preview, and no published evidence this site has found on whether deploying one changes how often a site is cited. Anyone telling you otherwise in September 2026 is ahead of the record.
A file a crawler fetches
- Path fixed by a specification, so it is findable
- robots.txt, llms.txt, sitemap.xml, the proposed ai.txt
- Costs nothing to serve and nothing when unused
- Observable by any scan of the hostname
- Read without the site agreeing to anything first
A server you run
- Path chosen by the operator, no standard location
- NLWeb /ask and /mcp, an arbitrary MCP endpoint
- A running service with a model bill behind it
- Not discoverable from outside without being told
- Answers only clients that already know the address
What this changes for what Lantad measures, and what it does not
Nothing, is the short answer, and the reason matters more than the answer. This scanner takes a hostname and looks at what a crawler gets when it asks that host for pages: the robots.txt verdict per crawler token, whether the words a person sees survive into the HTML a crawler receives, and what the structured data declares. The methodology page sets out the whole procedure. An NLWeb instance sits outside all of it, because a scan finds files at known paths and an /ask endpoint is not at a known path.
That is the honest limit and it should be stated rather than worked around. Lantad has measured no NLWeb deployment, holds no data on how many sites run one, and cannot tell you whether the sites that do are cited more often. This post is a reading of three published pages on 4 September 2026 and it is not a measurement of anything.
What the scanner does measure remains the input side of the announcement's own proposition, and there the published record is not encouraging. Parsed out of five real pages this repository captured on 15 July 2026, three of five carried any JSON-LD at all, those blocks held 102 typed nodes between them, one page contributed 96 of the 102, and only 17 of the 102 carried a type this scanner recognises as a business fact. If Schema.org is the data model, then on that small sample the model is thin or empty for most of the pages, and standing up a query layer over nothing returns nothing. Five pages is five pages and not a survey of the web, a caveat that post makes at length.
So the practical reading, for somebody who came here wondering whether to act. Deploying NLWeb is a product decision about whether you want a conversational interface on your own site, and it is a reasonable one to make on its own merits. It is not an AI visibility fix, and it will not change what GPTBot sees when it fetches your page. If your reason for considering it is that you want to be read and cited by answer engines, the cheaper work is the same work as last month: make the crawler's copy of the page carry the same words the visitor's copy does, which is what prose parity measures, and get the markup right first. Our crawlability study is where that evidence sits.
Lantad
Published .
Almost every AI visibility control this blog has taken apart is a file that a crawler fetches without being invited. A robots.txt sits at one fixed path and a crawler reads it before it reads anything else. An llms.txt sits at another and waits. The ai.txt draft filed with IANA proposes a third at a third path. In every case the site publishes a document, the crawler comes to it, and the exchange needs no prior agreement between the two parties.
Common questions
What is NLWeb?
NLWeb is an open project Microsoft introduced on 19 May 2025 for building natural language interfaces on websites. Its announcement describes it as leveraging semi-structured formats like Schema.org, RSS and other data that websites already publish, and states that every NLWeb instance is also a Model Context Protocol server. The reference implementation is a Python project under the MIT licence at github.com/nlweb-ai/NLWeb.
Which endpoints does NLWeb expose?
Cloudflare's NLWeb documentation, last updated 10 July 2026, names two. /ask is described as NLWeb's standard conversational endpoint, powering the conversational interface at the root path and an embeddable widget at /snippet.html. /mcp is described as NLWeb's MCP server endpoint for trusted AI agents. That page describes the feature as a public preview.
Do I need Schema.org markup to use NLWeb?
The two published pages read for this post do not agree. Microsoft's announcement names Schema.org first among the formats NLWeb uses. Cloudflare's setup procedure, read on 4 September 2026, tells the reader to use Website as the data source and crawl and ingest pages with the Rendered Sites option, and names no Schema.org or RSS input in the procedure. The answer therefore depends on which implementation you deploy.
Will running NLWeb make my site more visible to ChatGPT or Perplexity?
There is no published evidence this site has found either way, and Lantad has measured none. An /ask endpoint sits at a path of the operator's choosing rather than a standard location, so an AI crawler does not discover it while fetching your pages, and nothing about deploying one changes what that crawler reads from your HTML. Treat it as a product decision about your own site's interface rather than as a visibility control.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
Structured data for AI (schema markup)
Structured data is schema.org markup, usually written as JSON-LD, that labels a page's facts in a machine-readable format so AI systems can extract and attribute them without having to interpret the surrounding prose.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
AI visibility
AI visibility is whether AI systems can access, read, and faithfully represent your web pages, the prerequisite for being cited in an AI answer.
llms.txt
llms.txt is a proposed plain-text markdown file placed at a site's root that gives AI systems a curated map of its most important content, so models can find and read what matters without parsing every page.
