Microsoft's AI opt out is a meta tag, not a robots.txt token

What Microsoft published, and what it never mentions

On 22 September 2023, Fabrice Canel, a Principal Product Manager at Microsoft Bing, published new options for webmasters to control usage of their content in Bing Chat. It is short, it is the guidance Microsoft put out for this purpose, and it turns on two directive names that predate generative AI by a long way: NOCACHE and NOARCHIVE.

The definitions are given in Microsoft's own words and the two are not variants of one another. Of the first, the post states that content with the NOCACHE tag may be included in Bing Chat answers, and that Microsoft will only display URL, snippet and title in the answer. Going forward, it says, only URLs, titles and snippets may be used in training Microsoft's generative AI foundation models. Of the second, it states that content tagged NOARCHIVE will not be included in Bing Chat answers and will not be linked to in the answers, and that going forward Microsoft will not use the content for training its generative AI foundation models. There is a precedence rule, which is the kind of detail that decides real deployments: if content carries both tags, the post says Microsoft treats it as NOCACHE. The weaker directive wins, which is the opposite of what most people assume when they stack two restrictions.

The post shows where these go, and it is a meta tag in the page. One worked example reads meta name robots with content noarchive and nocache together. The other splits them, a robots meta tag carrying noarchive alongside a second tag named for bingbot carrying nocache. That second form is worth pausing on. Microsoft scoped an AI usage directive to its search crawler's name rather than to a new AI specific name, which is precisely the move the other vendors did not make.

What the post does not contain is the thing a scanner would want. It does not mention robots.txt. Not as an alternative, not as a supplement, not to say the file is the wrong place. The control surface Microsoft documented for AI usage is the page, and this matters because robots.txt is the layer most site owners now reach for first, without noticing that the layer that answers is not always the one they are editing. A site owner who writes a careful robots.txt and believes they have stated a position to Microsoft has stated nothing to Microsoft at all. That is a gap in AI visibility management that costs nothing to close and is invisible until somebody names it.

A Disallow rule stops a fetch, a meta directive governs a use

These two mechanisms are often described as alternatives, as though a site owner picks whichever is more convenient. They are not alternatives. They act at different moments and on different things, and the ordering between them is the part that decides what is possible.

RFC 9309, the Robots Exclusion Protocol, published in September 2022, defines rules that decide whether a crawler may access a URI. That is its subject: access. It says nothing about what may be done with a representation once it has been fetched, because it has no way to. A Disallow rule is evaluated before the request, from a file at a fixed path, and its whole effect is that a request does or does not happen. The consequences of that design run deeper than they look, which is why a missing robots.txt and a broken one mean opposite things under the same specification.

A meta directive sits at the other end of the sequence. It is bytes in the document, so a crawler must request the page, receive it, and parse it before it can learn the directive exists. The fetch has already happened. Nothing about NOCACHE or NOARCHIVE prevents retrieval, and nothing about them is knowable from the file most people check. Google states the general form of this plainly in its robots meta tag documentation, last updated 24 March 2026, which lists the rules its own parser accepts and notes that any rule usable in a robots meta tag can also be specified as an X-Robots-Tag HTTP header. That page is Google describing Google, and it is cited here for the mechanism rather than as evidence about Microsoft: Microsoft's announcement shows meta tag examples only and does not mention the header form, so a site owner delivering nocache as a response header is doing something Microsoft has not published support for.

The practical consequence is an asymmetry no amount of care in robots.txt can fix. A Disallow rule is one statement covering a whole site, retrievable in one request by anyone. A meta directive is one statement per URL, and surveying a site's position requires fetching every page on it. There is no summary view, no canonical location, and no way for an outside observer to answer "what is this site's stance" without a full crawl. We have written before about the crawler our own scanner does not model and why that blind spot is worth naming rather than hiding; this is a second one, and it has the same shape.

Why our crawler registry has nine vendors and no Microsoft

This is the point at which the post has to talk about its own product, because the honest version of this finding is partly a limitation of ours.

Lantad keeps a registry of robots.txt product tokens in core/src/bots.ts, published openly as a crawler directory rather than held internally. Read on 30 July 2026, it holds 15 tokens across nine vendors: three from OpenAI, four from Anthropic, two from Perplexity, and one each from Google, Apple, Meta, ByteDance, Common Crawl and Amazon. Microsoft appears nowhere in it, and that is not an accident of maintenance. There is no Microsoft entry because a registry of robots.txt tokens has no row to put Microsoft's control in. The control is not of that kind.

The contrast with Google and Apple is the sharpest way to see it, because both of those companies faced the same problem and answered it differently. Google-Extended and Applebot-Extended are robots.txt tokens whose subject is generative AI training rather than ordinary search crawling: our registry annotates the first as controlling Gemini training and the second as controlling Apple Intelligence training, each pointing at the vendor's own documentation. That is close to the ground NOCACHE and NOARCHIVE cover, and Google and Apple expressed it as robots.txt names. Both of those tokens are unusual in their own right: they send no request user agent of their own, so they are among the tokens that will never appear in your logs, and our registry records a null user agent string for three of its fifteen entries for exactly that reason, those two plus the deprecated anthropic-ai. But unusual or not, they are names, and a name is something a file can address and a scanner can test.

Other vendors split the problem across several named crawlers instead. OpenAI's crawler documentation separates training from search from user initiated fetching, giving three tokens that a site can treat differently, and Anthropic and Perplexity follow the same broad shape. Whatever you think of the granularity, all of it is expressible in one file, which is why per token evaluation is the core of how our scanner works and why our own crawler policy page is published at a fixed path.

So the disclosure is this. Lantad measures robots.txt per crawler, and Microsoft's documented AI control is not reachable that way. A Lantad scan will not tell you whether your pages carry NOCACHE or NOARCHIVE, and a clean report from us says nothing at all about your position toward Copilot. We would rather write that sentence than let a report imply a coverage it does not have.

NOCACHE is not a block, it is a shorter quotation

The naming here invites a mistake, and the mistake runs in the direction site owners least want.

Both directives sound prohibitive. Only one of them is. On Microsoft's own account, a page carrying NOCACHE remains eligible to appear in Bing Chat answers; what changes is how much of it is shown, reduced to the URL, the snippet and the title. That is a restriction on quantity, not a withdrawal. A publisher who adds NOCACHE across a site expecting to be absent from AI answers will instead find themselves present in a thinner form, still cited, still linked, with less of the article carried across.

Whether that is bad news depends entirely on what the site is for, and the answer flips depending on the goal. If the objective is to stop the content being reproduced, NOCACHE is close to the weakest useful setting and NOARCHIVE is the one that matches the intent. If the objective is the opposite, which for most commercial sites it is, then NOCACHE is a hedge that keeps the site quotable while limiting how much text travels. Sites pursuing citation in AI assistants are usually trying to be quoted more rather than less, and for them the interesting reading of this pair is that the aggressive looking option, NOARCHIVE, removes them from Microsoft's answers entirely, including the link.

Two further details from the announcement are easy to miss and both cut against the intuition. The first is the precedence rule already noted: Microsoft's guidance states that content carrying both tags is treated as NOCACHE, so belt and braces produces the weaker outcome. The second is that neither tag removes a page from search results, which the post is explicit about. The controls are scoped to AI usage and leave ordinary indexing alone, which is the same separation of concerns Google and Apple built their opt out tokens to provide, arrived at by a different route.

A last note on names. The 2023 announcement is written about Bing Chat, which is what Microsoft called the product then, while its recent material discusses Microsoft Copilot and AI generated summaries in Bing. Anyone searching their own documentation or their own pages should expect both names to be in play.

Microsoft's citation reporting also sits off your site

The same pattern repeats on the measurement side, and noticing it makes the design look deliberate rather than incidental.

On 10 February 2026, Microsoft announced AI Performance in Bing Webmaster Tools as a public preview, authored by Krishna Madhavan, Meenaz Merchant, Fabrice Canel and Saral Nigam. It reports how a site's content appears in AI generated answers across Microsoft Copilot, AI generated summaries in Bing, and what the announcement calls select partner integrations. The metrics it names are total citations, average cited pages, grounding queries, citation activity for specific URLs, and visibility trends over time. Grounding queries is the one worth the most to a publisher, because it reports the phrases the system used when retrieving the content rather than the phrases a person typed.

That is a genuinely useful data set and it is also, structurally, unreachable from outside. It lives inside Bing Webmaster Tools behind an authenticated account with a verified site, which means the only party who can read it is the site owner. No external tool can produce those numbers for a domain it does not control, and any tool claiming to has produced them some other way.

This is the second time the same shape has turned up in this area, and the parallel is close enough to be worth stating: Google's exclusion setting for its AI surfaces is an account level control in Search Console rather than a file, which we wrote up as an opt out that does not live on your site. Both companies have now put an AI control or an AI report in a place that requires ownership to reach. External measurement and owner side measurement are answering different questions, and the honest position for any scanner is to say which of the two it is doing. Ours does the external half, which is also why the aggregate picture in our research is framed as what is observable from outside rather than as the whole truth about any site.

What to check on your own pages

None of this requires a tool, and the first step is one request against a page you already own.

Fetch one of your important URLs and read the head of the document for a meta tag named robots, and for one named bingbot. You are looking for the strings nocache and noarchive. Most sites will find neither, which is the default and means full eligibility on Microsoft's account of it. The failure mode worth hunting is the opposite one: a noarchive that arrived years ago as a generic anti scraping measure, written when the directive only governed a search cache, now carrying a meaning about AI answers that nobody at the company has revisited. That is a real risk on older sites and on anything built from a template with opinionated defaults.

Then check more than one page. This is the part that has no shortcut, and it is the whole practical difference between a page level control and a file level one. Meta tags are commonly injected by a layout or a plugin with per template overrides, so the tag on a product page and the tag on an article can differ without anyone having decided that they should. If you want to see what a crawler receives rather than what your CMS believes it sends, fetch a page the way a bot would and read the response you actually get.

Two cautions on interpreting what you find. Any per crawler control that keys on a name, whether in robots.txt or in a meta tag scoped to bingbot, rests on the request identifying itself honestly, and a user agent is a claim rather than an identity. More importantly for this subject: reading a tag tells you what your site asks for, not what any system does with it. Everything above is transcribed from what Microsoft has published about its own behaviour. We have not measured Copilot, and treating a vendor's stated policy as a verified outcome would be the same error as printing a grade for a page we could not measure.

The transferable lesson is smaller than the detail suggests. When you want to know a site's stance toward a given AI system, find out which layer that vendor put the control in before you go looking, because the answer varies by vendor and the file everyone checks first is only sometimes the right one.