A sitemap tells a crawler where pages are, not that they changed

What Google's sitemap documentation says about lastmod

Google's page on building a sitemap, read on 8 August 2026, carries the line Last updated 2026-07-08 UTC and settles its position on lastmod in a single sentence: Google uses the lastmod value if it is consistently and verifiably, for example by comparing to the last modification of the page, accurate.

Two words in that sentence do all the work, and neither of them is lastmod. Consistently is a statement about the whole file rather than about the one URL you happen to care about. A sitemap generator that stamps today's date on all forty thousand entries every night is consistent in the ordinary English sense and is exactly what the sentence is written to exclude, because the consistency being asked for is between the claim and the page. Verifiably is the sharper of the two, and the parenthetical spells out how: by comparing to the last modification of the page. That comparison happens outside the sitemap, against the resource itself. A document cannot verify itself, so the sentence is really describing a two step process in which the sitemap proposes and something else disposes.

The same page is blunter about the two neighbouring elements. It states that Google ignores priority and changefreq values. Those elements have been in the sitemaps protocol since version 0.9, published by sitemaps.org at https://www.sitemaps.org/protocol.html, and they are the two that ask a publisher to predict rather than report. It is worth noticing which survived. The element that reports a fact about the past is conditionally used. The two that forecast the future are discarded, and the documentation does not hedge about it. The page also states a hard bound worth knowing before a site grows into it: all formats limit a single sitemap to 50MB uncompressed or 50,000 URLs.

None of that makes sitemaps pointless, and this post is not an argument against maintaining one. Discovery is a real problem and a sitemap solves it: a page nothing links to is a page most crawlers will never reach, and listing it is the cheapest fix available. The claim being examined here is narrower, which is that the file is a good way to tell a crawler a page has changed. On the only vendor documentation that addresses the question, that use is admitted conditionally and the condition is verification elsewhere. There is a related timing problem in the same neighbourhood that this blog has measured from the literature before, in when a robots.txt edit reaches a crawler: a file you control changes instantly for you and much later for everyone reading it.

For completeness, the sitemap does have one officially supported route into the robots.txt file. Google's robots.txt documentation lists four supported fields, user-agent, allow, disallow and sitemap, so the pointer line is read even though the file it points at is a separate protocol. If you want to see what a parser makes of your own file, the robots.txt tester answers for a specific path and agent.

The thing that can prove a page changed is in HTTP

If a sitemap can only propose, something has to dispose, and the mechanism for that predates every AI crawler by decades. RFC 9110, the current HTTP semantics specification, defines four pieces that together answer the question the sitemap only asks.

Last-Modified, in section 8.8.2, is defined as a header field in a response providing a timestamp indicating when the origin server believes the selected representation was last modified. ETag, in section 8.8.3, is defined as an opaque validator for distinguishing between multiple representations of the same resource when needed. If-Modified-Since, in section 13.1.3, is defined as making a GET or HEAD request conditional on the selected representation's modification date being more recent than the date provided in the field value. And 304 Not Modified, in section 15.4.5, indicates that a conditional GET or HEAD request has been received and would have resulted in a 200 response if it were not for the fact that the condition evaluated to false.

Read those four together and the difference from a sitemap is structural rather than a matter of degree. The sitemap timestamp is written by whoever generates the sitemap, which on most sites is a plugin with no visibility into whether the article body actually changed. The HTTP validator is written by the origin server at the moment it serves the resource, and the client can put it straight back on the wire and be told, by that same server, whether the copy it holds is still current. One is a claim about a resource made in a different document. The other is a claim the resource makes about itself, and the 304 is the server agreeing to be held to it.

This is also why a freshness question and a readability question are not the same question, and why answering one tells you nothing about the other. A crawler can establish perfectly that a page has changed and still receive a document with no article text in it, which is the ordinary outcome for a client that does not run JavaScript. That gap is prose parity, it is the largest weighted part of our score, and it is measured by comparing what a plain fetch returns against what a real browser assembles. The archive evidence for how common the gap is sits in what a Common Crawl archive holds without JavaScript. The fastest way to settle it for one URL is to look at what a crawler receives rather than at what your browser shows you.

The practical shape of all this for a site owner is undramatic. Serve correct validators, do not fabricate them, and let the conditional request do the work it was designed for. A crawler that gets an honest 304 has learned something it can act on. A crawler that gets a sitemap full of today's dates has learned that your sitemap generator ran.

What OpenAI, Anthropic and Perplexity document about sitemaps

The three crawler operators whose tokens dominate every AI visibility conversation publish a documentation page each, and all three were read on 8 August 2026. None of them mentions a sitemap, a sitemap.xml, or a lastmod value.

OpenAI's crawler overview documents four tokens with four purposes: OAI-SearchBot for surfacing sites in ChatGPT search, GPTBot for training, ChatGPT-User for user triggered actions, and OAI-AdsBot for validating ad landing pages. It describes how robots.txt controls each, and it states that for search results it can take around 24 hours from a robots.txt update for OpenAI's systems to adjust. Anthropic's crawler support article documents ClaudeBot, Claude-User and Claude-SearchBot, states that its bots honour industry standard directives in robots.txt, and adds that it supports the non-standard crawl-delay extension. Perplexity's bots guide documents PerplexityBot and Perplexity-User and spends most of its length on firewall configuration and IP ranges.

What that absence does and does not license is worth being exact about, because the temptation is to over-read it. It is not evidence that these crawlers ignore sitemaps. A vendor that reads a file is under no obligation to say so, and the pages are written to answer how do I block you rather than how do I feed you. What the absence does establish is narrower and still useful: if you are choosing where to spend an afternoon on the basis of what these operators have committed to in writing, robots.txt is documented in detail on all three pages and sitemap handling is documented on none of them. A control the vendor describes is a control you can hold them to. A behaviour nobody has described is a hope.

That asymmetry between what is documented and what is done runs through most of what this blog covers, and it usually resolves against the documentation rather than in favour of it. When a research team fetched as well as read, 234 of 592 sites that ban GPTBot in robots.txt served it a 200 anyway, which is the same lesson pointed the other way: the file and the system are separate, and only one of them is easy to inspect. The tokens themselves, and which of them our scanner tests, are listed in the AI crawlers reference.

What Lantad reads out of a sitemap, and what it drops

Since the argument here is that a freshness claim should be treated as unverified until something checks it, the consistent thing is to say what our own code does with one, and it is less than a reader might assume.

Lantad reads sitemaps in two places. The robots.txt parser in core/src/robots.ts carries a case for the sitemap field and collects the URLs it finds into a list, alongside the user-agent, allow, disallow and crawl-delay handling. That is a pointer, not a fetch. The second place is the bulk audit in core/src/bulkaudit.ts, which backs the whole-site runs on paid plans and is described in its own file comment as the no-render slice of the scanner across a whole sitemap. Its parser extracts loc elements with a single regular expression, follows a sitemapindex to its child sitemaps, and filters the result to the sitemap's own host, tolerating the www variant in either direction because an off-site entry in somebody's sitemap is not ours to fetch.

The part worth publishing is what it does not read. The strings lastmod, changefreq and priority do not appear anywhere in that file, verified on 8 August 2026. Neither do the HTTP validators: ETag, Last-Modified, If-Modified-Since and If-None-Match appear nowhere in the scanning library either, which means our scanner does not send conditional requests and holds no cached copy to condition on. Every page in a bulk run is fetched fresh, and the freshness of a page is not an input to anything we report. There is no freshness signal in the score. The four weighted parts are prose parity at 50 points, access at 25, structure at 15 and schema at 10, and those are decisions somebody made rather than measurements anybody took.

Reading loc and dropping everything else is a defensible choice for what a bulk audit is for, which is answering whether each page in a list is readable rather than whether it is current. It is also a limitation, and stating it is the point of this section rather than an aside. If your sitemap carries accurate lastmod values, we do not use them. If it carries fabricated ones, we do not notice. A run that covers ten thousand of your URLs will tell you what an AI crawler receives from each of them and nothing whatsoever about how recently any of them changed.

There is a specification point underneath the first of those two places that is easy to get backwards. The Sitemap line in robots.txt is not part of the robots.txt standard. RFC 9309 says that crawlers may interpret other records that are not part of the robots.txt protocol, and gives Sitemaps as its example of exactly that, with the single obligation that parsing such records must not interfere with the parsing of the defined ones. So the most widely deployed line in the file after user-agent and disallow is formally an extension that a compliant crawler is free to skip. The same sentence has been the load-bearing one elsewhere on this blog, in what a Content-signal line does and does not oblige, and it cuts both ways every time.

What to check on your own sitemap this week

Four checks, ordered by how much they change what a crawler can do with your site, and the first one is not about the sitemap.

First, find out whether your pages are readable at all before worrying about how fresh they look. A perfectly maintained sitemap pointing at pages that arrive as an empty shell to a client that does not execute JavaScript has solved a discovery problem your site did not have. The single-page version of that question is one request, and the whole-site version is what the multi-page scan and the sitemap-driven audits on paid plans exist for. This ordering is not a preference: access and text presence gate everything downstream, which is why our methodology states what a scan does not measure before it states any result.

Second, open your sitemap and check whether the lastmod values are true. The test is the one Google's documentation implies rather than any tool: pick five URLs, compare the timestamp in the file against when the page content actually last changed, and see whether the two agree. If your generator stamps the build time on every entry, then every entry is now carrying today's date and none of them is carrying information. The honest options are to fix the generator or to remove the element, and removing it is genuinely fine. An absent lastmod makes no claim, and no claim is better than a claim that will not survive being checked.

Third, check what your server sends on the response, because that is the layer with actual leverage. Fetch one of your pages and look for Last-Modified and ETag in the response headers, then repeat the request with If-Modified-Since set to that value and see whether you get a 304. A stack that answers 200 with the full body to every conditional request is not broken, but it is declining the one chance it has to tell a crawler that nothing changed. Most origins and CDNs do this correctly by default and it costs nothing to confirm.

Fourth, keep the expectations proportionate to the evidence. No crawler operator among the three read for this post has documented that it reads your sitemap, so a sitemap is not the lever that gets you cited, and treating it as one is how an afternoon disappears. The questions that decide AI visibility are further up: whether the crawlers you care about are admitted by your robots.txt, and whether your text exists in the served HTML. Both are measurable today, and both are cheaper to fix than they are to argue about. We publish what our own crawler sends and we withhold a grade when a scan could not establish enough to justify one, for the same reason this post says which four documents were read and on what date: a tool that asks you to verify your own claims should be verifiable itself.