Answer engine optimization: 250 of 647 home pages sent no signal that anything had changed

What is answer engine optimization, and which part of it can be measured?

Answer engine optimization is the work of getting a page quoted inside an answer produced by an assistant rather than listed as a blue link, and our glossary entry for AEO sets out the vocabulary the field uses. In practice it is a stack of three claims, and they are not equally checkable. The top claim is about judgement: that a model will prefer your page over another when composing an answer. Nobody outside the model vendors can verify that directly, which is why our guide to getting cited by ChatGPT is built around what a site controls rather than what a model decides. The middle claim is about the document: that clear structure, plain prose and correct markup make a page easier to lift from. The bottom claim is about mechanics: that a crawler can fetch the page, read it, and know when to come back.

Only the bottom layer is fully observable from outside, and it is the layer this scanner exists to measure. Whether a crawler can reach the page at all is a question we put to this same frame yesterday, finding that 79 of 115 sites refused a crawler their own robots.txt allows. Whether the prose survives the fetch is what our prose parity check counts. What has been missing from that set, and from this blog, is the third mechanical question. Once a crawler has your page, how does it find out that the page is no longer the one it has?

That question matters more for answer engines than it did for search, and the reason is retrieval. A search index can afford to be a few days stale because a ranked list of links degrades gracefully. An assistant composing a sentence from a cached copy of your pricing page states an old price as a current fact. The cost of staleness moved from a slightly outdated snippet to a confident wrong answer, which is the same failure mode our own methodology is written to avoid in the other direction.

So the measurable form of the freshness advice is narrow and answerable. Not "is your content fresh", which is a judgement, but "does your server tell a crawler when the representation changed, in the way the HTTP specification defines". That is a yes or no per site, and it is what we went and counted.

How a crawler is supposed to learn that a page changed

RFC 9110, the HTTP Semantics specification published in June 2022, defines two validators a server can attach to a response. Section 8.8.2 describes Last-Modified as a header field that "provides a timestamp indicating the date and time at which the origin server believes the selected representation was last modified, as determined at the conclusion of handling the request". Section 8.8.3 defines the ETag, an opaque string that identifies one version of a representation without saying anything about when it was made.

A client that holds either one can ask a conditional question on its next visit. It sends the timestamp back in If-Modified-Since, or the tag back in If-None-Match, and the server compares. If nothing has changed the correct reply is 304 Not Modified, which carries headers and no body. That is the whole mechanism, and the specification is explicit about why it exists: section 8.8.2.1 says an origin server SHOULD send Last-Modified for any representation whose modification date can be reasonably and consistently determined, "since its use in conditional requests and evaluating cache freshness can substantially reduce unnecessary transfers and significantly improve service availability and scalability".

The crawler operators agree in the one place they discuss it. Google's crawl budget documentation, carrying a last updated date of 22 July 2026, tells site owners to "support 304 (Not Modified) HTTP status codes" and says that "if a page hasn't changed since Google last crawled it, returning a 304 code tells Google to reuse the cached version, saving your server bandwidth and resources". That sentence is written about Googlebot, and we are not extending it to any AI crawler here. Most AI crawler documentation says nothing at all on the subject, in the same way that no crawler vendor documents Retry-After. The wider body of crawl budget advice tends to arrive somewhere else entirely, which we followed when we found that crawl budget advice ends in a Disallow line that AI crawlers read too.

There is a second signal, and it is the one the marketing advice usually means. A page can state a date in its own markup: a dateModified property in JSON-LD, defined by schema.org, or an article:modified_time meta tag. That is a claim the page makes about itself rather than a protocol the crawler can interrogate, which is exactly the distinction we drew about sitemap timestamps. Both are worth having and they do different jobs, so we counted both.

What we requested, and what this sample is not

The frame is the fixed list of 1,027 hostnames committed to this repository and used across this series, grouped into eight sectors of roughly equal size: government at 129 hosts, education at 130, healthcare at 123, news at 128, software at 130, ecommerce at 130, travel at 130 and finance at 127. It is weighted toward large organisations and it is not a random sample of the web, so every proportion below describes this frame and nothing wider. The same caveat applies to our crawlability study.

Each hostname received a GET for the site root over HTTPS, identifying as LantadBot with the string published on our crawler page, following up to five redirects with a twenty second timeout. Where the response carried a Last-Modified header, an ETag, or both, a second GET went out within seconds carrying exactly those values back as If-Modified-Since and If-None-Match. Nothing else differed between the two requests.

936 hostnames answered. 91 failed at the transport layer and are counted nowhere below. Of the 936, 729 returned HTTP 200, 182 returned 403, five returned 429, four returned 404, three each returned 202 and 401, two returned 426 and two returned 400. The 403s are consistent with what we found measuring AI crawler access at the server layer, and they are not a finding of this post.

Of the 729 that returned 200, 647 declared an HTML content type. Of the remaining 82, one declared text/markdown and 81 sent no Content-Type header at all, which is its own small problem and not this one. Every count from here is over those 647.

Three limits are worth stating before the numbers. First, this is a home page, and a home page is the page most likely to be assembled per request, so these results are the pessimistic end of what a site can do rather than a verdict on its article pages. Second, a 200 answer to a conditional request is not necessarily a fault: if the representation genuinely changed in the seconds between our two requests, 200 is the correct reply, and for a busy front page that is a real possibility we cannot exclude. Third, we measured what servers do with a validator, not what any AI crawler does with one. No claim here says that GPTBot or ClaudeBot sends conditional requests, because we did not observe them and their operators do not document it.

How many home pages sent a validator at all

374 of the 647, which is 57.8 percent, sent something a crawler could ask a conditional question with. 257 sent a Last-Modified header, 256 sent an ETag, and 139 sent both. The remaining 273, or 42.2 percent, sent neither, and for those the conversation is over before it starts: there is no question a crawler can ask, so the only way to discover a change is to download the page again and compare it.

The ETags split 124 weak and 132 strong. A weak ETag, prefixed W/, indicates that two representations are semantically equivalent rather than byte identical, which is the right choice for a page whose only differences between requests are a timestamp in a footer or a rotating session identifier. It is the more useful of the two for this purpose and it is the minority.

Two of the 257 Last-Modified values are not valid HTTP dates. RFC 9110 requires the IMF-fixdate form, the one that reads Tue, 15 Nov 1994 12:45:26 GMT. sonarsource.com sent 2026-08-31T03:26:36.798Z and clarin.com sent 2026-09-13T18:38:42-03:00, both ISO 8601 rather than IMF-fixdate. A strict parser rejects them, which converts a freshness signal into no signal at all, and this is the same class of defect as the non-breaking hyphen we found in a robots.txt token when reading AI company robots.txt files. The syntax is not decoration.

Sector changes the answer more than anything else we varied. Ecommerce is worst at 34 of 61 pages sending no validator, followed by healthcare at 36 of 74 and news at 46 of 97. Software is best at 39 of 114. There is no obvious technical story in that ordering, and the honest reading is that this is a property of the platform and the CDN in front of it rather than a decision anybody made. Nobody sets out to withhold a validator.

111 servers ignored the validator they had just issued

373 of the 374 answered the conditional request. 260 returned 304 Not Modified, which is the mechanism working exactly as specified. Two returned 403, mountsinai.org and barclays.co.uk, which had answered the first request with a 200 and refused the second, and the only difference between them was the two conditional headers.

The remaining 111, which is 29.8 percent of the sites that offered a validator, returned 200 and sent the entire page again. Between them they re-transmitted 72,941,121 bytes, roughly 69.6 MiB, to answer a question about whether anything had changed in the previous few seconds. The median page in that group is 440,744 bytes. The largest are news front pages: clarin.com at 4,386,437 bytes, chosun.com at 3,270,866, inquirer.com at 3,249,198, bostonglobe.com at 3,104,187 and theglobeandmail.com at 3,073,640.

Splitting the 111 by what they had offered explains most of it. 56 had sent an ETag and no Last-Modified, so the tag itself failed to match on a second request seconds later, which means the tag is derived from something that varies per response rather than from the content. Of the 55 that had sent a Last-Modified, seven carried a timestamp within five minutes of our scan and 21 within the hour, both consistent with a page regenerated on every request: the server reports the moment it built the response, so the value can never match anything. Of the rest, 15 carried a timestamp from earlier the same day, 10 one older than a day, and two were the malformed ISO 8601 values described above.

Government is the outlier in the good direction, and by a wide margin: two of the 45 government pages that offered a validator ignored it, and 43 returned 304. Ecommerce is worst, with 19 of its 27 re-sending the page, and news next at 24 of 51. The pattern fits the page rather than the sector. A page assembled fresh for every visitor cannot honestly claim it is unchanged, so a dynamic front page tends to produce a validator that never matches, and the sectors that build their front page per request are the ones at the bottom of this table. That is not a bug in the same sense as a missing header, but the effect on a crawler is identical, and it compounds with the payload problem we found measuring Next.js pages that sent 15 times more payload than prose.

The date in the markup, and the 250 pages with no signal of any kind

The other half of the freshness advice is the visible date, and it is rarer than the header. 77 of the 647 home pages, which is 11.9 percent, carried a modified date anywhere in the markup that arrived: 51 published a dateModified inside a JSON-LD block, 42 carried an article:modified_time meta tag and 15 carried an og:updated_time. Counting published dates as well as modified ones raises it only to 87. A separate 68 pages carried a time element with a datetime attribute, which is a date about something on the page rather than about the page.

None of that is damning on its own. A home page is not an article and schema.org does not ask an Organization or a WebSite to carry a dateModified, so the low count is partly correct behaviour rather than an omission. It is worth knowing anyway, because the advice to publish a modified date is given without that distinction, and because the number of home pages carrying structured data at all is much higher: we counted 241 of 382 home pages with at least one JSON-LD block two days earlier when measuring how many home pages carry structured data in the raw HTML. The blocks are there. The dates are mostly not.

Crossing the two signals is where the headline number comes from. 54 pages sent a validator and published a modified date. 320 sent a validator and no date. 23 published a date and sent no validator. 250 did neither, which is 38.6 percent of the 647, and those are the pages that tell a crawler nothing whatsoever about change through either channel. Only 43 of the 647 did the complete job of publishing a date and honouring their validator with a 304.

We should say plainly what this does and does not change about our own product, because the inconvenient version is the true one. Lantad does not score freshness. The four weighted parts of the AI Visibility Score, read from core/src/config.ts on 13 September 2026, are prose parity at 50 points, access at 25, structure at 15 and schema at 10, and there is no freshness component in that list. We have also published no evidence that any AI crawler sends conditional requests or acts on a dateModified, so adding a weight for it would be scoring a decision rather than a finding, which is the confusion our AEO and GEO reference pages exist to keep out of the report.

What to check on your own site

This is one curl away and almost nobody runs it. Request your own page, read the Last-Modified and ETag headers off the response, then request it again handing those values back as If-Modified-Since and If-None-Match, and look at the status code. A 304 with no body means the mechanism works. A 200 means your server offered a question and then refused to answer it, and every crawler that takes you up on the offer pays the full page size to learn nothing.

If you get no validator at all, the fix is usually one layer up rather than in your application. A CDN or a reverse proxy that strips or rewrites these headers is the common cause, and on a statically generated site the headers are normally there until something in front removes them. If your page is assembled per request and genuinely cannot claim to be unchanged, a weak ETag computed over the content that matters, rather than over the whole rendered response, is the honest answer.

Then decide whether the markup date is a claim you can keep true. A dateModified that updates when a template changes is worse than no date, because it tells an answer engine that stale content is current, and this is the same asymmetry that makes us refuse to guess elsewhere: a wrong signal costs more than a missing one. Our what GPTBot sees tool shows you the response a crawler actually receives, headers included, which is the view this whole study was run from.

Set against the rest of the stack, freshness is a smaller problem than access. A page a crawler cannot fetch scores nothing regardless of how carefully it announces its own updates, and the sites in this frame that returned 403 to a crawler user agent have a more urgent question to answer than this one. But among the things you can fix in an afternoon, sending a validator that means something is unusually cheap, and 250 of these 647 sites are getting nothing back from a page that changes.