Eight named reasons our scanner refuses a URL before fetching it

What a public URL box actually is

The threat is not exotic. A scanner runs somewhere, and that somewhere has a network position no visitor has. It may sit inside a private network, next to an internal admin service, or on infrastructure where a link-local address answers with instance credentials. A URL is an instruction to open a socket, and if a stranger writes the URL then a stranger chooses the socket. The attacker does not need a bug. They need the feature to work as advertised.

That is why the guard is a separate module with no network code in it at all. Everything in core/src/ssrf.ts is pure: URL normalisation, hostname rules, and address classification for both IP families. DNS is injected through a Resolver interface, which is Node's resolver in the command line tool and DNS over HTTPS in the worker. Pure logic can be tested exhaustively without a network, and the module is exercised by a dedicated spec file covering IPv4 ranges, IPv6 forms, normalisation, bare hosts, the DNS half, redirects and IP literal detection.

The placement matters as much as the logic. In the worker, the guard runs inside a single function that every outbound path funnels through: scans, compare, batch, the micro tools, the brand check, the opportunity analysis and the bulk sitemap audits. That consolidation is recent and the comment above it says why. The scanning opt-out list used to be an optional callback with two call sites, so four of five fetch paths ignored it, and the public promise on our crawler conduct page that an opted-out domain gets a notice instead of a fetch was false on most of them. A lever that works sometimes is not a lever.

None of this is unique to measuring AI crawler access. It is the ordinary cost of accepting a URL from the internet, and anyone building against our API inherits the same problem one layer up.

The URL you paste is not the URL we fetch

Almost nobody types a scheme. People paste example.com, or www.example.com/pricing, and a validator that rejects those is a validator nobody reaches. So the first step supplies one, and the rules for doing that are more delicate than they look. A string already carrying a scheme and an authority is left alone, so ftp:// is preserved in order to be rejected honestly a moment later rather than silently rewritten into something fetchable. A protocol-relative string beginning with two slashes keeps its host and gains https. A bare host with an explicit port keeps the port, because the colon there separates a port and not a scheme. An opaque scheme such as javascript: or data: is left untouched so that the scheme check refuses it rather than mangling it into a bad hostname.

Only then does validation run, and by that point the string has already changed in ways worth knowing about. The fragment is stripped, because no server ever sees it. The host is lowercased, which the WHATWG URL standard does during parsing. A trailing dot on a fully qualified name is handled when the forbidden host list is checked, so example.local. cannot slip past a suffix test that example.local fails.

The most useful thing the URL standard does here is arithmetic. Its IPv4 parser accepts hexadecimal parts after 0x, octal parts after a leading zero, and a host given as a single integer, and its serialiser always emits four decimal numbers separated by dots. So 0x7f.1, 017700000001 and 2130706433 all arrive at the range check already written as 127.0.0.1, and the guard only has to understand one form. The specification is at url.spec.whatwg.org, which is not on this site's registered outbound host list, so it is named here rather than linked. Knowing what a machine actually received rather than what a human typed is the same discipline behind what an AI crawler sees on a page, and the reason we refuse to print a grade when the fetch did not happen.

One more static rule comes straight from a specification. Embedded credentials are rejected outright, and RFC 9110, the June 2022 Standards Track document defining HTTP semantics, is explicit about why: a recipient handling an http or https URI reference from an untrusted source should parse for userinfo and treat its presence as an error, because it is likely being used to obscure the authority for the sake of phishing attacks.

Fourteen IPv4 ranges, and the document behind each

The IPv4 half of the guard is a table of 14 blocks, checked by masking rather than by string comparison. Thirteen of the 14 appear in the IANA special-purpose address registries recorded by RFC 6890, a Best Current Practice document published in April 2013 as BCP 153, which obsoleted four earlier registry documents in order to put every special-purpose block in one place. Reading the registry is a quicker route to a correct list than reasoning from memory, and it names the defining document for each entry.

Three of the blocks are the private-use ranges from RFC 1918: 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16. Two come from RFC 1122: 0.0.0.0/8, registered as this host on this network, and 127.0.0.0/8, the loopback range. One is shared address space for carrier-grade NAT, 100.64.0.0/10, from RFC 6598. One is link-local, 169.254.0.0/16, from RFC 3927, and the comment beside it in our source names the reason it is not merely tidiness: that range contains the address cloud platforms answer instance metadata on. One is the IETF protocol assignments block 192.0.0.0/24, defined in RFC 6890 itself. Three are the documentation ranges from RFC 5737, 192.0.2.0/24, 198.51.100.0/24 and 203.0.113.0/24. One is the benchmarking range 198.18.0.0/15 from RFC 2544. One is 240.0.0.0/4, registered as Reserved and citing section 4 of RFC 1112.

The fourteenth is the exception, and it is worth stating precisely rather than rounding off. 224.0.0.0/4 does not appear in the RFC 6890 unicast registry at all. It is multicast space: RFC 1112, from August 1989, defines host group addresses as running from 224.0.0.0 to 239.255.255.255. Refusing it is our decision rather than a registry entry, which is the sort of distinction this blog tries to keep visible. A constant is something a person chose, and calling it a finding would be dressing a decision as evidence.

Nothing in that list is a measurement of the web. It is a list of places a scan will not go, and its value to a reader of an AI visibility report is narrow but real: a URL that resolves into one of these ranges never becomes a number, so no report can quietly grade an internal host you did not mean to expose.

IPv6 needs eight rules, and two of them look at IPv4

An address family with 128 bits does not get a longer list. It gets a stranger one. The IPv6 half of the guard applies eight rules, and only three of them are the obvious prefix tests: fc00::/7 for unique local addresses, fe80::/10 for link local, and ff00::/8 for multicast. RFC 4193, the Standards Track document from October 2005 that defines the unique local range, states plainly that these addresses are not expected to be routable on the global internet, which is exactly the property that makes them interesting to an attacker and useless to a scanner.

Two more are single addresses rather than ranges: the unspecified address, all zeroes, and the loopback ::1. One more is the documentation prefix 2001:db8::/32, refused for the same reason the IPv4 documentation ranges are.

The last two are the ones that would be easy to miss, and both work by looking at an IPv4 address hidden inside an IPv6 one. The first is the IPv4-mapped form ::ffff:a.b.c.d, where the guard extracts the embedded 32 bits and hands them to the IPv4 rules rather than duplicating the table. The second is NAT64. RFC 6052, from October 2010, defines a Well-Known Prefix of 64:ff9b::/96 for representing IPv4 addresses inside IPv6, and it is explicit that this prefix must not be used to represent non-global IPv4 addresses, requiring translators to drop such packets. That is a rule for translators, not for scanners, so our guard checks the embedded address itself instead of trusting that everything upstream obeyed the specification.

There is also a small rejection worth naming because it is a decision rather than a rule: an address carrying a zone index, as in fe80::1 followed by a percent sign and an interface name, is refused as unparseable. A zone index is never meaningful in a URL we would scan, so treating it as invalid is safer than trying to interpret it. That is the same posture as the robots.txt tester, which reports what a file says rather than guessing what its author meant.

A redirect is a new URL, so the whole guard runs again

A guard that runs once is a guard that runs on the wrong URL. Pass a public hostname that answers 302 with a Location header pointing at a link-local address and a scanner that trusted its first check will fetch it happily. The runtime will not stop this for you, because following redirects is what the runtime is for.

So redirects are followed by hand. The fetch call sets redirect to manual, reads the Location header itself, resolves it against the current URL, and runs the complete guard on the result, DNS resolution included, before making the next request. The comment above that loop states the reason directly: the automatic behaviour would let a URL that passed the initial check bounce the fetcher to an internal host, a private IP, or the cloud metadata endpoint. The hop count is capped by FETCH_LIMITS.maxRedirects, which is set to 5 in core/src/config.ts, and exceeding it produces the too_many_redirects rejection rather than an error a caller might mistake for a network problem.

Two properties of that loop are worth separating. It is a security boundary first: every hop is a fresh untrusted URL and gets the treatment of one. It is a correctness boundary second, because a report about a page ought to describe the page that answered rather than the address that was typed. Those are different concerns that happen to share an implementation, and conflating them is how a redirect chain ends up graded as if it were one document.

Size is bounded by the same instinct. The reader streams the body and stops at the cap, cancelling the transfer rather than draining a response it is discarding, with the cap set to 3 MB of HTML. The comment records what it replaced: the scan pipeline used to buffer the entire body and check the size afterwards, with no pre-check on the declared length, so an unbounded response could exhaust the isolate. An isolate memory kill is not catchable, so the queue message retried, killed the next isolate and dead-lettered. A hardened reader already existed for other paths and had never been carried across to the one path every scan takes.

None of this decides whether an AI crawler can read your site. It decides whether we can honestly claim to have looked, which is the same reason a robots.txt result depends on what status code the file answered with and on when a crawler last re-fetched it.

The gap we document rather than close

Every guard of this shape has the same hole, and pretending otherwise would be worse than having it. The check resolves a hostname to a set of addresses, confirms all of them are public, and then makes a request by hostname. Between those two steps the name is resolved again by the HTTP client, and an authoritative nameserver under an attacker's control can answer differently the second time. That is DNS rebinding, and the source comment names it as a known limitation, accepted and documented.

Closing it properly means connecting to the vetted address directly and carrying the original hostname in the Host header and the TLS SNI, which the Workers runtime does not expose. What we have instead is a backstop rather than a fix: the worker runs on Cloudflare, whose egress cannot reach RFC 1918 space, and the guard re-runs on every redirect hop so a rebind buys one request rather than a session. That is a real mitigation and it is not the same thing as a solution, so the comment says so where an engineer will read it, and this paragraph says so where a customer will.

Writing the limitation down is the cheap part. The expensive part is the habit, and the two repairs quoted earlier are what the habit is for: an opt-out that covered two of five paths, and a size cap tested after the damage it was meant to prevent. Both were found by asking which paths a rule actually reaches rather than whether the rule exists. That is the same question behind treating a user agent as a claim rather than an identity and behind crawler detection being an ordering problem: a control that is present in the code and absent from most of the call sites reads exactly like a control that works.

For a reader deciding whether to point a scanner at their own property, the useful summary is short. A URL that resolves anywhere private is refused, and no partial result is emitted for it. A redirect into private space ends the fetch rather than downgrading the report. Nothing is stored for a domain on the opt-out list, and what we retain is written down separately. And the one weakness in the chain is named above rather than left for someone to find, which is the same commitment behind publishing that text inside shadow DOM never reaches our extractor even though it costs us a signal. Measurement tools should be legible about their own edges, and prose parity is worth nothing from a scanner that will not say what it cannot do.