mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-10-20 11:16:59 +00:00
bdbd691c24
`crawl_url` validated its input with `validators.url` alone, which answers a syntactic question and says nothing about where the URL points. Both `http://127.0.0.1:8000/` and `http://169.254.169.254/latest/meta-data/` — the cloud metadata endpoint — pass it and are then fetched from inside the backend's own network namespace, by all three tiers in turn. Resolve the target host before any tier runs and refuse it unless every address it answers with is publicly routable. `crawl_url` is the single choke point: the spider reaches it through `_ConnectorSession.fetch`, so the same gate covers link-following, where a public page linking to an internal address would otherwise be followed. Two details the obvious implementation gets wrong: - `::ffff:127.0.0.1` reports `is_loopback == False`. The flags only read true on the mapped IPv4 form, so the mapping is unwrapped before the address is judged; reading them off the IPv6 object admits loopback under its v6 spelling. - A union of `is_private`/`is_loopback`/`is_link_local`/`is_reserved`/ `is_multicast` admits carrier-grade NAT (100.64.0.0/10, RFC 6598), which is not publicly routable. `is_global` covers that, but reports True for multicast, so multicast is excluded separately. Resolution failures refuse, so an unreachable resolver is not a way past the guard, and a host answering with both a public and a private address is refused because the fetcher resolves independently and may pick either. This narrows the reachable surface; it is not a defence against DNS rebinding, which a resolve-then-fetch guard cannot address. The malformed-URL and restricted-URL refusals carry distinct messages so the two diagnoses stay apart. DNS blocks, so it is offloaded with `asyncio.to_thread`, as the browser tiers already are. Closes #1709 |
||
|---|---|---|
| .. | ||
| platforms | ||
| web_crawler | ||
| __init__.py | ||
| LICENSE | ||