- Google Search: Google's sorry wall is recognised by host and path, not a
"/sorry" substring, so a result linking to a page with "sorry" in its URL
no longer burns the wave's exit (goto and the captcha check share
captcha.is_sorry_url).
- Walmart: a redirect still pending after the last allowed hop is a refused
try, so the next exit is tried instead of reading an empty page.
- Amazon: a hedge whose sticky lease is unavailable falls back to the
gateway instead of spending the call's hedge budget on nothing.
- Fixtures: live tracking and session tokens scrubbed from the captured
pages (Indeed, Amazon, Google, Google Maps, TikTok, YouTube).
- plugin-host.md: shared_access() everywhere, and the Reddit and Instagram
gaps now describe their fail-fast latches.
- No per-call home-page warm-up: go straight to the page and solve Cloudflare
only when a challenge is there.
- One kept, country-pinned browser context holds the clearance across calls,
so a warm call is about 2 s instead of 30-125 s.
- Job details come in batches from Indeed's own spa=1 JSON inside the page;
a call's targets run concurrently; the inline JSON is read with raw_decode.
- Fixes: jobKey on direct /viewjob URLs, detail salaries, company rating,
review count, job types and country on details, clean companyUrl, 0 ratings
read as unknown, cross-target dedupe, a 403 page no longer read as empty,
and a failed warm-up no longer recorded as done.
INDEED_KEPT_CONTEXT=0, INDEED_SPA_DETAILS=0 and INDEED_TARGET_CONCURRENCY=1
turn the new paths off.
keygen.py held Keygen Cloud's address and an empty account id, so the host could
not start with a paid plugin, and could not reach the self-hosted Keygen CE that
holds production's licenses. It now reads KEYGEN_API_URL (the accounts endpoint,
Keygen Cloud's when unset), KEYGEN_ACCOUNT_ID and KEYGEN_HOST, with the backend's
names and meanings, so a deployment gives both the same values. With KEYGEN_HOST
set it sends Host and X-Forwarded-Proto: https, as CE needs.
A 4xx or a redirect from Keygen refuses the key as invalid, with a warning in the
log, instead of failing the request with a 500. A reply with no code is no
answer, so the 24-hour grace still applies.
proprietary/indeed/ is the backend's Indeed scraper behind one tool, scrape, at
/indeed/mcp, licensed like Reddit. Its Cloudflare-solving session is Scrapling's
stealth Chromium, so the image needs no Camoufox, and each session now holds
one sticky IP, since its clearance belongs to that IP.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>