Commit graph

4 commits

Author SHA1 Message Date
DESKTOP-RTLN3BA\$punk
433d2664fd fix(plugins): review fixes before merge
- Google Search: Google's sorry wall is recognised by host and path, not a
  "/sorry" substring, so a result linking to a page with "sorry" in its URL
  no longer burns the wave's exit (goto and the captcha check share
  captcha.is_sorry_url).
- Walmart: a redirect still pending after the last allowed hop is a refused
  try, so the next exit is tried instead of reading an empty page.
- Amazon: a hedge whose sticky lease is unavailable falls back to the
  gateway instead of spending the call's hedge budget on nothing.
- Fixtures: live tracking and session tokens scrubbed from the captured
  pages (Indeed, Amazon, Google, Google Maps, TikTok, YouTube).
- plugin-host.md: shared_access() everywhere, and the Reddit and Instagram
  gaps now describe their fail-fast latches.
2026-10-14 15:01:56 -07:00
DESKTOP-RTLN3BA\$punk
e64a2ae344 perf(indeed): keep a solved browser context per country, and fetch details as JSON
- No per-call home-page warm-up: go straight to the page and solve Cloudflare
  only when a challenge is there.
- One kept, country-pinned browser context holds the clearance across calls,
  so a warm call is about 2 s instead of 30-125 s.
- Job details come in batches from Indeed's own spa=1 JSON inside the page;
  a call's targets run concurrently; the inline JSON is read with raw_decode.
- Fixes: jobKey on direct /viewjob URLs, detail salaries, company rating,
  review count, job types and country on details, clean companyUrl, 0 ratings
  read as unknown, cross-target dedupe, a 403 page no longer read as empty,
  and a failed warm-up no longer recorded as done.

INDEED_KEPT_CONTEXT=0, INDEED_SPA_DETAILS=0 and INDEED_TARGET_CONCURRENCY=1
turn the new paths off.
2026-10-14 05:56:31 -07:00
DESKTOP-RTLN3BA\$punk
2107dedb2e fix(plugins): read the host's Keygen settings from the environment, as the backend does
keygen.py held Keygen Cloud's address and an empty account id, so the host could
not start with a paid plugin, and could not reach the self-hosted Keygen CE that
holds production's licenses. It now reads KEYGEN_API_URL (the accounts endpoint,
Keygen Cloud's when unset), KEYGEN_ACCOUNT_ID and KEYGEN_HOST, with the backend's
names and meanings, so a deployment gives both the same values. With KEYGEN_HOST
set it sends Host and X-Forwarded-Proto: https, as CE needs.

A 4xx or a redirect from Keygen refuses the key as invalid, with a warning in the
log, instead of failing the request with a 500. A reply with no code is no
answer, so the 24-hour grace still applies.
2026-10-13 20:36:40 -07:00
Anish Sarkar
42e039487f feat(plugins): run SurfSense's Indeed scraper as its own plugin
proprietary/indeed/ is the backend's Indeed scraper behind one tool, scrape, at
/indeed/mcp, licensed like Reddit. Its Cloudflare-solving session is Scrapling's
stealth Chromium, so the image needs no Camoufox, and each session now holds
one sticky IP, since its clearance belongs to that IP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 12:07:28 +05:30