- A plugin's server.py may expose a lifespan: the host enters it at startup,
warm-ups run in the background, and scraping/lifecycle.py closes the shared
sessions, browsers and API clients once at shutdown.
- One license cache per process, stale-while-revalidate, instead of one per
plugin: the app's 5-minute check no longer pays a Keygen round trip per
plugin. Refusals and the 24 h grace work as before.
- Kept API clients for Keygen, SearXNG and the captcha solver, which now polls
every second.
- mcp_server/results.py: tool_result puts each item in the text as one
compact JSON line, within 32 KB. opencode reads only a result's text when
there is any, so the agent never saw prices, page text or reviews. A request
with x-surfsense-text-only: 1 gets no structured copy.
- mcp_server/progress.py: stage progress, so a long call never sits silent.
- GET and DELETE on /<id>/mcp answer 405: the stateless host has no stream to
offer, and each GET held an idle SSE connection open per plugin.
proprietary/web_crawl/ is the backend's web crawler behind one tool, crawl, at
/web-crawl/mcp, licensed like Reddit: pages read as markdown, or a site
followed by its links. It refuses private addresses before any fetch, and its
browser tiers await the async captcha solver. The first-party list renames the
plugin from web-crawler to web-crawl, and Google Search's refusal of a page to
read now points the agent at it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proprietary/google_search/ is the backend's Google Search scraper behind one
tool, scrape, at /google-search/mcp, licensed like Reddit. Google walling every
address, with no SearXNG page to fall back on, is the tool's error; a SearXNG
page says it is not Google's ranking.
The captcha solvers move to proprietary/scraping/captcha/, async on httpx2,
with 2captcha over HTTPS, set by SCRAPING_CAPTCHA_PROVIDER and
SCRAPING_CAPTCHA_API_KEY. The shared warm-IP pool uses async Redis at
REDIS_URL, and sticky IPs come from the proxy provider, so NodeMaven holds
them too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
