SurfSense/plugins/remote/proprietary/scraping
DESKTOP-RTLN3BA\$punk 2fb720cd03 feat(plugin-host): plugin lifespans, one license cache, and results the agent reads
- A plugin's server.py may expose a lifespan: the host enters it at startup,
  warm-ups run in the background, and scraping/lifecycle.py closes the shared
  sessions, browsers and API clients once at shutdown.
- One license cache per process, stale-while-revalidate, instead of one per
  plugin: the app's 5-minute check no longer pays a Keygen round trip per
  plugin. Refusals and the 24 h grace work as before.
- Kept API clients for Keygen, SearXNG and the captcha solver, which now polls
  every second.
- mcp_server/results.py: tool_result puts each item in the text as one
  compact JSON line, within 32 KB. opencode reads only a result's text when
  there is any, so the agent never saw prices, page text or reviews. A request
  with x-surfsense-text-only: 1 gets no structured copy.
- mcp_server/progress.py: stage progress, so a long call never sits silent.
- GET and DELETE on /<id>/mcp answer 405: the stateless host has no stream to
  offer, and each GET held an idle SSE connection open per plugin.
2026-10-14 05:56:11 -07:00
..
captcha feat(plugin-host): plugin lifespans, one license cache, and results the agent reads 2026-10-14 05:56:11 -07:00
proxy feat(plugins): a fail-fast proxied transport, keep-alive sessions and ordered fan-out 2026-10-14 05:56:10 -07:00
tests feat(plugin-host): plugin lifespans, one license cache, and results the agent reads 2026-10-14 05:56:11 -07:00
__init__.py feat(plugins): run SurfSense's Reddit scraper as its own plugin on the plugin host 2026-10-12 00:48:27 +05:30
api_clients.py feat(plugin-host): plugin lifespans, one license cache, and results the agent reads 2026-10-14 05:56:11 -07:00
browser_loop.py feat(plugins): run SurfSense's Reddit scraper as its own plugin on the plugin host 2026-10-12 00:48:27 +05:30
browser_pool.py feat(plugins): one browser pool for every scraper plugin 2026-10-14 05:56:11 -07:00
concurrency.py feat(plugins): a fail-fast proxied transport, keep-alive sessions and ordered fan-out 2026-10-14 05:56:10 -07:00
config.py feat(plugins): a fail-fast proxied transport, keep-alive sessions and ordered fan-out 2026-10-14 05:56:10 -07:00
http.py feat(plugins): a fail-fast proxied transport, keep-alive sessions and ordered fan-out 2026-10-14 05:56:10 -07:00
lifecycle.py feat(plugin-host): plugin lifespans, one license cache, and results the agent reads 2026-10-14 05:56:11 -07:00
sessions.py feat(plugins): a fail-fast proxied transport, keep-alive sessions and ordered fan-out 2026-10-14 05:56:10 -07:00
settle.py feat(plugins): one browser pool for every scraper plugin 2026-10-14 05:56:11 -07:00