mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-11-01 11:21:31 +00:00
e1d734f2a7
Shared by the scraper plugins: - scraping/http.py: one attempt per request with a split (connect, read) timeout, no hidden Scrapling retries or sleeps on the same exit, raw bytes back (no lxml round trip on the event loop), and one perf line per request. FAIL_FAST gives the same timeouts to call sites still on Scrapling. - scraping/sessions.py: a per-job keep-alive SessionHolder that rotates or burns its exit on a block, sticky exits leased so no two jobs share one, and an optional, capped pool of idle sessions kept across calls. - scraping/concurrency.py: ordered_map, a bounded fan-out that yields in input order and cancels what is left when the caller stops, and hedged, a second attempt on a fresh exit when the first is slow. curl_cffi and orjson, already installed through Scrapling, are declared, and Scrapling is pinned to 0.4.15, whose private browser API the pool uses.
27 lines
1.3 KiB
Python
27 lines
1.3 KiB
Python
"""Settings every scraper plugin shares, from SCRAPING_* environment variables.
|
|
|
|
Secrets come from the deployment's secret store into the environment; none reaches the app.
|
|
"""
|
|
|
|
import os
|
|
|
|
|
|
class Config:
|
|
"""Read once at import, as the backend's settings were."""
|
|
|
|
# The proxy vendor: `dataimpulse` or `nodemaven`. Unset, scrapes go direct.
|
|
PROXY_PROVIDER = os.getenv("SCRAPING_PROXY_PROVIDER")
|
|
# The vendor's endpoint, http://user:pass@host:port. # pragma: allowlist secret
|
|
PROXY_URL = os.getenv("SCRAPING_PROXY_URL")
|
|
# The captcha solver, `capsolver` or `2captcha`, and its key. No key, no solves.
|
|
CAPTCHA_PROVIDER = os.getenv("SCRAPING_CAPTCHA_PROVIDER", "capsolver")
|
|
CAPTCHA_API_KEY = os.getenv("SCRAPING_CAPTCHA_API_KEY")
|
|
|
|
# Requests in flight through scraping/http.py at once, across every plugin: what
|
|
# the host lets the proxy plan see. A request over the limit waits its turn.
|
|
MAX_HTTP = int(os.getenv("SCRAPING_MAX_HTTP", "48"))
|
|
# Keep-alive sessions kept warm between tool calls by plugins that ask for it
|
|
# (scraping/sessions.py). 0 turns that reuse off everywhere.
|
|
MAX_IDLE_SESSIONS = int(os.getenv("SCRAPING_MAX_IDLE_SESSIONS", "16"))
|
|
# 0 turns hedging off: scraping/concurrency.hedged then runs only the first attempt.
|
|
HEDGE = os.getenv("SCRAPING_HEDGE", "1") != "0"
|