Commit graph

111 commits

Author SHA1 Message Date
Claude
cc8edf87b6
fix(license): refuse trial emails on the disposable-email-domains list
The trial check refused 20 hand-picked domains by exact match, so trial
spam walked around it with any other throwaway provider, any subdomain
of a listed one, or a punycode spelling.

Refuse the community disposable-email-domains list (~9.2k domains, CC0),
installed as its official PyPI package and pinned in uv.lock, plus the
three hand-picked domains it lacks. Match every parent domain short of
the TLD, as upstream requires, and look up the punycode form, since
EmailStr hands the route Unicode. Purchases and resends are unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KrM98FAv8YT3boAVJpf6PW
2026-09-30 23:10:24 +00:00
DESKTOP-RTLN3BA\$punk
a7d797d2c1 chore: point code comments at the moved docs
Comments, docstrings and .env.example notes that cited moved or
deleted plan files now cite their new homes under docs/.

The three llama.cpp checksums in fetch-llamacpp.mjs and a URL-format
example in the backend config are marked `pragma: allowlist secret`.
Touching those files puts them in front of detect-secrets, and both
are false positives.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-22 21:27:27 -07:00
CREDO23
c58b4296ac Merge branch 'pr-1777' into sunset-fixes
# Conflicts:
#	surfsense_local/frontend/src/components/ui/icons.tsx
#	surfsense_local/frontend/src/features/updates/update-settings.test.tsx
#	surfsense_local/frontend/src/features/updates/update-settings.tsx
#	surfsense_web/lib/auth-utils.ts
2026-09-17 07:06:57 +02:00
CREDO23
e2c277573f feat(license): send the app download link with the licence, and run trials for 30 days
A trial is handed to an email address, not to an installation, so the email now opens with an install link built from NEXT_FRONTEND_URL. Deployments without a portal keep the original steps rather than printing a dead link.

Trial length moves to 30 days so the week before the first plugin ships does not eat half of it. The number is read from LICENSE_TRIAL_DAYS everywhere, including the subject line and the pricing copy, which previously hardcoded 14.
2026-09-17 04:50:27 +02:00
Anish Sarkar
1333c263b3 feat(sunset): refuse writes and redirect the app when SUNSET_MODE is on 2026-09-14 23:54:27 +05:30
Anish Sarkar
cf5d658468 refactor(mail): make the SMTP connection deployment-wide, drop the console transport 2026-09-14 00:51:51 +05:30
Anish Sarkar
e0eb3b0c2f chore(config): add Keygen, license mail and rate limit settings
LICENSE_MAIL_TRANSPORT selects how mail leaves the process (null, console, smtp), never which vendor delivers it: every provider exposes SMTP, so changing company means editing the connection strings and leaving this alone.
2026-09-14 00:07:57 +05:30
Anish Sarkar
3adf478720 Merge remote-tracking branch 'upstream/dev' into feat/workstream-b 2026-09-13 11:43:54 +05:30
thierry
b64f28635c revert: cloud pricing and no-login chat retirement 2026-09-12 21:03:52 +02:00
Anish Sarkar
b65004f99f feat(license): implement license purchase and issuance system 2026-09-12 11:54:00 +05:30
MODSetter
a507fe45f7 feat: changed pricing and retired no login chat 2026-09-06 22:09:08 +05:30
CREDO23
ebbea423ee Merge upstream/dev into observability refactor
Reconcile the observability module refactor (god-files split into
core/signals/setup/domains) with 86 upstream commits that kept evolving the
now-deleted otel.py / metrics.py / bootstrap.py.

Ported upstream's net-new telemetry into the new layout instead of dropping it:
- drift-sweep + per-workspace drift-check spans and git-remote
  connect/push/sweep spans + counters -> domains/knowledge_store
- video render/admission/segment/verify metrics -> domains/media
- OTEL_EXPORTER_OTLP_LOGS_ENDPOINT added to core.config.is_configured
  (log export itself already lived in setup/providers)

Re-swept upstream call-sites that referenced the deleted modules onto domain
imports (remote facade + push task, deliverables sandbox, artifact
verification, drift monitor) and updated the two upstream tests that patched
the old module attributes. Deleted otel.py/metrics.py/bootstrap.py and the
stale test_otel.py.

Verified: 4688 tests collect with zero import errors; observability, sandbox,
artifacts, and knowledge-store-remote unit suites pass. The drift-monitor
integration test needs a live Postgres and is unaffected by this merge.
2026-08-29 17:09:16 +02:00
Anish Sarkar
8180b36a0b Reapply "feat: Git remote exports, Google SERP parsing fixes, & testing video sandbox" 2026-08-29 19:51:37 +05:30
Anish Sarkar
dc2a75f8b9
Revert "feat: Git remote exports, Google SERP parsing fixes, & testing video sandbox" 2026-08-29 19:43:00 +05:30
Anish Sarkar
0c0925ea7e Merge main into dev 2026-08-29 19:07:56 +05:30
CREDO23
770ec76c66 refactor(observability): retire the PostHog $ai callback in favor of gen_ai OTLP
Drop build_llm_callback_handler from the chat orchestrators and anonymous
route, delete POSTHOG_AI_PRIVACY_MODE, and default LANGSMITH_TRACING off
(dev-only). The anonymous tier gains OtelSpanMiddleware so its LLM calls still
emit gen_ai.* to LGTM. PostHog now carries product analytics only.
2026-08-26 21:23:20 +02:00
CREDO23
95c8389fce chore(observability): point OTLP examples at otel-lgtm
The collector hostname and Grafana Cloud keys are gone. Set the endpoint to enable export.
2026-08-24 22:49:04 +02:00
Anish Sarkar
1d5db1105f feat(video): configure rendering limits and metrics 2026-08-24 14:27:39 +05:30
CREDO23
e5530e318f chore(docker): pass Daytona settings through to the backend service
Uncomment the DAYTONA_* env passthrough on the backend service and add
DAYTONA_SNAPSHOT_ID, so a deployment that sets SANDBOX_PROVIDER=daytona
actually forwards the key, URL, target, and snapshot into the container.
Document the snapshot name in the compose .env.example.
2026-08-19 04:22:18 +02:00
Anish Sarkar
01f5a56aa8 feat(config): update maximum sessions per workspace in environment configurations 2026-08-18 19:19:17 +05:30
Anish Sarkar
8ac898eccb feat(sandbox): introduce sandbox operation timeout configuration and refactor execution handling
- Added `SANDBOX_OPERATION_TIMEOUT_SECONDS` to configuration for controlling sandbox operation timeouts.
- Updated various components to utilize the new timeout setting, ensuring consistent timeout behavior across different sandbox providers.
- Refactored execution logic in the sandbox tools to improve error handling and cleanup processes.
- Enhanced tests to validate the new timeout functionality and ensure proper execution flow.
2026-08-18 04:32:26 +05:30
CREDO23
e21d8fbe8b docs(config): point the image reserve at the generate_image tool
The POST /image-generations endpoint it named no longer exists; the
reservation is taken by the tool.
2026-08-13 15:59:03 +02:00
Anish Sarkar
3c4126f6c5 feat(sandbox): enable sandbox by default in configuration files 2026-08-08 17:40:57 +05:30
Anish Sarkar
335c942d96 Merge remote-tracking branch 'upstream/dev' into feat/sandbox-artifacts 2026-08-08 13:20:54 +05:30
Anish Sarkar
ccafdcbe35 feat(sandbox): integrate OpenSandbox and enhance Docker workflows
- Updated backend-tests.yml to include OpenSandbox integration for running tests in a sandbox environment.
- Modified docker-build.yml to support building and testing the sandbox image.
- Enhanced docker-compose files to manage sandbox image creation and configuration.
- Added PDF generation skills and scripts for the sandbox, including quality checks and rendering capabilities.
2026-08-08 11:29:23 +05:30
Rohan Verma
48c2f96706
Merge pull request #1648 from Benebo7/feat/add-max-file-size
Feat: adjust max file size to have customizable limit
2026-08-08 11:17:38 +05:30
Anish Sarkar
7521499c18 feat(sandbox): enhance sandbox configuration for cloud execution
- Updated the .env.example file to include new sandbox settings for code execution, including SANDBOX_ENABLED, SANDBOX_PROVIDER, and session management parameters.
- Modified docker-compose files to reflect the new sandbox environment variables, allowing for easier configuration of cloud execution options.
2026-08-07 09:27:08 +05:30
CREDO23
d875431728 Merge dev_hotpatch into dev
Frontend documents right-panel rebuild, model-connections/LM Studio
discovery + context limits, chat error-handling improvements, and i18n
fixes. No KB-core changes; clean merge over the git-native KB backend.
2026-08-06 17:52:17 +02:00
Yigtwxx
927ae56974 fix(video): narrate presentations in the slide language
The video presentation agent built its Kokoro pipeline with a literal
lang_code="a" and took its voice from a hand-rolled provider map that only
ever returned American English. Slides generated from Japanese or Chinese
source content were therefore narrated by an English voice reading text it
could not pronounce.

The podcast pipeline already solved this: a voice catalog with per-language
rosters, a BCP-47 tag normaliser, and a TTS port whose Kokoro adapter maps the
tag to the right pipeline and caches one pipeline per language. The video path
simply sat outside that infrastructure. Route it through the same port instead
of duplicating a second, English-only implementation.

The language now comes from the model that writes the slides, which is the
only component that knows what language it wrote in. An absent or unusable tag
falls back to a new VIDEO_PRESENTATION_DEFAULT_LANGUAGE and finally to English,
so a bad value degrades instead of failing the render.

app/services/kokoro_tts_service.py is removed: routing through the port left it
without a caller, and it also pulled kokoro and torch into the agent's import
chain at module scope.
2026-08-05 18:47:07 +03:00
Anish Sarkar
c07c273118 refactor(env-variables): rename GEN_AI_MODEL_FALLBACK_MAX_TOKENS to SURFSENSE_UNKNOWN_MODEL_MAX_INPUT_TOKENS for clarity and consistency 2026-08-04 03:39:28 +05:30
Anish Sarkar
9c44785d4f fix(model-connections): cap discovered ollama input budgets 2026-08-04 03:25:33 +05:30
Anish Sarkar
af3897b6ec feat(llm): introduce handling for insufficient memory errors and enhance context limit management 2026-08-03 21:51:57 +05:30
Benebo7
5ef02cdecc feat: add max file size env providing customizable limitation 2026-07-30 00:06:52 -03:00
Anish Sarkar
bde2b827f2 refactor(searxng): rename SEARXNG_FALLBACK_URL to SEARXNG_URL 2026-07-27 19:58:40 +05:30
Anish Sarkar
db9af89ace docs(env): add SearXNG exposed port to the Docker environment example 2026-07-27 19:50:55 +05:30
Anish Sarkar
99538e9924 docs(env): document the searxng fallback in the docker example 2026-07-27 19:47:54 +05:30
DESKTOP-RTLN3BA\$punk
dbedf0cfa5 feat(backend): integrate PostHog analytics for enhanced observability
Some checks failed
Obsidian Plugin Lint / lint (push) Has been cancelled
- Added PostHog configuration options to .env.example files for both Docker and Surfsense backend.
- Introduced PostHog dependency in pyproject.toml.
- Implemented analytics middleware to capture various events across the application, including user authentication, automation runs, and API requests.
- Enhanced existing routes and services to emit analytics events, providing insights into user interactions and system performance.
- Ensured graceful shutdown of analytics clients in worker processes and application lifecycles.
2026-07-22 22:16:28 -07:00
CREDO23
ca317c4686 Merge remote-tracking branch 'upstream/dev' into feature-walmart-scraper
# Conflicts:
#	docker/.env.example
#	surfsense_backend/app/capabilities/core/billing.py
#	surfsense_backend/app/capabilities/core/types.py
#	surfsense_backend/app/config/__init__.py
#	surfsense_mcp/mcp_server/features/scrapers/__init__.py
#	surfsense_web/content/docs/connectors/index.mdx
#	surfsense_web/content/docs/connectors/native/index.mdx
#	surfsense_web/content/docs/how-to/mcp-server.mdx
2026-07-21 02:42:41 +02:00
CREDO23
b5cddcfb2b docs(walmart): document billing rates in env examples 2026-07-19 08:49:10 +02:00
CREDO23
91aa265afb Merge remote-tracking branch 'upstream/dev' into feature-indeed-jobs-scraper
# Conflicts:
#	README.es.md
#	README.hi.md
#	README.md
#	README.pt-BR.md
#	README.zh-CN.md
#	surfsense_backend/tests/unit/capabilities/google_maps/test_registry.py
#	surfsense_backend/tests/unit/capabilities/reddit/test_registry.py
#	surfsense_backend/tests/unit/capabilities/youtube/test_registry.py
#	surfsense_mcp/mcp_server/features/scrapers/__init__.py
#	surfsense_web/content/docs/connectors/index.mdx
#	surfsense_web/content/docs/connectors/native/index.mdx
#	surfsense_web/content/docs/connectors/native/meta.json
#	surfsense_web/content/docs/how-to/mcp-server.mdx
#	surfsense_web/lib/connectors-marketing/index.ts
#	surfsense_web/lib/playground/catalog.ts
2026-07-18 22:11:30 +02:00
CREDO23
e19c876fbc feat: wire indeed connector into env, readmes, playground and marketing 2026-07-15 19:03:51 +02:00
Anish Sarkar
46a10f22a9 chore(env): update environment variable documentation to include Amazon scraping rates 2026-07-15 09:59:45 +05:30
CREDO23
867e940911 Merge upstream/dev into feature/tiktok-scraper
Sync with the advanced dev tip so the PR compares against current dev
(drops already-merged commits from the diff). No history rewritten.
2026-07-09 20:15:20 +02:00
CREDO23
7723b5b8b6 feat(tiktok): add tiktok.comments verb for public comment scraping
Comments load over a signed /api/comment/list XHR that TikTok serves to
anonymous sessions once the comments panel is opened (unlike profile-video and
general-search feeds), so this is a reliable verb. Given video URLs it returns
CommentItems (text, author, likes, reply counts; replies carry repliesToId),
deduped per video, capped, and degraded to an ErrorItem for empty/withheld
videos or a bad_url ErrorItem for non-video inputs.

Generalizes the browser capture over a pluggable interaction step so the
comments flow (open panel, scroll the panel to paginate) reuses the same
warm+capture scaffolding as listing/user-search. Billed per comment on a new
TIKTOK_COMMENT meter (TIKTOK_MICROS_PER_COMMENT, matching the per-comment
market), surfaced on the chat subagent alongside tiktok.scrape/user_search.
2026-07-09 18:35:26 +02:00
CREDO23
192b6dc31a feat(tiktok): add tiktok.user_search verb for account discovery
Video/general search is login-walled for anonymous sessions, but the Users
tab (/api/search/user) returns public account records without a redirect, so
this exposes the one reliably-unblocked search path. A keyword yields
TikTokProfileItems (name, followers, bio, verification), deduped per query,
capped, and degraded to an ErrorItem when a query is empty/withheld.

Reuses the browser capture (generalized over XHR markers + extractor) and the
shared profile item shape. Billed per account on a new TIKTOK_USER meter
(TIKTOK_MICROS_PER_USER), surfaced on the chat subagent alongside tiktok.scrape.
2026-07-09 18:00:40 +02:00
Dustin Persek
41edac7641 Merge remote-tracking branch 'upstream/dev' into feature/1354-embedding-base-url
# Conflicts:
#	surfsense_web/content/docs/docker-installation/docker-compose.mdx
#	surfsense_web/content/docs/manual-installation.mdx
2026-07-07 14:19:40 -04:00
DESKTOP-RTLN3BA\$punk
1fd58752a3 feat: update environment variables and enhance scraping capabilities
- Adjusted Google Maps and YouTube micro pricing in the .env.example file for better cost management.
- Introduced new environment variables for captcha solving and stealth browser hardening to improve scraping resilience.
- Removed outdated smoke test for scraper API endpoints to streamline testing.
- Enhanced anonymous chat agent's system prompt to clarify capabilities and suggest account creation for advanced features.
- Updated Reddit fetch logic to prioritize new session handling and improve resilience against IP-related issues.
- Added compacting functionality for scraper results to optimize data handling and presentation.
- Improved workspace and document management tools with clearer descriptions and enhanced functionality.
- Introduced new UI components for agent setup guidance in the web application.
2026-07-06 20:27:36 -07:00
Dustin Persek
6fafedca6a Support separate embedding base URL 2026-07-05 22:33:09 -04:00
DESKTOP-RTLN3BA\$punk
80927a2872 feat(billing): meter platform scrapers per item; consolidate web scraping onto web.crawl
Add per-item, per-platform billing for the platform-native connectors (Reddit, Google Search, Google Maps places/reviews, YouTube videos/comments) through the capability gate/charge seam. Rates are config-driven with a shared wallet-credit module (wallet_credit) and a dedicated PlatformScrapeCreditService; agent and REST capability runs now record cost_micros. Google Maps scrape dual-meters places and attached reviews.

Remove the main-agent scrape_webpage tool now that the web.crawl capability covers single-page (maxCrawlDepth=0) and site crawling. The main agent now reaches crawling via task(web_crawler, ...). Update prompts, tool catalog, receipts, skills, proprietary docs, and tests; drop the obsolete chat-turn crawl fold path.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-05 17:08:01 -07:00
DESKTOP-RTLN3BA\$punk
ab747e7a49 feat(agents): consolidate connectors under mcp_discovery; route web search through google_search
MCP consolidation:
- Route all MCP-capable connectors (Slack, Jira, Linear, ClickUp, Airtable,
  Notion, Confluence, interim Gmail/Calendar, custom MCP) through a single
  `mcp_discovery` subagent. Drive/OneDrive/Dropbox stay native to enrich the KB.
- Deprecate Discord/Teams/Luma: no viable official MCP server.

Google-only web search:
- Remove the main-agent `web_search` tool and the SearXNG platform service;
  all public web search now flows through the `google_search` subagent via task().
- Deprecate the Tavily/SearXNG/Linkup/Baidu search connectors (HTTP 410 on
  create, "Deprecated" badge); guide heavy users to the custom MCP connector.
- Remove web search from anonymous chat (pure Q&A).
- Tear SearXNG out of docker compose + install scripts; drop tavily-python
  and linkup-sdk deps and their config/env vars.

Fix:
- metrics._package_version() now swallows any metadata lookup failure. A
  malformed editable-install distribution with no `Version` field raised
  KeyError deep in importlib.metadata, and since it runs on every
  record_subagent_invoke_duration call it was crashing every task()
  delegation. Verified end-to-end against live GPT-5.4.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-04 21:06:04 -07:00