Rohan Verma
21b0ecba64
Merge pull request #1643 from AnishSarkar22/feat/searxng-fallback
...
Build and Push Docker Images / compute_version (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cpu, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, , production, false, cpu) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cpu, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, , production, false, cpu) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu126, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, -cuda126, production, true, cuda126) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu126, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, -cuda126, production, true, cuda126) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu128, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, -cuda, production, true, cuda) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu128, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, -cuda, production, true, cuda) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_web, cpu, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-24.04-arm, linux/arm64, arm64, , runner, false, cpu) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_web, cpu, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-latest, linux/amd64, amd64, , runner, false, cpu) (push) Has been cancelled
Build and Push Docker Images / verify_digests (push) Has been cancelled
Build and Push Docker Images / create_manifest (backend, surfsense-backend, , cpu) (push) Has been cancelled
Build and Push Docker Images / create_manifest (backend, surfsense-backend, -cuda, cuda) (push) Has been cancelled
Build and Push Docker Images / create_manifest (backend, surfsense-backend, -cuda126, cuda126) (push) Has been cancelled
Build and Push Docker Images / create_manifest (web, surfsense-web, , cpu) (push) Has been cancelled
Build and Push Docker Images / finalize_release (push) Has been cancelled
feat(google_search): fall back to SearXNG when Google walls every IP
2026-07-29 15:31:41 -07:00
Anish Sarkar
bde2b827f2
refactor(searxng): rename SEARXNG_FALLBACK_URL to SEARXNG_URL
2026-07-27 19:58:40 +05:30
Anish Sarkar
c844491ddf
docs(web-search): describe the bundled searxng fallback
2026-07-27 19:50:29 +05:30
Anish Sarkar
6924d5e34c
docs(web-search): document the optional SearXNG fallback
2026-07-27 16:14:19 +05:30
Anish Sarkar
66c7a39aa8
docs(local-models): note other-local-servers base URL is used verbatim
2026-07-26 20:34:21 +05:30
Anish Sarkar
759e9a3656
docs(local-models): note LM Studio base URL is used verbatim
2026-07-26 20:34:14 +05:30
Rohan Verma
66de380e85
Merge pull request #1624 from AnishSarkar22/feat/connector-ui
...
Build and Push Docker Images / build (./surfsense_backend, cu128, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, -cuda, production, true, cuda) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, cu128, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, -cuda, production, true, cuda) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, cpu, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-24.04-arm, linux/arm64, arm64, , runner, false, cpu) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, cpu, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-latest, linux/amd64, amd64, , runner, false, cpu) (push) Blocked by required conditions
Build and Push Docker Images / compute_version (push) Waiting to run
Build and Push Docker Images / build (./surfsense_backend, cpu, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, , production, false, cpu) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, cpu, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, , production, false, cpu) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, cu126, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, -cuda126, production, true, cuda126) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, cu126, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, -cuda126, production, true, cuda126) (push) Blocked by required conditions
Build and Push Docker Images / verify_digests (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (backend, surfsense-backend, , cpu) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (backend, surfsense-backend, -cuda, cuda) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (backend, surfsense-backend, -cuda126, cuda126) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (web, surfsense-web, , cpu) (push) Blocked by required conditions
Build and Push Docker Images / finalize_release (push) Blocked by required conditions
feat(connectors): replace connector modal with a dedicated connectors layout
2026-07-23 15:26:15 -07:00
Anish Sarkar
094ef7af6f
docs(connectors): rename tools & live sources to integrations
2026-07-23 23:38:39 +05:30
DESKTOP-RTLN3BA\$punk
dbedf0cfa5
feat(backend): integrate PostHog analytics for enhanced observability
...
Obsidian Plugin Lint / lint (push) Has been cancelled
- Added PostHog configuration options to .env.example files for both Docker and Surfsense backend.
- Introduced PostHog dependency in pyproject.toml.
- Implemented analytics middleware to capture various events across the application, including user authentication, automation runs, and API requests.
- Enhanced existing routes and services to emit analytics events, providing insights into user interactions and system performance.
- Ensured graceful shutdown of analytics clients in worker processes and application lifecycles.
2026-07-22 22:16:28 -07:00
CREDO23
ca317c4686
Merge remote-tracking branch 'upstream/dev' into feature-walmart-scraper
...
# Conflicts:
# docker/.env.example
# surfsense_backend/app/capabilities/core/billing.py
# surfsense_backend/app/capabilities/core/types.py
# surfsense_backend/app/config/__init__.py
# surfsense_mcp/mcp_server/features/scrapers/__init__.py
# surfsense_web/content/docs/connectors/index.mdx
# surfsense_web/content/docs/connectors/native/index.mdx
# surfsense_web/content/docs/how-to/mcp-server.mdx
2026-07-21 02:42:41 +02:00
CREDO23
6c98bd33a8
docs(walmart): add connector docs and list in indexes
2026-07-19 09:01:11 +02:00
CREDO23
91aa265afb
Merge remote-tracking branch 'upstream/dev' into feature-indeed-jobs-scraper
...
# Conflicts:
# README.es.md
# README.hi.md
# README.md
# README.pt-BR.md
# README.zh-CN.md
# surfsense_backend/tests/unit/capabilities/google_maps/test_registry.py
# surfsense_backend/tests/unit/capabilities/reddit/test_registry.py
# surfsense_backend/tests/unit/capabilities/youtube/test_registry.py
# surfsense_mcp/mcp_server/features/scrapers/__init__.py
# surfsense_web/content/docs/connectors/index.mdx
# surfsense_web/content/docs/connectors/native/index.mdx
# surfsense_web/content/docs/connectors/native/meta.json
# surfsense_web/content/docs/how-to/mcp-server.mdx
# surfsense_web/lib/connectors-marketing/index.ts
# surfsense_web/lib/playground/catalog.ts
2026-07-18 22:11:30 +02:00
DESKTOP-RTLN3BA\$punk
bd7647b314
refactor: update README files and system prompts to include Amazon as a data source for AI agents, enhancing the platform's capabilities for live web research
2026-07-17 16:44:11 -07:00
DESKTOP-RTLN3BA\$punk
7bc02ce55b
refactor: update references from competitive intelligence to open web research across various components and documentation
2026-07-16 19:41:16 -07:00
CREDO23
5b0982f80b
docs: add indeed native connector page and listings
2026-07-15 18:57:15 +02:00
Anish Sarkar
92055846d1
docs(web): document amazon eu marketplace support
2026-07-15 15:32:58 +05:30
Anish Sarkar
441f671ebc
feat(amazon): add documentation for Amazon scraper API endpoint and input parameters
2026-07-15 10:14:15 +05:30
DESKTOP-RTLN3BA\$punk
2b018c4474
refactor: streamline TikTok and Instagram scraping logic by removing search_queries and enhancing documentation for clarity
Build and Push Docker Images / compute_version (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cpu, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, , production, false, cpu) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cpu, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, , production, false, cpu) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu126, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, -cuda126, production, true, cuda126) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu126, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, -cuda126, production, true, cuda126) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu128, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, -cuda, production, true, cuda) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu128, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, -cuda, production, true, cuda) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_web, cpu, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-24.04-arm, linux/arm64, arm64, , runner, false, cpu) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_web, cpu, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-latest, linux/amd64, amd64, , runner, false, cpu) (push) Has been cancelled
Build and Push Docker Images / verify_digests (push) Has been cancelled
Build and Push Docker Images / create_manifest (backend, surfsense-backend, , cpu) (push) Has been cancelled
Build and Push Docker Images / create_manifest (backend, surfsense-backend, -cuda, cuda) (push) Has been cancelled
Build and Push Docker Images / create_manifest (backend, surfsense-backend, -cuda126, cuda126) (push) Has been cancelled
Build and Push Docker Images / create_manifest (web, surfsense-web, , cpu) (push) Has been cancelled
Build and Push Docker Images / finalize_release (push) Has been cancelled
2026-07-13 17:11:25 -07:00
DESKTOP-RTLN3BA\$punk
e8b3692b54
chore: linting
2026-07-13 16:30:37 -07:00
Benebo7
a6c6b4b3d8
docs: Improve contributor Docker setup guide and fix broken link
2026-07-11 14:53:35 -03:00
Anish Sarkar
cb256729a3
chore: Enhance SurfSense MCP server documentation and toolset by adding TikTok scraping capabilities, including comments, user search, and trending features. Updated README and documentation to reflect the addition of 24 native tools.
2026-07-11 04:38:42 +05:30
Anish Sarkar
e38ca19b18
Merge remote-tracking branch 'upstream/dev' into feat/instagram-scraper
2026-07-11 04:31:11 +05:30
Anish Sarkar
42f9f35ec7
docs(web): document new media fields and search_type aliasing
2026-07-11 03:44:45 +05:30
Anish Sarkar
b75ab81261
docs(web): refresh Instagram connector docs
2026-07-11 03:25:30 +05:30
Anish Sarkar
69a8433910
docs(web): refresh Instagram connector and MCP docs
2026-07-11 02:21:05 +05:30
CREDO23
867e940911
Merge upstream/dev into feature/tiktok-scraper
...
Sync with the advanced dev tip so the PR compares against current dev
(drops already-merged commits from the diff). No history rewritten.
2026-07-09 20:15:20 +02:00
CREDO23
6b15d610d9
feat(tiktok): surface comments, user_search, trending across all product surfaces
...
Brings the three newer verbs to parity with tiktok.scrape everywhere it was
wired: MCP tools (+ selfcheck manifest), the chat subagent prompt/description,
the playground catalog, the native docs page, and the SEO marketing page. Also
adds live e2e stages for comments, user search, and trending. Docs/FAQ now
state the real contract: profile metadata is reliable while its video list can
be withheld, and keyword video search is walled (use user search for accounts).
2026-07-09 19:09:00 +02:00
Anish Sarkar
ba83d16cba
docs(web): add Instagram to MCP server tool docs
2026-07-09 16:07:50 +05:30
Anish Sarkar
f5aad487ac
docs(web): document Instagram native connector
2026-07-09 16:07:42 +05:30
Rohan Verma
3eb485ff31
Merge pull request #1575 from SyntaxSawdust/feature/1354-embedding-base-url
...
Build and Push Docker Images / compute_version (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cpu, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, , production, false, cpu) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cpu, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, , production, false, cpu) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu126, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, -cuda126, production, true, cuda126) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu126, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, -cuda126, production, true, cuda126) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu128, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, -cuda, production, true, cuda) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu128, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, -cuda, production, true, cuda) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_web, cpu, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-24.04-arm, linux/arm64, arm64, , runner, false, cpu) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_web, cpu, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-latest, linux/amd64, amd64, , runner, false, cpu) (push) Has been cancelled
Build and Push Docker Images / verify_digests (push) Has been cancelled
Build and Push Docker Images / create_manifest (backend, surfsense-backend, , cpu) (push) Has been cancelled
Build and Push Docker Images / create_manifest (backend, surfsense-backend, -cuda, cuda) (push) Has been cancelled
Build and Push Docker Images / create_manifest (backend, surfsense-backend, -cuda126, cuda126) (push) Has been cancelled
Build and Push Docker Images / create_manifest (web, surfsense-web, , cpu) (push) Has been cancelled
Build and Push Docker Images / finalize_release (push) Has been cancelled
Support separate embedding base URL configuration
2026-07-08 17:11:42 -07:00
CREDO23
f731d371ab
feat(tiktok): add TikTok SEO landing page and site-wide marketing parity
2026-07-08 20:36:46 +02:00
CREDO23
5f9472b122
feat(tiktok): advertise tiktok across agent prompts, MCP instructions, docs
2026-07-08 20:30:38 +02:00
CREDO23
ea6befdf59
docs(tiktok): native connector page + nav card
2026-07-08 20:21:29 +02:00
CREDO23
116291a3b6
refactor(mcp): flatten to mcp_server package, drop src layout
...
Rename the import package surfsense_mcp -> mcp_server and remove the
src/ layer so the project mirrors the backend's shape (project folder
!= package name, e.g. surfsense_backend/app). Kills the redundant
surfsense_mcp/src/surfsense_mcp nesting.
Distribution name and console command (surfsense-mcp) are unchanged;
only python -m and internal import paths move to mcp_server. Dockerfile
CMD updated; no PYTHONPATH added since the editable install already
makes the package importable.
2026-07-07 20:24:53 +02:00
Dustin Persek
41edac7641
Merge remote-tracking branch 'upstream/dev' into feature/1354-embedding-base-url
...
# Conflicts:
# surfsense_web/content/docs/docker-installation/docker-compose.mdx
# surfsense_web/content/docs/manual-installation.mdx
2026-07-07 14:19:40 -04:00
CREDO23
6da1dd8f86
docs(mcp): document hosted /mcp remote endpoint
2026-07-07 18:27:19 +02:00
DESKTOP-RTLN3BA\$punk
1fd58752a3
feat: update environment variables and enhance scraping capabilities
...
- Adjusted Google Maps and YouTube micro pricing in the .env.example file for better cost management.
- Introduced new environment variables for captcha solving and stealth browser hardening to improve scraping resilience.
- Removed outdated smoke test for scraper API endpoints to streamline testing.
- Enhanced anonymous chat agent's system prompt to clarify capabilities and suggest account creation for advanced features.
- Updated Reddit fetch logic to prioritize new session handling and improve resilience against IP-related issues.
- Added compacting functionality for scraper results to optimize data handling and presentation.
- Improved workspace and document management tools with clearer descriptions and enhanced functionality.
- Introduced new UI components for agent setup guidance in the web application.
2026-07-06 20:27:36 -07:00
DESKTOP-RTLN3BA\$punk
271a21aee6
feat: docs and ui tweaks
2026-07-06 19:26:35 -07:00
Anish Sarkar
a8c1fb660d
feat(rename): complete searchSpace to workspace transition across frontend and backend
...
- Introduced a comprehensive specification for renaming `searchSpace` to `workspace` in `surfsense_web` and `surfsense_desktop`, ensuring all TypeScript identifiers, React props, and local data structures are updated.
- Implemented migration shims for persisted local state to prevent data loss during the transition.
- Updated observability metrics and IPC channels to reflect the new naming convention.
- Removed legacy `active-search-space` module and replaced it with `active-workspace` to maintain consistency.
- Ensured no behavioral changes or data loss for users during the renaming process.
2026-07-06 15:12:40 +05:30
DESKTOP-RTLN3BA\$punk
a64c8205fe
feat: implement ensure_publication for zero_publication management
...
- Added the ensure_publication function to create and verify the zero_publication if it is missing, ensuring idempotency during database initialization.
- Integrated ensure_publication into the create_db_and_tables function to prevent zero-cache crash loops on startup.
- Introduced a self-check script to validate the ensure_publication functionality on a create_all-bootstrapped database.
- Updated various components to reflect the transition from search space to workspace, including adjustments in imports and routing paths.
2026-07-05 23:17:13 -07:00
Dustin Persek
6fafedca6a
Support separate embedding base URL
2026-07-05 22:33:09 -04:00
DESKTOP-RTLN3BA\$punk
ab747e7a49
feat(agents): consolidate connectors under mcp_discovery; route web search through google_search
...
MCP consolidation:
- Route all MCP-capable connectors (Slack, Jira, Linear, ClickUp, Airtable,
Notion, Confluence, interim Gmail/Calendar, custom MCP) through a single
`mcp_discovery` subagent. Drive/OneDrive/Dropbox stay native to enrich the KB.
- Deprecate Discord/Teams/Luma: no viable official MCP server.
Google-only web search:
- Remove the main-agent `web_search` tool and the SearXNG platform service;
all public web search now flows through the `google_search` subagent via task().
- Deprecate the Tavily/SearXNG/Linkup/Baidu search connectors (HTTP 410 on
create, "Deprecated" badge); guide heavy users to the custom MCP connector.
- Remove web search from anonymous chat (pure Q&A).
- Tear SearXNG out of docker compose + install scripts; drop tavily-python
and linkup-sdk deps and their config/env vars.
Fix:
- metrics._package_version() now swallows any metadata lookup failure. A
malformed editable-install distribution with no `Version` field raised
KeyError deep in importlib.metadata, and since it runs on every
record_subagent_invoke_duration call it was crashing every task()
delegation. Verified end-to-end against live GPT-5.4.
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-04 21:06:04 -07:00
DESKTOP-RTLN3BA\$punk
701f888b9e
feat(proxy): refactor proxy configuration and add DataImpulse provider
...
- Updated proxy configuration in `.env.example` files to use `PROXY_URL` and `PROXY_URLS` instead of `CUSTOM_PROXY_URL` and `CUSTOM_PROXY_URLS`.
- Introduced `DataImpulseProvider` for proxy management, replacing the deprecated `AnonymousProxiesProvider`.
- Enhanced documentation to reflect changes in proxy setup and usage.
- Adjusted related code in the proxy registry and configuration files to support the new provider structure.
2026-07-04 17:18:40 -07:00
Anish Sarkar
34a49df8a3
docs: document runtime env vars and same-origin proxy access
2026-06-16 02:12:11 +05:30
Anish Sarkar
6d9540a1e8
docs(docker): document single-origin proxy deployment
2026-06-15 11:04:31 +05:30
Anish Sarkar
ceace003aa
feat(local-models): add documentation for connecting local model providers
2026-06-13 21:52:45 +05:30
Anish Sarkar
e462839929
refactor(chat): simplify model selector connection flow
2026-06-10 21:50:10 +05:30
DESKTOP-RTLN3BA\$punk
ce952d2ad1
chore: linting
2026-06-09 00:42:26 -07:00
DESKTOP-RTLN3BA\$punk
41a93ca8fb
refactor(env): streamline Redis configuration and remove deprecated variables
...
- Consolidated Redis configuration by introducing a single `REDIS_URL` variable for Celery broker, result backend, and app cache.
- Removed deprecated variables related to Firecrawl and Stripe token limits from `.env.example` files.
- Updated documentation to reflect changes in environment variable usage for improved clarity and maintainability.
2026-06-08 14:37:44 -07:00
Anish Sarkar
4e00f24a03
feat(docker): add ZERO_AUTO_RESET configuration for improved replication safety
...
- Introduced the ZERO_AUTO_RESET environment variable to enable automatic reset of the SQLite replica in case of replication halts.
- Updated Docker Compose files to include ZERO_AUTO_RESET in service configurations.
- Enhanced documentation to clarify the purpose and usage of the new variable.
2026-06-06 14:21:14 +05:30
