DESKTOP-RTLN3BA\$punk
847b3a3a27
feat(docs): update README files to include Walmart as a data source for all language versions
2026-07-25 15:58:46 -07:00
DESKTOP-RTLN3BA\$punk
4a727a2c57
chore: linting
2026-07-24 14:30:46 -07:00
DESKTOP-RTLN3BA\$punk
dbedf0cfa5
feat(backend): integrate PostHog analytics for enhanced observability
...
Obsidian Plugin Lint / lint (push) Has been cancelled
- Added PostHog configuration options to .env.example files for both Docker and Surfsense backend.
- Introduced PostHog dependency in pyproject.toml.
- Implemented analytics middleware to capture various events across the application, including user authentication, automation runs, and API requests.
- Enhanced existing routes and services to emit analytics events, providing insights into user interactions and system performance.
- Ensured graceful shutdown of analytics clients in worker processes and application lifecycles.
2026-07-22 22:16:28 -07:00
Rohan Verma
08d431454f
Merge pull request #1617 from CREDO23/feature-okf
...
Build and Push Docker Images / finalize_release (push) Blocked by required conditions
Build and Push Docker Images / compute_version (push) Waiting to run
Build and Push Docker Images / build (./surfsense_backend, cpu, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, , production, false, cpu) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, cpu, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, , production, false, cpu) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, cu126, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, -cuda126, production, true, cuda126) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, cu126, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, -cuda126, production, true, cuda126) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, cu128, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, -cuda, production, true, cuda) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, cu128, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, -cuda, production, true, cuda) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, cpu, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-24.04-arm, linux/arm64, arm64, , runner, false, cpu) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, cpu, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-latest, linux/amd64, amd64, , runner, false, cpu) (push) Blocked by required conditions
Build and Push Docker Images / verify_digests (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (backend, surfsense-backend, , cpu) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (backend, surfsense-backend, -cuda, cuda) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (backend, surfsense-backend, -cuda126, cuda126) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (web, surfsense-web, , cpu) (push) Blocked by required conditions
[Feat] Serve the knowledge base in Open Knowledge Format (OKF)
2026-07-21 15:39:59 -07:00
CREDO23
5f1cc7a634
test: MCP get_document forwards Accept header
2026-07-21 19:59:01 +02:00
CREDO23
a01b292886
request OKF concept via content negotiation
2026-07-21 19:58:53 +02:00
CREDO23
edf38edc3f
support per-request headers
2026-07-21 19:58:52 +02:00
CREDO23
ca317c4686
Merge remote-tracking branch 'upstream/dev' into feature-walmart-scraper
...
# Conflicts:
# docker/.env.example
# surfsense_backend/app/capabilities/core/billing.py
# surfsense_backend/app/capabilities/core/types.py
# surfsense_backend/app/config/__init__.py
# surfsense_mcp/mcp_server/features/scrapers/__init__.py
# surfsense_web/content/docs/connectors/index.mdx
# surfsense_web/content/docs/connectors/native/index.mdx
# surfsense_web/content/docs/how-to/mcp-server.mdx
2026-07-21 02:42:41 +02:00
CREDO23
a3f8a7e741
feat(walmart): register routes and MCP tools
2026-07-19 08:19:22 +02:00
CREDO23
91aa265afb
Merge remote-tracking branch 'upstream/dev' into feature-indeed-jobs-scraper
...
# Conflicts:
# README.es.md
# README.hi.md
# README.md
# README.pt-BR.md
# README.zh-CN.md
# surfsense_backend/tests/unit/capabilities/google_maps/test_registry.py
# surfsense_backend/tests/unit/capabilities/reddit/test_registry.py
# surfsense_backend/tests/unit/capabilities/youtube/test_registry.py
# surfsense_mcp/mcp_server/features/scrapers/__init__.py
# surfsense_web/content/docs/connectors/index.mdx
# surfsense_web/content/docs/connectors/native/index.mdx
# surfsense_web/content/docs/connectors/native/meta.json
# surfsense_web/content/docs/how-to/mcp-server.mdx
# surfsense_web/lib/connectors-marketing/index.ts
# surfsense_web/lib/playground/catalog.ts
2026-07-18 22:11:30 +02:00
CREDO23
65c757fbc6
feat: mention indeed in MCP server instructions
2026-07-15 18:54:49 +02:00
CREDO23
e88ff8cef7
test: expect indeed MCP tool in selfcheck
2026-07-15 18:54:49 +02:00
CREDO23
b89fe1a4f0
feat: register indeed MCP tool
2026-07-15 18:54:49 +02:00
CREDO23
457c073c44
feat: add indeed MCP scraper tool
2026-07-15 18:54:49 +02:00
Anish Sarkar
8d7ea2b269
feat(selfcheck): register Amazon scraper tool in self-check configuration
2026-07-15 10:15:27 +05:30
Anish Sarkar
964ad2e083
feat(amazon): implement Amazon product scraper tool and register scraping functionality
2026-07-15 10:15:19 +05:30
DESKTOP-RTLN3BA\$punk
2b018c4474
refactor: streamline TikTok and Instagram scraping logic by removing search_queries and enhancing documentation for clarity
Build and Push Docker Images / compute_version (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cpu, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, , production, false, cpu) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cpu, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, , production, false, cpu) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu126, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, -cuda126, production, true, cuda126) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu126, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, -cuda126, production, true, cuda126) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu128, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, -cuda, production, true, cuda) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu128, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, -cuda, production, true, cuda) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_web, cpu, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-24.04-arm, linux/arm64, arm64, , runner, false, cpu) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_web, cpu, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-latest, linux/amd64, amd64, , runner, false, cpu) (push) Has been cancelled
Build and Push Docker Images / verify_digests (push) Has been cancelled
Build and Push Docker Images / create_manifest (backend, surfsense-backend, , cpu) (push) Has been cancelled
Build and Push Docker Images / create_manifest (backend, surfsense-backend, -cuda, cuda) (push) Has been cancelled
Build and Push Docker Images / create_manifest (backend, surfsense-backend, -cuda126, cuda126) (push) Has been cancelled
Build and Push Docker Images / create_manifest (web, surfsense-web, , cpu) (push) Has been cancelled
Build and Push Docker Images / finalize_release (push) Has been cancelled
2026-07-13 17:11:25 -07:00
DESKTOP-RTLN3BA\$punk
1131da5ed7
feat: bumped version to 0.0.32
2026-07-13 16:29:39 -07:00
Anish Sarkar
cb256729a3
chore: Enhance SurfSense MCP server documentation and toolset by adding TikTok scraping capabilities, including comments, user search, and trending features. Updated README and documentation to reflect the addition of 24 native tools.
2026-07-11 04:38:42 +05:30
Anish Sarkar
e38ca19b18
Merge remote-tracking branch 'upstream/dev' into feat/instagram-scraper
2026-07-11 04:31:11 +05:30
Anish Sarkar
ca7d23d33a
refactor(mcp): align Instagram scraper tool with capability changes
2026-07-11 03:25:23 +05:30
Anish Sarkar
8fc86b7fa5
refactor(mcp): align Instagram scraper tool with new pipeline
2026-07-11 02:20:58 +05:30
CREDO23
6a0ff1495f
fix(tiktok-mcp): flag that search_queries returns no videos in the scrape tool
2026-07-10 17:40:04 +02:00
CREDO23
6b15d610d9
feat(tiktok): surface comments, user_search, trending across all product surfaces
...
Brings the three newer verbs to parity with tiktok.scrape everywhere it was
wired: MCP tools (+ selfcheck manifest), the chat subagent prompt/description,
the playground catalog, the native docs page, and the SEO marketing page. Also
adds live e2e stages for comments, user search, and trending. Docs/FAQ now
state the real contract: profile metadata is reliable while its video list can
be withheld, and keyword video search is walled (use user search for accounts).
2026-07-09 19:09:00 +02:00
Anish Sarkar
c5e3fd4006
feat(mcp): add Instagram scrape/comments/details tools
2026-07-09 16:01:55 +05:30
CREDO23
5f9472b122
feat(tiktok): advertise tiktok across agent prompts, MCP instructions, docs
2026-07-08 20:30:38 +02:00
CREDO23
ed1c3a1f3d
feat(tiktok): expose tiktok.scrape on MCP, chat subagent, playground
2026-07-08 20:20:20 +02:00
CREDO23
116291a3b6
refactor(mcp): flatten to mcp_server package, drop src layout
...
Rename the import package surfsense_mcp -> mcp_server and remove the
src/ layer so the project mirrors the backend's shape (project folder
!= package name, e.g. surfsense_backend/app). Kills the redundant
surfsense_mcp/src/surfsense_mcp nesting.
Distribution name and console command (surfsense-mcp) are unchanged;
only python -m and internal import paths move to mcp_server. Dockerfile
CMD updated; no PYTHONPATH added since the editable install already
makes the package importable.
2026-07-07 20:24:53 +02:00
CREDO23
839618ef09
fix(mcp): install curl for container healthcheck probe
2026-07-07 19:29:19 +02:00
CREDO23
d09ecd16c9
feat(mcp): add Dockerfile for remote streamable-http deployment
2026-07-07 18:27:19 +02:00
CREDO23
6da1dd8f86
docs(mcp): document hosted /mcp remote endpoint
2026-07-07 18:27:19 +02:00
CREDO23
768609d886
feat(mcp): add health endpoint with public-path auth exemption
2026-07-07 18:27:19 +02:00
CREDO23
ea5b27eaa8
refactor(mcp): split scraper and knowledge-base features into SRP modules
2026-07-07 18:27:19 +02:00
CREDO23
1c01aaeca5
docs(mcp): tighten comments to intent-only
2026-07-07 18:27:19 +02:00
CREDO23
ec9ae1d238
build(mcp): declare starlette and uvicorn dependencies
2026-07-07 18:27:19 +02:00
CREDO23
e1fd77c791
test(mcp): cover per-request auth and workspace isolation
2026-07-07 18:27:19 +02:00
CREDO23
0e16fe846a
fix(mcp): scope active workspace per caller identity
2026-07-07 18:27:19 +02:00
CREDO23
730face126
feat(mcp): serve stateless streamable-http with stdio dispatch
2026-07-07 18:27:19 +02:00
CREDO23
f3c06b1fbf
feat(mcp): make api_key optional, add http host/port config
2026-07-07 18:27:19 +02:00
CREDO23
953eeb11b2
refactor(mcp): resolve API key per request in client
2026-07-07 18:27:19 +02:00
CREDO23
7a91aced68
feat(mcp): assemble streamable-http transport app
2026-07-07 18:27:19 +02:00
CREDO23
39fd7a2218
feat(mcp): add per-request API key identity auth slice
2026-07-07 18:27:19 +02:00
DESKTOP-RTLN3BA\$punk
1fd58752a3
feat: update environment variables and enhance scraping capabilities
...
- Adjusted Google Maps and YouTube micro pricing in the .env.example file for better cost management.
- Introduced new environment variables for captcha solving and stealth browser hardening to improve scraping resilience.
- Removed outdated smoke test for scraper API endpoints to streamline testing.
- Enhanced anonymous chat agent's system prompt to clarify capabilities and suggest account creation for advanced features.
- Updated Reddit fetch logic to prioritize new session handling and improve resilience against IP-related issues.
- Added compacting functionality for scraper results to optimize data handling and presentation.
- Improved workspace and document management tools with clearer descriptions and enhanced functionality.
- Introduced new UI components for agent setup guidance in the web application.
2026-07-06 20:27:36 -07:00
CREDO23
faa6d6baea
refactor(mcp): rename SURFSENSE_PAT to SURFSENSE_API_KEY
2026-07-06 03:40:36 +02:00
CREDO23
b2bf925f63
test(mcp): cover dropping of unset query params
2026-07-06 03:17:25 +02:00
CREDO23
abbb43632f
fix(mcp): omit unset query params from REST requests
2026-07-06 03:17:25 +02:00
CREDO23
479ca3bbee
feat(mcp): add server entrypoint and test suite
2026-07-06 02:29:19 +02:00
CREDO23
34efe3aa44
feat(mcp): add knowledge base tools
2026-07-06 02:29:19 +02:00
CREDO23
39dd413711
feat(mcp): add scraper tools for all platforms
2026-07-06 02:29:19 +02:00
CREDO23
bbb21fb010
feat(mcp): add workspace selector tools
2026-07-06 02:29:19 +02:00
