Anish Sarkar
d39dac9016
fix(billing): exclude fallback results from Google SERP page charges
2026-07-27 16:14:03 +05:30
CREDO23
06c7e27c72
Merge remote-tracking branch 'upstream/dev' into fixes-playground-ui
...
# Conflicts:
# surfsense_web/lib/apis/base-api.service.ts
2026-07-24 08:29:39 +02:00
Rohan Verma
66de380e85
Merge pull request #1624 from AnishSarkar22/feat/connector-ui
...
Build and Push Docker Images / build (./surfsense_backend, cu128, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, -cuda, production, true, cuda) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, cu128, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, -cuda, production, true, cuda) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, cpu, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-24.04-arm, linux/arm64, arm64, , runner, false, cpu) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, cpu, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-latest, linux/amd64, amd64, , runner, false, cpu) (push) Blocked by required conditions
Build and Push Docker Images / compute_version (push) Waiting to run
Build and Push Docker Images / build (./surfsense_backend, cpu, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, , production, false, cpu) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, cpu, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, , production, false, cpu) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, cu126, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, -cuda126, production, true, cuda126) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, cu126, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, -cuda126, production, true, cuda126) (push) Blocked by required conditions
Build and Push Docker Images / verify_digests (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (backend, surfsense-backend, , cpu) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (backend, surfsense-backend, -cuda, cuda) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (backend, surfsense-backend, -cuda126, cuda126) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (web, surfsense-web, , cpu) (push) Blocked by required conditions
Build and Push Docker Images / finalize_release (push) Blocked by required conditions
feat(connectors): replace connector modal with a dedicated connectors layout
2026-07-23 15:26:15 -07:00
CREDO23
574b40be40
feat: validate indeed scrape URLs
2026-07-23 18:52:39 +02:00
CREDO23
9e04eb2b89
feat: validate google maps scrape and reviews URLs
2026-07-23 18:52:39 +02:00
CREDO23
a5ccd40387
feat: validate tiktok scrape and comments URLs
2026-07-23 18:52:39 +02:00
CREDO23
96f05991eb
feat: validate reddit scrape URLs
2026-07-23 18:52:38 +02:00
CREDO23
babefecee7
feat: validate youtube scrape and comments URLs
2026-07-23 18:52:32 +02:00
CREDO23
4e4b9236c3
feat: validate walmart scrape and reviews URLs
2026-07-23 18:52:32 +02:00
CREDO23
1996d3e54d
feat: validate amazon scrape URLs
2026-07-23 18:52:32 +02:00
CREDO23
a123c95703
feat: validate web.crawl start URLs
2026-07-23 18:52:32 +02:00
CREDO23
797dd2df47
feat: add HttpUrlStr shared URL validator
2026-07-23 18:52:32 +02:00
Anish Sarkar
4e39b411be
refactor: simplify docstring in capability registry for clarity
2026-07-23 17:02:06 +05:30
DESKTOP-RTLN3BA\$punk
dbedf0cfa5
feat(backend): integrate PostHog analytics for enhanced observability
...
Obsidian Plugin Lint / lint (push) Has been cancelled
- Added PostHog configuration options to .env.example files for both Docker and Surfsense backend.
- Introduced PostHog dependency in pyproject.toml.
- Implemented analytics middleware to capture various events across the application, including user authentication, automation runs, and API requests.
- Enhanced existing routes and services to emit analytics events, providing insights into user interactions and system performance.
- Ensured graceful shutdown of analytics clients in worker processes and application lifecycles.
2026-07-22 22:16:28 -07:00
CREDO23
950de5e670
fix: forward ToolRuntime to capability tools
2026-07-22 20:06:18 +02:00
CREDO23
c1ba4215fd
feat: mint run citation from capability tool
2026-07-22 19:34:56 +02:00
CREDO23
957e57bb82
feat: register a scraper run as a citation
2026-07-22 19:34:56 +02:00
CREDO23
ca317c4686
Merge remote-tracking branch 'upstream/dev' into feature-walmart-scraper
...
# Conflicts:
# docker/.env.example
# surfsense_backend/app/capabilities/core/billing.py
# surfsense_backend/app/capabilities/core/types.py
# surfsense_backend/app/config/__init__.py
# surfsense_mcp/mcp_server/features/scrapers/__init__.py
# surfsense_web/content/docs/connectors/index.mdx
# surfsense_web/content/docs/connectors/native/index.mdx
# surfsense_web/content/docs/how-to/mcp-server.mdx
2026-07-21 02:42:41 +02:00
CREDO23
d7c30cac2f
refactor(walmart): align reviews executor docstring
2026-07-19 08:45:27 +02:00
CREDO23
9b9091ae97
feat(walmart): add scrape and reviews capabilities
2026-07-19 08:19:22 +02:00
CREDO23
0d2c8df5bd
feat(walmart): add billing units and rate config
2026-07-19 08:19:21 +02:00
CREDO23
91aa265afb
Merge remote-tracking branch 'upstream/dev' into feature-indeed-jobs-scraper
...
# Conflicts:
# README.es.md
# README.hi.md
# README.md
# README.pt-BR.md
# README.zh-CN.md
# surfsense_backend/tests/unit/capabilities/google_maps/test_registry.py
# surfsense_backend/tests/unit/capabilities/reddit/test_registry.py
# surfsense_backend/tests/unit/capabilities/youtube/test_registry.py
# surfsense_mcp/mcp_server/features/scrapers/__init__.py
# surfsense_web/content/docs/connectors/index.mdx
# surfsense_web/content/docs/connectors/native/index.mdx
# surfsense_web/content/docs/connectors/native/meta.json
# surfsense_web/content/docs/how-to/mcp-server.mdx
# surfsense_web/lib/connectors-marketing/index.ts
# surfsense_web/lib/playground/catalog.ts
2026-07-18 22:11:30 +02:00
CREDO23
ec258d8387
feat: map scrape_job_details in indeed executor
2026-07-15 17:54:38 +02:00
CREDO23
81f535e9d0
feat: add scrape_job_details to indeed.scrape input
2026-07-15 17:54:38 +02:00
Anish Sarkar
21a7a0a0b0
feat(amazon): register scrape capability
2026-07-15 01:57:32 +05:30
Anish Sarkar
3e0caa2027
feat(amazon): add scrape capability schema, executor, definition
2026-07-15 01:57:24 +05:30
Anish Sarkar
01fd6172ce
feat(billing): add amazon product billing unit and rate
2026-07-15 01:54:49 +05:30
CREDO23
8c410d8682
feat: indeed capability namespace
2026-07-14 21:49:23 +02:00
CREDO23
17159172ee
feat: indeed.scrape verb package
2026-07-14 21:49:23 +02:00
CREDO23
3f91a006cd
feat: register indeed.scrape capability
2026-07-14 21:49:23 +02:00
CREDO23
b457599fd4
feat: indeed.scrape executor
2026-07-14 21:49:23 +02:00
CREDO23
e887c1cae7
feat: indeed.scrape io schemas
2026-07-14 21:49:23 +02:00
CREDO23
d89ca141a4
feat: meter indeed_job billing unit
2026-07-14 21:49:23 +02:00
CREDO23
05bfc6a10e
feat: add INDEED_JOB billing unit
2026-07-14 21:49:23 +02:00
DESKTOP-RTLN3BA\$punk
2b018c4474
refactor: streamline TikTok and Instagram scraping logic by removing search_queries and enhancing documentation for clarity
Build and Push Docker Images / compute_version (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cpu, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, , production, false, cpu) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cpu, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, , production, false, cpu) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu126, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, -cuda126, production, true, cuda126) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu126, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, -cuda126, production, true, cuda126) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu128, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, -cuda, production, true, cuda) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, cu128, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, -cuda, production, true, cuda) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_web, cpu, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-24.04-arm, linux/arm64, arm64, , runner, false, cpu) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_web, cpu, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-latest, linux/amd64, amd64, , runner, false, cpu) (push) Has been cancelled
Build and Push Docker Images / verify_digests (push) Has been cancelled
Build and Push Docker Images / create_manifest (backend, surfsense-backend, , cpu) (push) Has been cancelled
Build and Push Docker Images / create_manifest (backend, surfsense-backend, -cuda, cuda) (push) Has been cancelled
Build and Push Docker Images / create_manifest (backend, surfsense-backend, -cuda126, cuda126) (push) Has been cancelled
Build and Push Docker Images / create_manifest (web, surfsense-web, , cpu) (push) Has been cancelled
Build and Push Docker Images / finalize_release (push) Has been cancelled
2026-07-13 17:11:25 -07:00
DESKTOP-RTLN3BA\$punk
1131da5ed7
feat: bumped version to 0.0.32
2026-07-13 16:29:39 -07:00
Anish Sarkar
e38ca19b18
Merge remote-tracking branch 'upstream/dev' into feat/instagram-scraper
2026-07-11 04:31:11 +05:30
Anish Sarkar
27d22a9a2a
refactor(instagram): drop mentions from scrape capability
2026-07-11 03:25:20 +05:30
Anish Sarkar
de1990f9f6
refactor(instagram): simplify scrape and details capability schemas
2026-07-11 02:20:46 +05:30
Anish Sarkar
8afa4c6fc6
refactor(instagram): remove unused comments capability
2026-07-11 02:20:39 +05:30
CREDO23
67b5472b9f
feat(tiktok): add tiktok.trending verb for the Explore feed
...
The Explore feed (/api/explore/item_list) is a global trending-video feed
served to anonymous sessions, and it returns the same itemStruct shape as the
other listings — so the verb reuses parse_video, the listing flow, the
TikTokVideoItem output, and the per-video billing meter wholesale. Adds a
browser-capture marker + fetch_trending, a synthetic-target orchestrator entry,
and the tiktok.trending capability, surfaced on the chat subagent.
2026-07-09 18:52:56 +02:00
CREDO23
7723b5b8b6
feat(tiktok): add tiktok.comments verb for public comment scraping
...
Comments load over a signed /api/comment/list XHR that TikTok serves to
anonymous sessions once the comments panel is opened (unlike profile-video and
general-search feeds), so this is a reliable verb. Given video URLs it returns
CommentItems (text, author, likes, reply counts; replies carry repliesToId),
deduped per video, capped, and degraded to an ErrorItem for empty/withheld
videos or a bad_url ErrorItem for non-video inputs.
Generalizes the browser capture over a pluggable interaction step so the
comments flow (open panel, scroll the panel to paginate) reuses the same
warm+capture scaffolding as listing/user-search. Billed per comment on a new
TIKTOK_COMMENT meter (TIKTOK_MICROS_PER_COMMENT, matching the per-comment
market), surfaced on the chat subagent alongside tiktok.scrape/user_search.
2026-07-09 18:35:26 +02:00
CREDO23
192b6dc31a
feat(tiktok): add tiktok.user_search verb for account discovery
...
Video/general search is login-walled for anonymous sessions, but the Users
tab (/api/search/user) returns public account records without a redirect, so
this exposes the one reliably-unblocked search path. A keyword yields
TikTokProfileItems (name, followers, bio, verification), deduped per query,
capped, and degraded to an ErrorItem when a query is empty/withheld.
Reuses the browser capture (generalized over XHR markers + extractor) and the
shared profile item shape. Billed per account on a new TIKTOK_USER meter
(TIKTOK_MICROS_PER_USER), surfaced on the chat subagent alongside tiktok.scrape.
2026-07-09 18:00:40 +02:00
Anish Sarkar
82f1d0b4e5
refactor(instagram): improve anonymous scraping logic and error messaging
...
Enhanced the Instagram scraper to clarify the requirements for accessing user profiles and hashtags. Updated the error message for blocked access to provide detailed guidance on necessary credentials. Introduced a regex for validating Instagram usernames and refined the discovery function to handle profile queries directly, improving user experience and error handling in anonymous mode.
2026-07-09 20:23:09 +05:30
Anish Sarkar
929efe8152
feat(instagram): register instagram.* capability namespace
2026-07-09 16:01:49 +05:30
Anish Sarkar
7ea474be93
feat(instagram): add details capability verb
2026-07-09 16:01:38 +05:30
Anish Sarkar
38a26b1ccf
feat(instagram): add comments capability verb
2026-07-09 14:58:58 +05:30
Anish Sarkar
ff55537bce
feat(instagram): add scrape capability verb
2026-07-09 14:58:54 +05:30
Anish Sarkar
43660ee51b
feat(billing): map Instagram units to rate keys and display nouns
2026-07-09 14:57:53 +05:30
Anish Sarkar
f204db7bce
feat(billing): add INSTAGRAM_ITEM and INSTAGRAM_COMMENT units
2026-07-09 14:57:44 +05:30
