Commit graph

6201 commits

Author SHA1 Message Date
Rohan Verma
b1b51ada89
Merge pull request #1450 from MODSetter/dev_mod
Some checks are pending
Build and Push Docker Images / tag_release (push) Waiting to run
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, production) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, production) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-24.04-arm, linux/arm64, arm64, runner) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-latest, linux/amd64, amd64, runner) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (backend, surfsense-backend) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (web, surfsense-web) (push) Blocked by required conditions
feat: updated automations UI
2026-05-29 20:35:50 -07:00
DESKTOP-RTLN3BA\$punk
80daf46fbf Merge commit '7972901f15' into dev_mod 2026-05-29 20:28:12 -07:00
DESKTOP-RTLN3BA\$punk
9d1a01eb0c refactor(automations): streamline model eligibility handling in automation creation
- Removed the eligibility gate for model selection in the automation creation process, allowing users to choose models directly in the builder.
- Updated the `AutomationBuilderForm` to incorporate model selection logic, ensuring that selected models are validated and preserved during automation creation and editing.
- Simplified the `AutomationsContent` and `AutomationNewContent` components by eliminating unnecessary eligibility checks and alerts.
- Enhanced the user experience by integrating model selection directly into the automation approval process, ensuring that only billable models are used.
- Refactored related tests to cover new model selection behavior and ensure proper validation of user-selected models.
2026-05-29 20:27:40 -07:00
Rohan Verma
7972901f15
Merge pull request #1449 from CREDO23/feature-automations-v2
Some checks are pending
Build and Push Docker Images / tag_release (push) Waiting to run
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, production) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, production) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-24.04-arm, linux/arm64, arm64, runner) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-latest, linux/amd64, amd64, runner) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (backend, surfsense-backend) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (web, surfsense-web) (push) Blocked by required conditions
[Feat] [Automations] Event-Driven Trigger Type with document.entered_folder
2026-05-29 19:07:21 -07:00
CREDO23
7b0e7a4c34 chore: merge upstream/dev — keep builtin schedule path, add SearchSpace 2026-05-29 23:40:52 +02:00
CREDO23
38b73858b2 fix(json-view): coerce numeric strings to numbers on edit 2026-05-29 23:17:49 +02:00
CREDO23
cdb7ffb8d7 chore: ignore automation planning notes 2026-05-29 22:57:23 +02:00
CREDO23
ac49f000ec feat(frontend): add event trigger type to Zod schema 2026-05-29 22:55:54 +02:00
DESKTOP-RTLN3BA\$punk
fade9d1b9d feat(run-details-panel): enhance run details display and error handling
- Improved the layout and readability of the run details panel by restructuring sections and adding collapsible error views.
- Introduced a new `RunErrorSection` component to present run-level errors more clearly, allowing users to toggle raw error details.
- Updated the handling of run outputs, step results, and artifacts for better user experience.
- Refactored duration calculation in `RunRow` to utilize a dedicated `formatDuration` function for consistency.
- Added a new `RunStepResult` interface to improve type safety and clarity in handling step results.
2026-05-29 13:43:58 -07:00
CREDO23
d2c4b046bf fix(documents): use ORM objects in bulk_move so session hooks fire 2026-05-29 22:25:04 +02:00
CREDO23
75a1fee4e3 feat(event_bus): register session hooks at app startup 2026-05-29 22:22:10 +02:00
CREDO23
3354143a28 feat(event_bus): add SQLAlchemy session hooks to publish document.entered_folder events 2026-05-29 22:15:25 +02:00
CREDO23
f112f34332 feat(event_bus): add payload_if_entered_folder predicate with unit tests 2026-05-29 22:15:21 +02:00
CREDO23
be27df34e1 refactor(event_bus): wire catalog and events into package, rename builtin to events 2026-05-29 22:15:18 +02:00
CREDO23
731d5231ff feat(event_bus): add document.entered_folder event type and payload schema 2026-05-29 22:15:15 +02:00
CREDO23
2a511b8559 feat(event_bus): add EventCatalog class with register/get/all methods 2026-05-29 22:15:09 +02:00
Rohan Verma
cb2e33e083
Merge pull request #1446 from MODSetter/dev_mod
feat: added resource gates for automations
2026-05-29 12:32:48 -07:00
CREDO23
30fff9e52f refactor(automations): move agent_task to builtin and restructure dispatch 2026-05-29 18:13:09 +02:00
CREDO23
f356e304e8 feat(automations): wire trigger selectors and beat into celery 2026-05-29 17:49:12 +02:00
CREDO23
b1642993ba feat(automations): add schedule trigger source, selector and inputs 2026-05-29 17:49:12 +02:00
CREDO23
f293aa6bdf refactor(automations): move schedule trigger into builtin package 2026-05-29 17:49:05 +02:00
CREDO23
acd673023a feat(automations): add event trigger source, selector and registration 2026-05-29 17:48:48 +02:00
CREDO23
4ba637ea44 feat(automations): add event trigger match and inputs 2026-05-29 17:48:48 +02:00
CREDO23
3ba18c7750 feat(automations): add event trigger filter grammar 2026-05-29 17:48:48 +02:00
CREDO23
f09e302d4f feat(automations): add event trigger params 2026-05-29 17:48:48 +02:00
CREDO23
6fa2e52361 feat(automations): add shared start_run dispatch helper 2026-05-29 17:48:39 +02:00
CREDO23
356400ae2a feat(automations): migrate automation_trigger_type enum to add event 2026-05-29 17:48:39 +02:00
CREDO23
9247a2337f feat(automations): add EVENT to TriggerType enum 2026-05-29 17:48:39 +02:00
CREDO23
d6dfe53d62 feat(event_bus): add in-process domain event bus
A standalone, domain-agnostic pub/sub seam: an EventBus that owns its
subscriber registry and streams Event values from producers to listeners
in process. Boundary-crossing (Celery/DB/workers) is left to subscribers,
keeping the bus single-responsibility. Includes the immutable Event value
object and full unit coverage.
2026-05-29 15:26:12 +02:00
DESKTOP-RTLN3BA\$punk
43c66008e4 fix(llm-role-manager): synchronize local state with preferences updates
- Added useEffect to sync local assignments state with preferences when they change, ensuring the UI reflects the latest data.
- Updated state initialization to use null instead of empty strings for clarity in role assignments.
- Adjusted role assignment handling to correctly manage "unassigned" values and preserve Auto mode configuration during updates.
2026-05-29 03:30:22 -07:00
DESKTOP-RTLN3BA\$punk
409fec94c3 feat(automations): implement model eligibility checks for automation creation
- Added model eligibility checks to ensure automations can only use billable models (premium or BYOK).
- Introduced new API endpoint to report model eligibility status for search spaces.
- Updated frontend components to display eligibility alerts and disable creation options when models are not billable.
- Enhanced automation creation forms to reflect model eligibility, preventing users from submitting invalid configurations.
- Implemented server-side logic to capture and preserve model preferences across automation edits, ensuring consistent behavior during execution.
2026-05-29 03:13:46 -07:00
Rohan Verma
5d90fbe99f
Merge pull request #1445 from MODSetter/dev_mod
Some checks are pending
Build and Push Docker Images / tag_release (push) Waiting to run
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, production) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, production) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-24.04-arm, linux/arm64, arm64, runner) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-latest, linux/amd64, amd64, runner) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (backend, surfsense-backend) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (web, surfsense-web) (push) Blocked by required conditions
feat: added basic UI for automations and removed surfsense docs in chat related code.
2026-05-28 22:41:07 -07:00
DESKTOP-RTLN3BA\$punk
40ca9e6ed2 refactor: remove search_surfsense_docs tool and related references
- Deleted the `search_surfsense_docs` tool and its associated files, streamlining the agent's toolset.
- Updated various components and prompts to remove references to the now-removed tool, ensuring consistency across the codebase.
- Adjusted documentation to direct users to the SurfSense documentation link for product-related queries instead.
2026-05-28 22:35:14 -07:00
DESKTOP-RTLN3BA\$punk
9b9e6828c7 refactor(automations): enhance mention handling in task parameters
- Updated the `mentionParams` function to separate document and connector mentions, improving clarity and organization of the output.
- Modified the `mentionsFromParams` function to correctly handle and categorize mentions from parameters, ensuring connectors are processed separately.
- Adjusted documentation comments for better understanding of the changes in mention handling.
2026-05-28 21:44:22 -07:00
DESKTOP-RTLN3BA\$punk
6b76f8c138 refactor(automations): update icons and button labels in empty state and header components
- Replaced the FileJson icon with SquarePen in both AutomationsEmptyState and AutomationsHeader components.
- Updated button label from "Create via JSON" to "Create manually" for clarity in the automation creation process.
2026-05-28 21:29:24 -07:00
DESKTOP-RTLN3BA\$punk
d013617bf6 feat(automations): added UI and improved mentions
- Added support for @-mentions in agent tasks, allowing users to reference documents, folders, and connectors directly in their queries.
- Updated `run_agent_task` to resolve mentions and include them in the context passed to the agent.
- Introduced new parameters in `AgentTaskActionParams` for handling mentioned document and connector IDs.
- Refactored the automation edit and new components to utilize the new `AutomationBuilderForm` for a more streamlined user experience.
- Removed deprecated JSON forms to simplify the automation creation process.
2026-05-28 21:26:32 -07:00
DESKTOP-RTLN3BA\$punk
c601a9b102 fix: biome errs 2026-05-28 19:22:54 -07:00
DESKTOP-RTLN3BA\$punk
94e834134f chore: linting 2026-05-28 19:21:29 -07:00
Rohan Verma
4dda02c06c
Merge pull request #1443 from CREDO23/feature-automations
Some checks are pending
Build and Push Docker Images / tag_release (push) Waiting to run
Build and Push Docker Images / create_manifest (backend, surfsense-backend) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64, production) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64, production) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-24.04-arm, linux/arm64, arm64, runner) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-latest, linux/amd64, amd64, runner) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (web, surfsense-web) (push) Blocked by required conditions
[Feat] Automation V1 — Scheduled Agent Tasks, Created via Chat (HITL) or JSON
2026-05-28 12:41:41 -07:00
CREDO23
958bf9f95a fix(automations/agent_task): use in-memory checkpointer to avoid Celery PoolTimeout
The shared AsyncPostgresSaver caches DB connections in a module-level
pool. Cached connections are bound to the asyncio loop that opened
them, but `run_async_celery_task` discards the loop on each task's
exit — so after the first task the pool holds connections pointing
to a dead loop, and the next automation hangs 30s before failing
with `PoolTimeout: couldn't get a connection after 30.00 sec`.

Swap agent_task to `InMemorySaver`; automation runs only need state
within one Celery task, so nothing is lost. Site-local TODO tracks
the proper future fix (dispose the checkpointer pool around each
Celery task, mirroring `_dispose_shared_db_engine`).
2026-05-28 21:10:24 +02:00
CREDO23
353755fd73 test(automations): cross-cutting registries, enums, side-effects + shared fixtures
Top-level tests that span multiple submodules:

- test_stores.py (7): the trigger + action registry contracts — register
  round-trip, unknown type → None (not raise), duplicate registration
  rejected, defensive snapshot from all_*.
- test_definition_types.py (2): params_schema property on both
  ActionDefinition and TriggerDefinition reflects the Pydantic model.
- test_persistence_enums.py (3): exact string values + member sets of
  AutomationStatus / RunStatus / TriggerType — the postgres-mirrored
  contract that breaks stored rows if drifted.
- test_import_registrations.py (2): the bundled agent_task action and
  schedule trigger self-register on package import (canary for the
  side-effect import chain).

conftest.py adds isolated_action_registry / isolated_trigger_registry
fixtures: snapshot + restore of the module-level _REGISTRY dicts so
tests that add their own definitions don't leak across the suite.

14 tests, pure unit.
2026-05-28 19:03:55 +02:00
CREDO23
822940b09e test(automations/schemas): lock definition + api validation gates
definition/ (29 tests): the envelope (defaults, extra=forbid, empty
plan/name rejection), Inputs schema-alias roundtrip (Python schema_ ↔
wire schema), PlanStep numeric bounds + addressing-field constraints,
Execution production defaults stability (10-min timeout, 2 retries,
exponential backoff, drop_if_running) + closed-set Literal gates,
Metadata's exceptional extra="allow" contract, and TriggerSpec type
requirement.

api/ (9 tests): AutomationCreate/Update cascade-validate into the
nested definition, reject unknown payload fields, enforce name length;
TriggerCreate exposes safe defaults (enabled=True, params={},
static_inputs={}) and rejects unknown TriggerType strings at the
boundary.

All pure unit, no DB.
2026-05-28 19:03:42 +02:00
CREDO23
acbeb60a43 test(automations/actions): lock agent_task helpers (auto_decide + finalize)
auto_decide.build_auto_decisions (3): produces one decision per
action_request entry, defaults to one decision for legacy scalar
interrupts, and skips malformed interrupts silently so a misbehaving
tool can't take down the whole agent_task step.

finalize.extract_final_assistant_message (4): string-content AIMessage
returned verbatim, list-of-parts content concatenated (skipping
non-text parts like tool_use), walks back past trailing ToolMessages
to find the last AIMessage, and returns None when no extractable text
is present (so callers can branch on silence vs. empty).

7 tests, pure unit.
2026-05-28 19:03:29 +02:00
CREDO23
db4eef651f test(automations/templating): lock render, filters, environment, context
render.py (4): variable substitution, StrictUndefined raises on missing
keys, evaluate_predicate coerces to bool, render_value walks dicts/lists
and renders string leaves.

filters.py (4): slugify produces URL-safe output, date formats datetime
with strftime, date(None) → "" so templates can write
{{ inputs.last_fired_at | date }} on first run, date(str) passes through.

environment.py (4): the sandbox boundary — disallowed Jinja built-ins
(e.g. pprint) raise, and the finalize hook coerces non-string outputs
to predictable wire shapes (datetime → ISO, None → "", dict → JSON).

context.py (1): build_run_context exposes {run, inputs, steps} with the
exact shape every plan template body relies on.

13 tests total, all pure unit.
2026-05-28 19:03:22 +02:00
CREDO23
49af95b652 test(automations/runtime): lock execute_step + with_retries
execute_step (6 tests): happy path, when=falsy → skipped, unknown action
→ ActionNotFound failure, retry budget exhaustion (attempts = 1 +
max_retries), retry recovery, and template-rendering of step params
against the run context.

with_retries (3 tests): first-try success returns attempts=1, recovery
returns the actual attempt that produced the result, and exhaustion
re-raises the last exception with the handler called 1 + max_retries
times.

All tests use backoff="none" to keep wall-clock time zero; timeout
testing is intentionally skipped (would need >= 1s per the int contract,
and exhaustion already locks that any Exception triggers retry).
2026-05-28 19:03:08 +02:00
CREDO23
18b4800e49 test(automations/dispatch): lock _validate_inputs + DispatchError
Cover the input-validation contract dispatch_run relies on:

- no declared schema → inputs pass through unchanged (regression site
  that previously stripped runtime keys like fired_at / last_fired_at
  and broke Jinja templates).
- declared schema, valid inputs → passthrough validated.
- declared schema, invalid inputs → DispatchError (uniform exception
  type, not raw jsonschema.ValidationError).

Plus the DispatchError exception identity (Exception subclass, message
preserved, isinstance-friendly for the dispatch layer's consumers).

4 tests, pure unit.
2026-05-28 19:03:00 +02:00
CREDO23
2a76f43387 test(automations/triggers): lock schedule cron + params
Cover the cron + IANA timezone + UTC normalization contract for the
schedule trigger: next-match strictly-after, DST offset shift across
spring-forward, malformed cron / unknown timezone rejection, and the
ScheduleTriggerParams Pydantic gate that surfaces InvalidCronError as
ValidationError at the API boundary.

8 tests, pure unit (no DB, no mocks).
2026-05-28 19:02:52 +02:00
CREDO23
4f202e1fa3 feat(web): inline edit on trigger cards 2026-05-28 17:58:00 +02:00
CREDO23
fa0cdb9760 feat(web): unified json viewer/editor + edit existing automation 2026-05-28 16:07:54 +02:00
CREDO23
2d8d42bd9c refactor(web): polish automations detail view 2026-05-28 15:40:18 +02:00