diff --git a/plans/plugins/runtime/01-process.md b/plans/plugins/runtime/01-process.md new file mode 100644 index 000000000..271926b0a --- /dev/null +++ b/plans/plugins/runtime/01-process.md @@ -0,0 +1,30 @@ +# Runtime — process + +> Owns: `surfsense_local/backend/modules/plugins/runner.py`, `modules/plugins/tasks.py`, the `plugin_runs` table. +> Contract: [`../01-protocol.md`](../01-protocol.md). + +## Goal + +The worker starts a plugin process for one run and records how it ended. [`02-calls-and-results.md`](02-calls-and-results.md) reads the results file afterward. + +## Work + +- Hand-written migration for `plugin_runs`: `id`, `workspace_id`, `plugin_id`, `entry`, `inputs` JSON, `status` (`running` `succeeded` `failed` `cancelled`), `error`, `log_tail`, `started_at`, `finished_at`. +- `import_models()` and `import_tasks()` gain this module the same way the other modules do. A test in the existing registration test fails if they are missing. +- Huey task `run_plugin(run_id)`. The API enqueues it. The task spawns the interpreter from `plugin_python()` ([`../python/01-interpreter.md`](../python/01-interpreter.md) supplies the packaged path; until then the task accepts an explicit interpreter path and tests pass `sys.executable`). +- Arguments, as the protocol specifies: ` --inputs --results --data `. The task writes the inputs file and an empty results file. Working directory is the plugin directory. `PYTHONPATH` is the SDK directory shipped next to the interpreter, then `/site-packages` when that directory exists. The SDK is not installed from PyPI and is not inside the tarball. +- For each manifest secret, set `SURFSENSE_PLUGIN_SECRET_`. The API has already refused the run when one is missing. +- Capture stdout and stderr into a ring of 16 KiB on `log_tail`. +- Exit 0 sets `succeeded`. Any other exit sets `failed` with `error` `exit `. Either way, hand the results path to the import in [`02-calls-and-results.md`](02-calls-and-results.md). +- Cancel sets a flag the task watches. The task sends `SIGTERM`, waits five seconds, sends `SIGKILL`, sets `cancelled`. Do not add a process-count limit. +- The plugin is never imported into the API or the worker. The test that proves it spawns a process whose pid is not the worker's. + +## Acceptance + +- A run of `plugins/example` with input `hi` exits 0 and leaves one results line. The run status is `succeeded`. +- A plugin that exits 1: status `failed`, worker process still alive. +- Cancel mid-run: status `cancelled` within ten seconds, and the pid is gone. + +## Needs from + +The protocol. `plugins/example` from the SDK stream for the success test. A one-file fake plugin in this stream's tests is enough for handshake, cancel, and crash, so this stream does not wait on the SDK. diff --git a/plans/plugins/runtime/02-calls-and-results.md b/plans/plugins/runtime/02-calls-and-results.md new file mode 100644 index 000000000..2e87f8a65 --- /dev/null +++ b/plans/plugins/runtime/02-calls-and-results.md @@ -0,0 +1,28 @@ +# Runtime — results + +> Owns: `surfsense_local/backend/modules/plugins/results.py`. +> Contract: [`../01-protocol.md`](../01-protocol.md). Process: [`01-process.md`](01-process.md). + +## Goal + +After the plugin process exits, the results file becomes notes and run rows. The plugin did its own HTTP and already had its secrets. This stream does not talk to it. + +## Work + +- Read the results path the process stream passes in. Import runs on `succeeded`, `failed`, and `cancelled`. A missing file imports nothing. +- Each line is one JSON object. A line over 1 MiB, or a line that is not JSON, is skipped and a line is appended to `log_tail`. The other lines are still imported. +- `kind: "document"` with `title` and `content`: insert a `Document` the same way `create_note` does (`NOTE`, `pending`), set `document_metadata` to `pluginId`, `entry`, `runId`, commit, call `ingest_document`. Do not set `dedup_key`. A second identical line inserts a second note. +- Any other kind: insert `plugin_results` (`run_id`, `kind`, `body` JSON). Hand-written migration. +- Do not open a connection to the plugin's hosts. Do not call `egress.require` for them. + +## Acceptance + +- A results file with one `document` line: a `NOTE` exists in the workspace and `ingest_document` was enqueued. +- Two identical `document` lines: two notes. +- A `kind` of `widget`: a `plugin_results` row, and no document. +- A file whose middle line is not JSON: the lines around it are imported, and `log_tail` mentions the skipped line. +- A plugin that performs HTTP during the run is not observed by this module. A test plugin can hit a local server; this module's code path does not connect to it. + +## Needs from + +[`01-process.md`](01-process.md) for the results path and the run row. The documents module, which already exists.