mirror of
https://github.com/razzant/ouroboros.git
synced 2026-08-05 08:39:48 +00:00
Merge pull request #88 from razzant/fix/promote-admission-6873
fix: make managed task routing admission durable
This commit is contained in:
Anton Razzhigaev
GitHub
commit
7c2ebd23cf
35 changed files with 2726 additions and 395 deletions
|
|
@ -9,7 +9,7 @@
|
|||
[](https://github.com/razzant/ouroboros/releases)
|
||||
[](https://github.com/razzant/ouroboros/releases)
|
||||
[](https://github.com/razzant/OuroborosHub)
|
||||
[](VERSION)
|
||||
[](VERSION)
|
||||
|
||||
Ouroboros is an open-source, general-purpose AI agent whose identity, durable memory, and history continue across tasks and restarts. It works on external projects, coordinates a live swarm of specialist agents, and can rewrite the implementation it runs on, including its code, architecture, prompts, tools, and dependencies. Reflection can also change how it understands itself without severing that continuity.
|
||||
|
||||
|
|
@ -255,6 +255,7 @@ and integration work.
|
|||
|
||||
| Version | Date | Description |
|
||||
|---------|------|-------------|
|
||||
| 6.87.3 | 2026-07-31 | **fix: routing tools prove their effect instead of treating a queue write as success.** A manager-backed event bus lives for the server process, survives worker-pool replacement and force-killed producers, and serializes writes before returning. Pool startup is atomic, managed-update recovery never overwrites a live generation, and crash-storm disablement is a durable admission fence. Promote, project-route, manual-target, and steer actions carry a unique token and wait for the supervisor's durable receipt; only a receipt for that exact attempt permits a positive final. Promoted/API tasks additionally require a persisted queue snapshot and scheduled task result, reject duplicate ids, and resolve source clone/attach only after authoritative executor admission. Rejections and 15-second unconfirmed outcomes are loud, self-contained, and never invite automatic retry. Real child-process and end-to-end transport regressions cover every routing outcome plus concurrent pool startup and API snapshot failure. |
|
||||
| 6.87.2 | 2026-07-31 | **fix: Telegram Mini App recovers a completed menu rollback and keeps the real Quick Tunnel failure visible during backoff.** An interrupted or older rollback could leave its ownership snapshot behind after Telegram had already restored the original button, so every later URL rotation was rejected as external drift. The exact original is now recognized as a completed rollback while any third value remains fail-closed. Cloudflared's bounded, redacted final error line survives reconnect backoff instead of being replaced by a generic status. The bundled Telegram skill moves to 1.0.1 so existing native installs resync the fix. |
|
||||
| 6.87.1 | 2026-07-31 | **fix: a clean review verdict stops being recorded as unparseable, and the release lane goes green again.** The shared prompt contract asks a reviewer that found nothing for an empty array plus the `NO_FINDINGS` sentinel; triad implemented it and the advisory parser never had a branch for it, so a reviewer returning exactly what was asked was recorded as `parse_failure` — blocking freshness, paying an extra extraction model, and forcing a retry that re-ran the full serial preflight. The contract text now lives beside the parser that enforces it and splits into findings-only and required-matrix modes, so scope review and skill advisory stop advertising an all-clear their own parsers reject, and `review_status` surfaces the typed per-run cause it already persisted. Three CI-only test defects that predate this change were fixed with it: a 1s subprocess budget that could not cover interpreter startup on Windows, a POSIX permission assertion evaluated on Windows, and a fixed 220ms sleep racing a 180ms drawer transition on the Linux WebKit runner. |
|
||||
| 6.87.0 | 2026-07-31 | **feat: the OSWorld working prompt separates a task's live surface from its stored one, and stops the contract's UNCHANGED item from forbidding the very edit the task asks for.** Two failures on the v6.86.0 run traced to the same paragraph. A task whose grader reads the LIVE window (VLC fullscreen) was answered by ticking the preference and never entering fullscreen, because the contract asked only where the result must PERSIST; WHERE now has two slots, live and persisted, each filled or explicitly marked not-applicable — and filling a slot must never invent an extra tab, window or dialog to inspect, which would break the 28 tasks graded on tab lists alone. A task asking for a bullet on an existing paragraph was answered by typing a new line, because the contract had recorded the existing paragraph as UNCHANGED; UNCHANGED now covers only content the task does not mention, and a MARKER or PROPERTY the task names — a bullet, a style, a colour, an alignment — is applied to the content already there rather than to a freshly typed line. Three further candidate clauses were written and dropped after adversarial review proved each would cost more than it won: preferring a slide master over instances loses a task whose shapes carry direct formatting that overrides styles, demanding a named resource be exhausted before any substitute loses a task won by detouring to another search engine, and requiring a configuration CLI to match the GUI's breadth was generalised from a single grader and contradicted the preamble's own rule against shaping work around guesses at the evaluator. |
|
||||
|
|
|
|||
2
VERSION
2
VERSION
|
|
@ -1 +1 @@
|
|||
6.87.2
|
||||
6.87.3
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Ouroboros v6.87.2 — Architecture & Reference
|
||||
# Ouroboros v6.87.3 — Architecture & Reference
|
||||
|
||||
This file is NOT a changelog. Version history lives in README.md, git tags, and commit log.
|
||||
|
||||
|
|
@ -25,6 +25,7 @@ server.py (Starlette+uvicorn) ← HTTP + WebSocket on configurable host:port (de
|
|||
│ ├── workers.py ← Multiprocessing worker pool (fork/spawn by platform)
|
||||
│ ├── state.py ← Persistent state (state.json) with file locking
|
||||
│ ├── queue.py ← Task queue management (PENDING/RUNNING lists) + activity-based timeout enforcement
|
||||
│ ├── task_admission.py ← Token-owned, in-process reservations that fence duplicate user-ingress ids before Project/workspace/attachment side effects; queue.py remains the state authority
|
||||
│ ├── task_lifecycle.py ← Queue-owned atomic acceptance fences, one root-budget admission marker plus replay-safe task resume, cascade cancellation, and fenced Project deletion/quiescence; extends queue state without creating a second lifecycle authority
|
||||
│ ├── task_reaper.py ← (v6.38.0) Variant A off-loop worker reaper (extracted from queue.py): kill/join/archive/respawn a timed-out worker on a single-owner background thread, off the loop critical path. (v6.38.1) STRICT fail-closed: if the worker will not confirm dead, it holds the slot `reaping` and leaves the task RUNNING (no terminal/task_done/retry/respawn while it may be alive), emits `task_reaper_wedged` + an owner /restart hint, and lets the custody reaper end the orphan on the next generation
|
||||
│ ├── schedule_time.py ← Cron/timezone schedule time parsing helpers
|
||||
|
|
@ -92,6 +93,7 @@ server.py (Starlette+uvicorn) ← HTTP + WebSocket on configurable host:port (de
|
|||
├── retention.py ← Unified GC retention SSOT: clamp/age-cutoff helpers + legacy-key seed picker used by worktree/task-drive/service-log startup pruning
|
||||
├── workspace_preflight.py ← Read-only external-workspace git/manifest/toolchain snapshot used by gateway task creation
|
||||
├── project_sources.py ← (v6.59.0) Project working-folder sources: attach an existing owner folder (resolved-realpath validation — exists/dir/not-home-root/no repo-data overlap; opt-in `init_git` attach-snapshot commit, NEVER auto-init) and server-side `git clone` into the durable projects root (atomic tmp→rename, `GIT_TERMINAL_PROMPT=0` + BatchMode ssh, typed `auth_required`); provenance (attached|cloned|genesis|none) + `clone_url` are recorded on the registry as historical facts, `trusted_at` stamps automatically (notification trust model — attaching IS the owner's grant)
|
||||
├── promotion_source.py ← Supervisor-side promoted-task source admission; attach/clone and registry binding run off the event-drain loop only after an executor/id reservation
|
||||
├── workspace_admission.py ← (v6.58.0) Workspace-task admission SSOT: the ONE workspace-root validator (git-worktree root, no repo/data overlap) shared by `/api/tasks` and the promote path; `resolve_room_workspace` defaults a project-room task to the room's registered `working_dir` (sentinel `workspace="none"` opts out) and LOUD-FAILS a set-but-broken working_dir (a room task must never silently degrade to a workspace-less self_modification-profile task); `compose_workspace_block` renders the shared [HEADLESS_WORKSPACE] guidance; `bounded_workspace_preflight` hard-caps the promote-path snapshot so the supervisor event-drain thread stays responsive
|
||||
├── local_model.py ← Local LLM lifecycle (llama-cpp-python)
|
||||
├── local_model_autostart.py ← Local model startup helper
|
||||
|
|
@ -682,7 +684,7 @@ finalization states.
|
|||
│ │ ├── projects.json ← Project registry: immutable id/chat identity, optional working folder, lifecycle/routing fence, visible revision, and deletion error; tombstones are durable and never age-pruned
|
||||
│ │ ├── project_task_bindings.json ← Task→project bindings (schema v1) with a REQUIRED typed origin: the ingress-captured source-row ref (+`source_text`, the retention-proof full copy, stored only for CROSS-thread origins — i.e. the message that started the project) or a closed-enum `origin_absent` reason. Immutable except ONE-WAY enrichment (a same-project re-bind may fill a missing ref; a valid ref is never changed); one root belongs to at most one Project and tombstoning never removes the binding. The retention-proof invariant is FORWARD-ONLY by owner decision: pre-v6.73.0 bindings (no `source_text`) are not migrated and their start messages remain rotation-vulnerable as before
|
||||
│ │ ├── ui_preferences.json ← Owner-local layout preferences and monotonic `project_seen_revision` paint ACKs; legacy `project_last_viewed`/`project_hidden` are one-minor deprecated no-ops
|
||||
│ │ ├── queue_snapshot.json
|
||||
│ │ ├── queue_snapshot.json ← Durable PENDING/RUNNING recovery projection plus actual worker/reaping/idle counts and explicit `worker_pool_disabled_reason` (empty during ordinary operation; typed crash-storm cause when user-facing task admission must refuse)
|
||||
│ │ ├── extension_companions.json ← Runtime snapshot for live extension companion processes
|
||||
│ │ ├── extension_reconcile/ ← Worker-written extension reconcile markers consumed by the server lifespan pickup task
|
||||
│ │ ├── review_continuations/ ← Per-task blocked-review continuation payloads (+ quarantined corrupt files under `corrupt/`)
|
||||
|
|
@ -972,7 +974,7 @@ visual-QA runner, endpoint, ledger, or browser auto-installer is introduced.
|
|||
|
||||
Dashboard hosts Logs, Evolution, Costs, and Updates. Logs and Chat share event summarization (`log_events.js`) so task phases are described consistently. The same `taskOutcomeSeverity` reducer drives both surfaces: a degraded review or best-effort/degraded objective is warning-colored, never presented as a green solved result. Compact review projections disclose every panel and actor's model/provider, role, transport/parse/semantic state, coverage, quorum contribution, reason, and enforcement impact by default; since v6.70.0 the actor/panel `reason` is the COMPLETE redacted rationale (the former 500/800-char caps destroyed the only owner-reachable copy — reviewer rationale is a cognitive artifact, BIBLE P1) and each actor carries a forensic `response_ref` into the private observability store; full model prompts/responses themselves remain in that private audit storage. Evolution reads `/api/evolution-data`; Costs reads the physical-attempt ledger projection from `/api/cost-breakdown` and distinguishes confirmed/settled, reserved, unresolved upper bound, and unknown/unmetered work together with `cost_final`; it exposes budget controls from the shared setup contract via `/api/settings` GET/POST. Updates exposes official managed updates plus local recovery commits/tags.
|
||||
The Logs page renders task/LLM/tool/progress events as grouped task cards, while Chat renders the same stream as a live task card so operational history and live dialogue stay visually consistent.
|
||||
Worker tasks forward their `append_jsonl` log lines to the live dashboard over `EVENT_Q` via a per-worker log sink installed in `supervisor/workers.py::worker_main` (the WS log sink only exists in the main process), suppressing types that already arrive through a dedicated live sibling event — `tool_call`/`llm_round`/`task_checkpoint`/`task_done`/`llm_usage` — to avoid double broadcast and a double `task_checkpoint` file write. On load and on every reconnect the Logs page backfills recent history from `/api/logs/{events,tools,progress,supervisor}` (`web/modules/logs.js::backfillRecentLogs`) and dedupes the live-overlap window by event identity so the pre-connect window is neither dropped nor shown twice.
|
||||
Worker tasks forward their `append_jsonl` log lines to the live dashboard over `EVENT_Q` via a per-worker log sink installed in `supervisor/workers.py::worker_main` (the WS log sink only exists in the main process), suppressing types that already arrive through a dedicated live sibling event — `tool_call`/`llm_round`/`task_checkpoint`/`task_done`/`llm_usage` — to avoid double broadcast and a double `task_checkpoint` file write. `EVENT_Q` belongs to the `server.py` process, not to a replaceable worker-pool generation: direct chat, consciousness, active turns, and every respawned worker retain one manager-backed queue proxy until process exit. The manager transport serializes in the producer and isolates producer connections, avoiding both feeder-thread false positives and pipe corruption when a worker is force-killed mid-frame. `spawn_workers` therefore reuses that bus, holds the lifecycle lock through complete pool publication, and refuses to overwrite a non-empty live pool; single-slot recovery uses `respawn_worker`. On load and on every reconnect the Logs page backfills recent history from `/api/logs/{events,tools,progress,supervisor}` (`web/modules/logs.js::backfillRecentLogs`) and dedupes the live-overlap window by event identity so the pre-connect window is neither dropped nor shown twice.
|
||||
Chart.js is bundled locally as `web/chart.umd.min.js`; no CDN dependency by design.
|
||||
|
||||
### Forensic Observability and Typed Outcomes
|
||||
|
|
@ -1146,7 +1148,7 @@ Runs in a background thread inside `server.py:_run_supervisor()`.
|
|||
Each iteration (0.5s sleep):
|
||||
1. `rotate_chat_log_if_needed()` — archive chat.jsonl if > 800KB
|
||||
2. `ensure_workers_healthy()` — respawn dead workers, detect crash storms
|
||||
3. Drain event queue (worker→supervisor events via multiprocessing.Queue)
|
||||
3. Drain the process-lifetime manager-backed event queue (worker/direct-chat/consciousness→supervisor; pool respawns never rotate it, and force-killed producers cannot corrupt later traffic)
|
||||
4. `enforce_task_timeouts()` — activity-based stop (v6.38.0): a task is stopped only when it makes no REAL progress (`llm_usage`/progress events, NOT the 30s liveness heartbeat) AND has no progressing/queued subtree, beyond `OUROBOROS_TASK_IDLE_TIMEOUT_SEC` (floored to the per-call ceiling); the only HARD axes are an explicit `deadline_at`, `OUROBOROS_TASK_ABS_CEILING_SEC`, and budget. The heavy teardown (kill/join/archive/respawn) is handed OFF the loop to a single-owner background reaper (`supervisor/task_reaper.py`); the slot is marked `reaping` under `_queue_lock` (assign + crash-detector skip it) and the terminal write + retry + respawn happen only after the worker is PROVABLY dead (v6.38.1 strict fail-closed) — if it will not confirm dead, the reaper does nothing downstream, holds the slot `reaping`, leaves the task RUNNING (no terminal/respawn while it may be alive), and emits `task_reaper_wedged` + an owner /restart hint, with the orphan custody-reaped on the next generation — so a still-alive worker is never raced
|
||||
5. Periodic custody reap (every 600s) and periodic zombie reconcile (every 300s,
|
||||
`server.py::_periodic_zombie_reconcile`): heals `review_job.json` files and
|
||||
|
|
@ -1632,6 +1634,7 @@ Multi-project ("штаб и проекты", v6.32.0) builds the owner-facing la
|
|||
|
||||
- **Projects registry and lifecycle** (`ouroboros/projects_registry.py`, `data/state/projects.json`): immutable id + deterministic `chat_id`, name (80-character SSOT), optional working folder/provenance, `last_active_at`, `visible_revision`, routing generation/fence, and `active | deleting | tombstoned`. Boot reconcile registers pre-existing stores and never age-prunes or resurrects a reserved id. Creation still supports attach, clone, genesis, or file-less Projects and keeps `project_room_lens_dir` / `room_chat_lens_dir` behavior. Rename mutates only the display name. Delete first closes admission/routing and increments the fence generation, then reuses queue cascade cancellation/quiescence; only after the subtree settles does it atomically tombstone. ID, canonical chat/history, immutable bindings, folder, journal/workpad, and memory are preserved. An interrupted deletion remains recoverably `deleting` and resumes at startup; `project_hidden` is a one-minor deprecated no-op, not lifecycle state. `GET /api/fs/dirs` remains the owner-facing home-confined directory browser. **Room lens (v6.61.3):** direct-chat reads and default shell cwd use the registered working folder; default-root writes return `ROOM_WRITE_VIA_TASK`, and a broken folder fails loudly rather than falling back to the system repo.
|
||||
- **Canonical chat and owner routing**: Web, CLI, and existing owner transports enter the same router. In Main, the ordinary decision turn gets a compact manifest of Projects, RUNNING/PENDING roots, recent final results/artifacts, and recent canonical dialogue; if there are no Projects/tasks it retains the prior direct flow. In a Project room, exactly one addressable RUNNING/PENDING root is a zero-call mailbox delivery. Zero or multiple candidates use one LLM decision scoped to that Project. Uncertainty, stale targets, and selection errors return typed `needs_manual_target` with concrete task options and `New task in Project`; they never pick or spawn randomly. `handle_chat_ephemeral`, `steer_task`, promote/bind/history/task-result lookup, attachment staging, and the existing serialized direct lane are reused—there is no parallel message lane. Each inbound row is stored once; Project history projects the binding-held source refs (immutable once valid; a ref-less binding may be one-way enriched), and when the canonical row has left the bounded read window the lens synthesizes the start message from the binding's own `source_text` (post-quota, identity-deduped, hard-capped — `origin_projected=true`); `chat_annotations.jsonl` carries only the latest owner-visible action/target/status. The typed annotation is presentation metadata attached to the owner's message, while the Agent normalizes blank model output to a visible warning and every finalized response remains a separate durable assistant reply across Web and non-Web transports. Inline Main replies create no routing annotation.
|
||||
- **Confirmed routing and task admission (v6.87.2)**: `promote_chat_to_task`, `route_to_project`, manual-target abstention, and `steer_task` never treat local event emission as proof of effect. Each attempt carries a fresh `routing_token`; the emitting turn waits at most 15 seconds for the exact supervisor-authored durable fact in an existing SSOT. Promote/project-route use the matching-token `promotion_admission` inside `task_results/<id>.json`; manual-target and steer use the matching token in the existing latest-action row in `chat_annotations.jsonl` (there is no parallel acknowledgement store). Promote/project-route become positive only after executor admission, duplicate-id fencing, off-loop source attach/clone, `PENDING` insertion, durable `queue_snapshot.json`, owner-visible routing annotation when applicable, and a matching-token `STATUS_SCHEDULED` task result. Steer is confirmed only after the owner-mailbox write; manual-target only after its typed outcome and concrete options are persisted. A missing fact returns an explicit unconfirmed final and forbids automatic retry. `POST /api/tasks` applies the same snapshot-before-success rule for CLI/A2A ingress. Crash-storm shutdown records `worker_pool_disabled_reason` before process teardown; user-facing managed work rejects `worker_pool_unavailable` before Project/workspace side effects, while busy/reaping pools remain valid queue targets and internal managed-update recovery may still enqueue before initial spawn. These admission facts prove asynchronous delivery but do not replace later task lifecycle authority.
|
||||
- **Multi-task chat steering (v6.34.0, WS1)**: when several tasks run in one chat, a new message must be able to STEER a chosen running task — "turn = decision" (WS10) alone could only answer or spawn. The agent makes that choice by JUDGMENT, not code: `build_runtime_section` (`context.py`) surfaces a STRUCTURAL `current_chat.running_tasks` fact (the running ROOT tasks in the current chat: `{task_id, title, objective, project_id, started_at, steerable}`, snapshotted from supervisor `RUNNING`) into the runtime JSON block, fed through `task_metadata` (`server.py::_decision_turn_metadata`) into both the direct and ephemeral decision turns. The new `steer_task(task_id, message)` capability (`tools/control.py`, beside `promote_chat_to_task`/`route_to_project`) enforces only TRANSPORT invariants — target still in `RUNNING`, same chat / project binding, not a subagent — and delivers via `write_owner_message` on the active task drive (`supervisor/events.py::_handle_steer_task`); the running task drains it at its next round. Idempotent via `client_message_id`-derived mailbox `msg_id` (no double-deliver on retry); a STALE target fails VISIBLY rather than auto-spawning (the agent's safe default when unsure is `promote_chat_to_task`). Code never decides "this message belongs to task A" — it only exposes chat state + enforces invariants (P5/BIBLE LLM-first). `forward_to_worker`'s parent→child descendant guard is unchanged. The project-room pre-LLM delivery (`_route_project_chat_to_running_task`) is narrowed to the UNAMBIGUOUS 1:1 case only — exactly one steerable pooled task in the room is a transport invariant; with zero or multiple candidates the message now flows to the decision turn so the agent picks via `steer_task` rather than code mechanically selecting the first of several.
|
||||
- **In-task project scoping** (`ensure_project_scope`, v6.37.0 C4.1): once work is ALREADY running, the agent can name+create a project and bind THE CURRENT task to it in one structural move — the affordance for "make this a project named X" without falling back to a bare `mkdir`. `tools/control_delegation.py::_ensure_project_scope` (the delegation/scope affordances extracted from `tools/control.py`, wired into its `get_tools`) validates/derives the project id, sets `ctx.project_id` for the rest of the loop (so `journal_write`/per-project knowledge target it immediately), and emits an `ensure_project_scope` event; `supervisor/events.py::_handle_ensure_project_scope` → `supervisor/workers.py::ensure_project_scope` creates the registry project, durably binds the task (`bind_task_to_project`), updates the live `RUNNING[tid].task.project_id` (so the one-writer lease sees it), and broadcasts `projects_changed`. Idempotent for the same project; refuses to re-scope to a different one; subagents inherit the parent's scope and cannot change it. `promote_chat_to_task` remains the preferred FIRST move from chat; this is the mid-run complement. The whole subagent tree's live/history frames then route to the project thread by lineage (`project_chat_for_task_tree`).
|
||||
- **Per-project memory**: beside `knowledge/`, each project store carries `journal.jsonl` (milestones via `journal_write`/`journal_read`: start/checkpoint/blocked/done/note), `workpad.md` (`workpad_read`/`workpad_write`), and a thread mirror. The project task's FOCUSED context injects these via `build_knowledge_sections` WITHOUT silent prefix-slicing (BIBLE P1): the workpad rides in full (a warning, not a clip, signals an oversized one), and the journal shows recent milestones in full with a VISIBLE `journal_read` index pointer for older entries. Generic data tools still cannot reach the store (`project_store_access_block`).
|
||||
|
|
|
|||
|
|
@ -250,9 +250,9 @@ async def api_update_check(_request: Request) -> JSONResponse:
|
|||
def _respawn_workers_after_failed_update() -> None:
|
||||
"""Revive workers when an update aborts after they were stopped (no restart follows)."""
|
||||
try:
|
||||
from supervisor.workers import spawn_workers
|
||||
from supervisor.workers import ensure_worker_pool_started
|
||||
|
||||
spawn_workers()
|
||||
ensure_worker_pool_started(allow_disabled_restart=True)
|
||||
except Exception:
|
||||
log.warning("update_apply: failed to respawn workers after aborted update", exc_info=True)
|
||||
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ import asyncio
|
|||
import json
|
||||
import logging
|
||||
import pathlib
|
||||
import shutil
|
||||
import time
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
|
@ -89,6 +90,29 @@ _RESERVED_METADATA_KEYS = frozenset({
|
|||
})
|
||||
|
||||
|
||||
def _cleanup_api_admission_attempt(
|
||||
drive_root: pathlib.Path,
|
||||
task_id: str,
|
||||
admission_token: str,
|
||||
child_drive: Optional[pathlib.Path] = None,
|
||||
) -> None:
|
||||
"""Release one token and remove only its pre-admission task-local state."""
|
||||
from supervisor.queue import release_task_admission
|
||||
|
||||
release_task_admission(task_id, admission_token)
|
||||
if child_drive is not None:
|
||||
try:
|
||||
from ouroboros.headless import remove_subagent_task_drive
|
||||
|
||||
remove_subagent_task_drive(drive_root, task_id)
|
||||
except Exception:
|
||||
log.warning("Failed to clean child drive for rejected task %s", task_id, exc_info=True)
|
||||
try:
|
||||
shutil.rmtree(task_artifacts_dir(drive_root, task_id, create=False), ignore_errors=True)
|
||||
except Exception:
|
||||
log.warning("Failed to clean admission artifacts for task %s", task_id, exc_info=True)
|
||||
|
||||
|
||||
def _external_subagent_label(body: Dict[str, Any], metadata: Dict[str, Any]) -> bool:
|
||||
role_values = [
|
||||
body.get("delegation_role"),
|
||||
|
|
@ -148,12 +172,23 @@ def _admission_rejection_response(
|
|||
project_id: str,
|
||||
workspace_root: Optional[pathlib.Path],
|
||||
child_drive: Optional[pathlib.Path],
|
||||
status_code: int = 409,
|
||||
detail: str = "Task was not scheduled because its admission fence is closed.",
|
||||
) -> Optional[JSONResponse]:
|
||||
"""Terminalize a typed queue refusal so no scheduled phantom remains."""
|
||||
if not (isinstance(admitted, dict) and admitted.get("_admission_blocked")):
|
||||
return None
|
||||
reason_code = str(admitted.get("_admission_blocked") or "admission_fence")
|
||||
detail = "Task was not scheduled because its admission fence is closed."
|
||||
if reason_code in {"duplicate_task_id", "admission_reservation_lost"}:
|
||||
return JSONResponse(
|
||||
{
|
||||
"error": "Task id is already owned by another admission attempt.",
|
||||
"task_id": task_id,
|
||||
"status": "rejected",
|
||||
"admission": {"reason_code": reason_code},
|
||||
},
|
||||
status_code=409,
|
||||
)
|
||||
admission = {
|
||||
"reason_code": reason_code,
|
||||
"project_id": str(admitted.get("_project_id") or project_id),
|
||||
|
|
@ -181,6 +216,10 @@ def _admission_rejection_response(
|
|||
STATUS_FAILED,
|
||||
admission_cleanup={"child_drive_removed": bool(removed)},
|
||||
)
|
||||
try:
|
||||
shutil.rmtree(task_artifacts_dir(drive_root, task_id, create=False), ignore_errors=True)
|
||||
except Exception:
|
||||
log.warning("Failed to clean rejected task artifacts for %s", task_id, exc_info=True)
|
||||
return JSONResponse(
|
||||
{
|
||||
"error": detail,
|
||||
|
|
@ -188,10 +227,155 @@ def _admission_rejection_response(
|
|||
"status": STATUS_FAILED,
|
||||
"admission": admission,
|
||||
},
|
||||
status_code=409,
|
||||
status_code=status_code,
|
||||
)
|
||||
|
||||
|
||||
def _enqueue_api_task_durably(
|
||||
task: Dict[str, Any],
|
||||
*,
|
||||
drive_root: pathlib.Path,
|
||||
task_id: str,
|
||||
admission_token: str,
|
||||
result_fields: Dict[str, Any],
|
||||
) -> Dict[str, Any]:
|
||||
"""Atomically enqueue, snapshot, and publish the scheduled task result."""
|
||||
from supervisor import queue
|
||||
|
||||
with queue._queue_lock:
|
||||
admitted = queue.enqueue_task(task)
|
||||
if isinstance(admitted, dict) and admitted.get("_admission_blocked"):
|
||||
return admitted
|
||||
if queue.persist_queue_snapshot(reason="api_task_create") is not True:
|
||||
queue.PENDING[:] = [
|
||||
row for row in queue.PENDING
|
||||
if not (
|
||||
isinstance(row, dict)
|
||||
and str(row.get("id") or "") == task_id
|
||||
and str(row.get("_admission_owner_token") or "") == admission_token
|
||||
)
|
||||
]
|
||||
queue.persist_queue_snapshot(reason="api_task_create_rollback")
|
||||
return {
|
||||
**task,
|
||||
"_admission_blocked": "queue_snapshot_persist_failed",
|
||||
"_admission_status_code": 503,
|
||||
}
|
||||
write_task_result(drive_root, task_id, STATUS_SCHEDULED, **result_fields)
|
||||
queue.release_task_admission(task_id, admission_token)
|
||||
return admitted
|
||||
|
||||
|
||||
def _complete_api_task_admission(
|
||||
task: Dict[str, Any],
|
||||
*,
|
||||
drive_root: pathlib.Path,
|
||||
task_id: str,
|
||||
admission_token: str,
|
||||
project_id: str,
|
||||
description: str,
|
||||
allowed_resources: Dict[str, Any],
|
||||
deadline_at: str,
|
||||
workspace_root: Optional[pathlib.Path],
|
||||
workspace_mode: str,
|
||||
memory_mode: str,
|
||||
child_drive: Optional[pathlib.Path],
|
||||
artifacts: List[Dict[str, Any]],
|
||||
metadata: Dict[str, Any],
|
||||
) -> JSONResponse:
|
||||
"""Publish one API admission or roll back only its token-owned queue row."""
|
||||
result_fields = {
|
||||
"parent_task_id": task.get("parent_task_id"),
|
||||
"root_task_id": task.get("root_task_id"),
|
||||
"session_id": task.get("session_id"),
|
||||
"actor_id": task.get("actor_id"),
|
||||
"delegation_role": task.get("delegation_role"),
|
||||
"project_id": project_id,
|
||||
"description": description,
|
||||
"context": task.get("context"),
|
||||
"expected_output": task.get("expected_output"),
|
||||
"constraints": task.get("constraints"),
|
||||
"allowed_resources": allowed_resources,
|
||||
"deadline_at": deadline_at,
|
||||
"task_contract": task.get("task_contract"),
|
||||
"workspace_root": task.get("workspace_root"),
|
||||
"workspace_mode": workspace_mode,
|
||||
"memory_mode": memory_mode,
|
||||
"child_drive_root": str(child_drive or ""),
|
||||
"budget_drive_root": str(drive_root) if child_drive is not None else "",
|
||||
"artifacts": artifacts,
|
||||
"artifact_status": ARTIFACT_STATUS_PENDING if workspace_root else "",
|
||||
"metadata": metadata,
|
||||
"result": "Task accepted and durably scheduled.",
|
||||
}
|
||||
try:
|
||||
admitted = _enqueue_api_task_durably(
|
||||
task,
|
||||
drive_root=drive_root,
|
||||
task_id=task_id,
|
||||
admission_token=admission_token,
|
||||
result_fields=result_fields,
|
||||
)
|
||||
snapshot_failed = (
|
||||
str(admitted.get("_admission_blocked") or "")
|
||||
== "queue_snapshot_persist_failed"
|
||||
)
|
||||
rejection = _admission_rejection_response(
|
||||
admitted,
|
||||
drive_root=drive_root,
|
||||
task_id=task_id,
|
||||
project_id=project_id,
|
||||
workspace_root=workspace_root,
|
||||
child_drive=child_drive,
|
||||
status_code=503 if snapshot_failed else 409,
|
||||
detail=(
|
||||
"Task was not scheduled because its durable queue snapshot could not be written."
|
||||
if snapshot_failed
|
||||
else "Task was not scheduled because its admission fence is closed."
|
||||
),
|
||||
)
|
||||
if rejection is not None:
|
||||
return rejection
|
||||
except Exception as exc:
|
||||
try:
|
||||
from supervisor import queue as supervisor_queue
|
||||
|
||||
with supervisor_queue._queue_lock:
|
||||
supervisor_queue.PENDING[:] = [
|
||||
row for row in supervisor_queue.PENDING
|
||||
if not (
|
||||
isinstance(row, dict)
|
||||
and str(row.get("id") or "") == task_id
|
||||
and str(row.get("_admission_owner_token") or "")
|
||||
== admission_token
|
||||
)
|
||||
]
|
||||
supervisor_queue.persist_queue_snapshot(
|
||||
reason="api_task_create_failed_rollback"
|
||||
)
|
||||
except Exception:
|
||||
log.warning(
|
||||
"Failed to roll back API task %s after admission error",
|
||||
task_id,
|
||||
exc_info=True,
|
||||
)
|
||||
write_task_result(
|
||||
drive_root,
|
||||
task_id,
|
||||
"failed",
|
||||
**{
|
||||
**result_fields,
|
||||
"artifact_status": ARTIFACT_STATUS_FAILED if workspace_root else "",
|
||||
"result": f"Failed to enqueue task: {exc}",
|
||||
},
|
||||
)
|
||||
_cleanup_api_admission_attempt(
|
||||
drive_root, task_id, admission_token, child_drive
|
||||
)
|
||||
return json_exception(exc, 503)
|
||||
return JSONResponse({"ok": True, "task_id": task_id, "status": STATUS_SCHEDULED})
|
||||
|
||||
|
||||
async def api_tasks_create(request: Request) -> JSONResponse:
|
||||
"""POST /api/tasks — enqueue a managed headless task."""
|
||||
|
||||
|
|
@ -209,7 +393,7 @@ async def api_tasks_create(request: Request) -> JSONResponse:
|
|||
drive_root = request_drive_root(request)
|
||||
repo_dir = request_repo_dir(request)
|
||||
try:
|
||||
task_id = validate_task_id(body.get("task_id") or uuid.uuid4().hex[:8])
|
||||
task_id = validate_task_id(body.get("task_id") or uuid.uuid4().hex[:16])
|
||||
except ValueError as exc:
|
||||
return json_error(str(exc), 400)
|
||||
if load_task_result(drive_root, task_id):
|
||||
|
|
@ -326,7 +510,34 @@ async def api_tasks_create(request: Request) -> JSONResponse:
|
|||
deadline_at = datetime.fromtimestamp(time.time() + timeout_sec, timezone.utc).isoformat().replace("+00:00", "Z")
|
||||
if deadline_at:
|
||||
metadata["deadline_at"] = deadline_at
|
||||
child_drive = prepare_task_drive(drive_root, task_id, effective_drive_mode, project_id=_task_project_id)
|
||||
admission_token = uuid.uuid4().hex
|
||||
from supervisor.queue import reserve_task_admission
|
||||
|
||||
reservation = reserve_task_admission(
|
||||
task_id,
|
||||
admission_token,
|
||||
require_worker_pool=True,
|
||||
drive_root=drive_root,
|
||||
)
|
||||
if reservation.get("status") != "reserved":
|
||||
reason = str(reservation.get("reason") or "admission_reservation_failed")
|
||||
status_code = 503 if reason.startswith("worker_pool_") else 409
|
||||
return json_error(
|
||||
f"task admission refused: {reason}",
|
||||
status_code,
|
||||
task_id=task_id,
|
||||
reason_code=reason,
|
||||
worker_pool_disabled_reason=str(
|
||||
reservation.get("worker_pool_disabled_reason") or ""
|
||||
),
|
||||
)
|
||||
try:
|
||||
child_drive = prepare_task_drive(
|
||||
drive_root, task_id, effective_drive_mode, project_id=_task_project_id
|
||||
)
|
||||
except Exception as exc:
|
||||
_cleanup_api_admission_attempt(drive_root, task_id, admission_token)
|
||||
return json_exception(exc, 503)
|
||||
# v6.52.0 (P1): stage attachments into the SAME drive the task will read from at
|
||||
# runtime — the child drive when forked/empty, else the shared drive (matches the
|
||||
# task['drive_root'] set at the end of this handler). The returned manifest renders
|
||||
|
|
@ -334,9 +545,15 @@ async def api_tasks_create(request: Request) -> JSONResponse:
|
|||
from ouroboros.artifacts import stage_task_attachments
|
||||
|
||||
effective_drive = child_drive or drive_root
|
||||
attachment_manifest = stage_task_attachments(
|
||||
effective_drive, task_id, _normalize_attachments(body.get("attachments"))
|
||||
)
|
||||
try:
|
||||
attachment_manifest = stage_task_attachments(
|
||||
effective_drive, task_id, _normalize_attachments(body.get("attachments"))
|
||||
)
|
||||
except Exception as exc:
|
||||
_cleanup_api_admission_attempt(
|
||||
drive_root, task_id, admission_token, child_drive
|
||||
)
|
||||
return json_exception(exc, 503)
|
||||
attachment_images = [m for m in attachment_manifest if m.get("is_image")]
|
||||
metadata.setdefault("session_id", str(body.get("session_id") or uuid.uuid4().hex))
|
||||
metadata.setdefault("actor_id", str(body.get("actor_id") or "cli"))
|
||||
|
|
@ -364,14 +581,20 @@ async def api_tasks_create(request: Request) -> JSONResponse:
|
|||
}
|
||||
metadata["workspace_preflight"] = workspace_preflight_summary
|
||||
|
||||
task_text = _compose_task_text(
|
||||
description,
|
||||
workspace_root=workspace_root,
|
||||
workspace_mode=workspace_mode,
|
||||
memory_mode=memory_mode,
|
||||
workspace_preflight=workspace_preflight_summary,
|
||||
attachments=attachment_manifest,
|
||||
)
|
||||
try:
|
||||
task_text = _compose_task_text(
|
||||
description,
|
||||
workspace_root=workspace_root,
|
||||
workspace_mode=workspace_mode,
|
||||
memory_mode=memory_mode,
|
||||
workspace_preflight=workspace_preflight_summary,
|
||||
attachments=attachment_manifest,
|
||||
)
|
||||
except Exception as exc:
|
||||
_cleanup_api_admission_attempt(
|
||||
drive_root, task_id, admission_token, child_drive
|
||||
)
|
||||
return json_exception(exc, 503)
|
||||
task = {
|
||||
"id": task_id,
|
||||
"type": task_type,
|
||||
|
|
@ -406,86 +629,39 @@ async def api_tasks_create(request: Request) -> JSONResponse:
|
|||
# drive) so build_user_content can resolve staged attachment IMAGES for EVERY task
|
||||
# shape — not just child-drive tasks. The child-drive block below re-affirms it.
|
||||
"drive_root": str(effective_drive),
|
||||
"_require_unique_task_id": True,
|
||||
"_require_worker_pool": True,
|
||||
"_admission_token": admission_token,
|
||||
}
|
||||
task = attach_task_contract(task)
|
||||
try:
|
||||
task = attach_task_contract(task)
|
||||
except Exception as exc:
|
||||
_cleanup_api_admission_attempt(
|
||||
drive_root, task_id, admission_token, child_drive
|
||||
)
|
||||
return json_exception(exc, 503)
|
||||
if child_drive is not None:
|
||||
task["drive_root"] = str(child_drive)
|
||||
task["child_drive_root"] = str(child_drive)
|
||||
task["budget_drive_root"] = str(drive_root)
|
||||
metadata["child_drive_root"] = str(child_drive)
|
||||
metadata["budget_drive_root"] = str(drive_root)
|
||||
write_task_result(
|
||||
drive_root,
|
||||
task_id,
|
||||
STATUS_SCHEDULED,
|
||||
parent_task_id=task.get("parent_task_id"),
|
||||
root_task_id=task.get("root_task_id"),
|
||||
session_id=task.get("session_id"),
|
||||
actor_id=task.get("actor_id"),
|
||||
delegation_role=task.get("delegation_role"),
|
||||
return _complete_api_task_admission(
|
||||
task,
|
||||
drive_root=drive_root,
|
||||
task_id=task_id,
|
||||
admission_token=admission_token,
|
||||
project_id=_task_project_id,
|
||||
description=description,
|
||||
context=task.get("context"),
|
||||
expected_output=task.get("expected_output"),
|
||||
constraints=task.get("constraints"),
|
||||
allowed_resources=allowed_resources,
|
||||
deadline_at=deadline_at,
|
||||
task_contract=task.get("task_contract"),
|
||||
workspace_root=task.get("workspace_root"),
|
||||
workspace_root=workspace_root,
|
||||
workspace_mode=workspace_mode,
|
||||
memory_mode=memory_mode,
|
||||
child_drive_root=str(child_drive or ""),
|
||||
budget_drive_root=str(drive_root) if child_drive is not None else "",
|
||||
child_drive=child_drive,
|
||||
artifacts=artifacts,
|
||||
artifact_status=ARTIFACT_STATUS_PENDING if workspace_root else "",
|
||||
metadata=metadata,
|
||||
result="Task accepted and scheduled.",
|
||||
)
|
||||
try:
|
||||
from supervisor.queue import enqueue_task, persist_queue_snapshot
|
||||
|
||||
admitted = enqueue_task(task)
|
||||
rejection = _admission_rejection_response(
|
||||
admitted,
|
||||
drive_root=drive_root,
|
||||
task_id=task_id,
|
||||
project_id=_task_project_id,
|
||||
workspace_root=workspace_root,
|
||||
child_drive=child_drive,
|
||||
)
|
||||
if rejection is not None:
|
||||
return rejection
|
||||
persist_queue_snapshot(reason="api_task_create")
|
||||
except Exception as exc:
|
||||
write_task_result(
|
||||
drive_root,
|
||||
task_id,
|
||||
"failed",
|
||||
parent_task_id=task.get("parent_task_id"),
|
||||
root_task_id=task.get("root_task_id"),
|
||||
session_id=task.get("session_id"),
|
||||
actor_id=task.get("actor_id"),
|
||||
project_id=_task_project_id,
|
||||
delegation_role=task.get("delegation_role"),
|
||||
description=description,
|
||||
context=task.get("context"),
|
||||
expected_output=task.get("expected_output"),
|
||||
constraints=task.get("constraints"),
|
||||
allowed_resources=allowed_resources,
|
||||
deadline_at=deadline_at,
|
||||
task_contract=task.get("task_contract"),
|
||||
workspace_root=task.get("workspace_root"),
|
||||
workspace_mode=workspace_mode,
|
||||
memory_mode=memory_mode,
|
||||
child_drive_root=str(child_drive or ""),
|
||||
budget_drive_root=str(drive_root) if child_drive is not None else "",
|
||||
artifacts=artifacts,
|
||||
artifact_status=ARTIFACT_STATUS_FAILED if workspace_root else "",
|
||||
metadata=metadata,
|
||||
result=f"Failed to enqueue task: {exc}",
|
||||
)
|
||||
return json_exception(exc, 503)
|
||||
return JSONResponse({"ok": True, "task_id": task_id, "status": STATUS_SCHEDULED})
|
||||
|
||||
|
||||
async def api_tasks_list(request: Request) -> JSONResponse:
|
||||
|
|
@ -982,12 +1158,24 @@ def _supervisor_ready_error(request: Request) -> Optional[JSONResponse]:
|
|||
if ready_event is not None and not ready_event.is_set():
|
||||
return json_error("supervisor is still starting", 503)
|
||||
try:
|
||||
from supervisor.workers import WORKERS
|
||||
from supervisor.workers import worker_pool_admission_state
|
||||
|
||||
if ready_event is not None and not WORKERS:
|
||||
return json_error("supervisor has no running workers", 503)
|
||||
except Exception:
|
||||
pass
|
||||
pool_state = worker_pool_admission_state()
|
||||
if ready_event is not None and not pool_state["available"]:
|
||||
return json_error(
|
||||
"supervisor worker pool is unavailable",
|
||||
503,
|
||||
reason_code="worker_pool_unavailable",
|
||||
worker_pool_disabled_reason=str(pool_state.get("disabled_reason") or ""),
|
||||
)
|
||||
except Exception as exc:
|
||||
if ready_event is not None:
|
||||
return json_error(
|
||||
"supervisor worker-pool state is unavailable",
|
||||
503,
|
||||
reason_code="worker_pool_state_unavailable",
|
||||
detail=f"{type(exc).__name__}: {exc}",
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -29,7 +29,7 @@ def write_owner_message(
|
|||
task_id: str,
|
||||
msg_id: Optional[str] = None,
|
||||
kind: str = KIND_OWNER_TEXT,
|
||||
) -> None:
|
||||
) -> bool:
|
||||
"""Write an owner message or typed control entry to a task's mailbox."""
|
||||
path = _mailbox_path(drive_root, task_id)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
|
@ -42,8 +42,11 @@ def write_owner_message(
|
|||
try:
|
||||
if not append_jsonl(path, entry):
|
||||
log.warning("Failed to durably append owner message for task %s", task_id)
|
||||
return False
|
||||
return True
|
||||
except Exception:
|
||||
log.warning("Failed to write owner message for task %s", task_id, exc_info=True)
|
||||
return False
|
||||
|
||||
|
||||
def drain_owner_entries(
|
||||
|
|
|
|||
|
|
@ -109,6 +109,7 @@ def record_process(
|
|||
purpose: str,
|
||||
scope: str,
|
||||
owner_task_id: str = "",
|
||||
reap_process_group: bool = True,
|
||||
) -> Dict[str, Any]:
|
||||
"""Append a custody record for an already-spawned process."""
|
||||
if scope not in _VALID_SCOPES:
|
||||
|
|
@ -120,7 +121,7 @@ def record_process(
|
|||
entry = {
|
||||
"ts": utc_now_iso(),
|
||||
"pid": int(pid),
|
||||
"pgid": int(pgid or 0),
|
||||
"pgid": int(pgid or 0) if reap_process_group else 0,
|
||||
"fingerprint": {
|
||||
"start_time": process_start_time(pid),
|
||||
# The LIVE command line (what the OS reports) is the reap-time
|
||||
|
|
|
|||
|
|
@ -152,6 +152,16 @@ def latest_chat_annotations(drive_root: Any) -> Dict[str, Dict[str, Any]]:
|
|||
return _latest_annotations(path)
|
||||
|
||||
|
||||
def chat_annotation_receipt(
|
||||
drive_root: Any, client_message_id: str, routing_token: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""Return the exact token-bound annotation for one routing attempt."""
|
||||
row = latest_chat_annotations(drive_root).get(str(client_message_id or ""), {})
|
||||
if str(row.get("routing_token") or "") != str(routing_token or ""):
|
||||
return {}
|
||||
return dict(row)
|
||||
|
||||
|
||||
def _compact_annotations_locked(drive_root: Any, path: pathlib.Path) -> None:
|
||||
if not path.is_file() or path.stat().st_size < _COMPACT_AT_BYTES:
|
||||
return
|
||||
|
|
@ -186,6 +196,10 @@ def append_chat_annotation(
|
|||
action: str,
|
||||
target: str = "",
|
||||
status: str,
|
||||
routing_token: str = "",
|
||||
reason: str = "",
|
||||
detail: str = "",
|
||||
options: Any = None,
|
||||
) -> bool:
|
||||
"""Append one compact UI annotation; no semantic routing state is stored."""
|
||||
message_id = str(client_message_id or "").strip()
|
||||
|
|
@ -199,6 +213,14 @@ def append_chat_annotation(
|
|||
"target": str(target or "")[:200],
|
||||
"status": str(status or "")[:80],
|
||||
}
|
||||
if str(routing_token or ""):
|
||||
row["routing_token"] = str(routing_token)[:128]
|
||||
if str(reason or ""):
|
||||
row["reason"] = str(reason)[:200]
|
||||
if str(detail or ""):
|
||||
row["detail"] = str(detail)[:1000]
|
||||
if isinstance(options, list):
|
||||
row["options"] = [dict(item) for item in options[:100] if isinstance(item, dict)]
|
||||
path = pathlib.Path(drive_root) / "logs" / _ANNOTATIONS_NAME
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
lock_path = jsonl_append_lock_path(path)
|
||||
|
|
@ -224,6 +246,7 @@ def append_chat_annotation(
|
|||
__all__ = [
|
||||
"append_chat_annotation",
|
||||
"build_owner_message_ref",
|
||||
"chat_annotation_receipt",
|
||||
"entry_matches_source_ref",
|
||||
"latest_chat_annotations",
|
||||
"project_origin_rows",
|
||||
|
|
|
|||
97
ouroboros/promotion_source.py
Normal file
97
ouroboros/promotion_source.py
Normal file
|
|
@ -0,0 +1,97 @@
|
|||
"""Supervisor-side source admission for promoted conversation work."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pathlib
|
||||
from typing import Any, Tuple
|
||||
|
||||
|
||||
def _source_project_id(source: str, is_git: bool) -> str:
|
||||
from ouroboros.project_facts import project_id_from_display_name
|
||||
from ouroboros.project_sources import derive_repo_dir_name
|
||||
|
||||
base = derive_repo_dir_name(source) if is_git else pathlib.Path(source.rstrip("/")).name
|
||||
return project_id_from_display_name(base or "project")
|
||||
|
||||
|
||||
def resolve_promote_source(
|
||||
ctx: Any, source: str, project_id: str,
|
||||
) -> Tuple[str, str, str, str]:
|
||||
"""Attach/clone only after the supervisor has admitted an executor.
|
||||
|
||||
Returns ``(workspace_root, note, error, effective_project_id)``. Keeping
|
||||
this side effect on the authoritative handler side prevents a stale tool
|
||||
snapshot from cloning/registering a project after the worker pool was
|
||||
disabled but before the queued event was rejected.
|
||||
"""
|
||||
from ouroboros.config import DATA_DIR
|
||||
from ouroboros.project_sources import clone_project_repo, valid_git_url, validate_attach_path
|
||||
|
||||
src = str(source or "").strip()
|
||||
pid = str(project_id or "").strip()
|
||||
drive_root = pathlib.Path(getattr(ctx, "DRIVE_ROOT", DATA_DIR))
|
||||
if not src:
|
||||
return "", "", "", pid
|
||||
is_git = valid_git_url(src)
|
||||
pid = pid or _source_project_id(src, is_git)
|
||||
try:
|
||||
from ouroboros.projects_registry import get_reserved_project
|
||||
|
||||
existing = get_reserved_project(drive_root, pid)
|
||||
except Exception as exc:
|
||||
return "", "", f"project_lookup_failed: {type(exc).__name__}: {exc}", pid
|
||||
lifecycle = str((existing or {}).get("lifecycle") or "active")
|
||||
if existing is not None and lifecycle != "active":
|
||||
return "", "", f"project_routing_fence: {pid!r} is {lifecycle}", pid
|
||||
if is_git:
|
||||
if str((existing or {}).get("working_dir") or "").strip():
|
||||
return "", "", (
|
||||
f"conflict: project {pid!r} already has folder {existing.get('working_dir')}; "
|
||||
"use another project id or omit source"
|
||||
), pid
|
||||
cloned, code, detail = clone_project_repo(src, pid)
|
||||
if code:
|
||||
return "", "", f"{code}: {detail}", pid
|
||||
folder, provenance, clone_url = cloned, "cloned", src
|
||||
note = f"cloned {src} -> {cloned}"
|
||||
else:
|
||||
from ouroboros.project_sources import is_git_worktree_root
|
||||
|
||||
resolved, err = validate_attach_path(
|
||||
src,
|
||||
system_repo_dir=getattr(ctx, "REPO_DIR", getattr(ctx, "repo_dir", "")),
|
||||
drive_root=drive_root,
|
||||
)
|
||||
if err:
|
||||
return "", "", f"attach: {err}", pid
|
||||
if not is_git_worktree_root(resolved):
|
||||
return "", "", f"attach: {resolved} is not a git repository", pid
|
||||
folder, provenance, clone_url = str(resolved), "attached", ""
|
||||
note = f"attached {resolved}"
|
||||
prior_wd = str((existing or {}).get("working_dir") or "").strip()
|
||||
if prior_wd and prior_wd != folder:
|
||||
return "", "", (
|
||||
f"conflict: project {pid!r} already has folder {prior_wd}; use another project id "
|
||||
"or omit source"
|
||||
), pid
|
||||
if prior_wd == folder and str((existing or {}).get("provenance") or "").strip() not in ("", "none"):
|
||||
return folder, note, "", pid
|
||||
try:
|
||||
from ouroboros.projects_registry import create_project, update_project
|
||||
from ouroboros.utils import utc_now_iso
|
||||
|
||||
create_project(drive_root, pid, origin="promote_chat_to_task")
|
||||
update_project(
|
||||
drive_root,
|
||||
pid,
|
||||
working_dir=folder,
|
||||
provenance=provenance,
|
||||
clone_url=clone_url,
|
||||
trusted_at=utc_now_iso(),
|
||||
)
|
||||
except Exception as exc:
|
||||
return "", "", f"register: {type(exc).__name__}: {exc}", pid
|
||||
return folder, note, "", pid
|
||||
|
||||
|
||||
__all__ = ["resolve_promote_source"]
|
||||
|
|
@ -56,6 +56,8 @@ VALID_SUBTASK_MEMORY_MODES = frozenset({"forked", "empty"})
|
|||
# schedule_subagent emission within one tool-call round. Process-local: a parent
|
||||
# ctx is never shared across processes, so a threading.Lock is sufficient.
|
||||
_SCHEDULE_EMIT_LOCK = threading.Lock()
|
||||
_PROMOTE_CONFIRM_TIMEOUT_SEC = 15.0
|
||||
_PROMOTE_CONFIRM_POLL_SEC = 0.05
|
||||
|
||||
|
||||
def _record_scheduled_subagent(ctx: ToolContext, record: Dict[str, Any]) -> None:
|
||||
|
|
@ -291,11 +293,54 @@ def _emit_control_event(ctx: ToolContext, evt: Dict[str, Any]) -> str:
|
|||
)
|
||||
setattr(ctx, "_typed_routing_action_emitted", action)
|
||||
|
||||
try:
|
||||
from multiprocessing.reduction import ForkingPickler
|
||||
|
||||
ForkingPickler.dumps(dict(evt))
|
||||
except Exception as exc:
|
||||
log.warning("Control event is not multiprocessing-serializable", exc_info=True)
|
||||
try:
|
||||
root = Path(str(getattr(ctx, "budget_drive_root", "") or ctx.drive_root))
|
||||
append_jsonl(
|
||||
root / "logs" / "supervisor.jsonl",
|
||||
{
|
||||
"ts": utc_now_iso(),
|
||||
"type": "control_event_serialization_failed",
|
||||
"event_type": str(evt.get("type") or ""),
|
||||
"task_id": str(evt.get("task_id") or ""),
|
||||
"routing_token": str(evt.get("routing_token") or ""),
|
||||
"error": f"{type(exc).__name__}: {exc}",
|
||||
},
|
||||
)
|
||||
except Exception:
|
||||
log.debug("Failed to record control-event serialization failure", exc_info=True)
|
||||
return "serialization_failed"
|
||||
|
||||
def _record_emitted(mode: str) -> None:
|
||||
if str(evt.get("type") or "") != "promote_chat_to_task":
|
||||
return
|
||||
try:
|
||||
root = Path(str(getattr(ctx, "budget_drive_root", "") or ctx.drive_root))
|
||||
append_jsonl(
|
||||
root / "logs" / "supervisor.jsonl",
|
||||
{
|
||||
"ts": utc_now_iso(),
|
||||
"type": "promote_chat_to_task_emitted",
|
||||
"task_id": str(evt.get("task_id") or ""),
|
||||
"routing_token": str(evt.get("routing_token") or ""),
|
||||
"transport_mode": mode,
|
||||
"sender_pid": os.getpid(),
|
||||
},
|
||||
)
|
||||
except Exception:
|
||||
log.debug("Failed to record promote emission", exc_info=True)
|
||||
|
||||
event_queue = getattr(ctx, "event_queue", None)
|
||||
if event_queue is not None:
|
||||
try:
|
||||
event_queue.put_nowait(dict(evt))
|
||||
_mark_typed_routing_action()
|
||||
_record_emitted("live")
|
||||
return "live"
|
||||
except (AttributeError, queue.Full):
|
||||
pass
|
||||
|
|
@ -304,9 +349,133 @@ def _emit_control_event(ctx: ToolContext, evt: Dict[str, Any]) -> str:
|
|||
with _SCHEDULE_EMIT_LOCK:
|
||||
ctx.pending_events.append(evt)
|
||||
_mark_typed_routing_action()
|
||||
_record_emitted("deferred")
|
||||
return "deferred"
|
||||
|
||||
|
||||
def _promotion_pool_disabled_from_snapshot(ctx: ToolContext) -> str:
|
||||
"""Cheap early refusal for the known crash-storm state.
|
||||
|
||||
The supervisor handler remains authoritative. This projection only keeps
|
||||
source/project side effects from starting when the latest durable snapshot
|
||||
already says the executor pool was deliberately disabled.
|
||||
"""
|
||||
try:
|
||||
root = Path(str(getattr(ctx, "budget_drive_root", "") or ctx.drive_root))
|
||||
snapshot = json.loads(
|
||||
(root / "state" / "queue_snapshot.json").read_text(encoding="utf-8")
|
||||
)
|
||||
reason = str(snapshot.get("worker_pool_disabled_reason") or "")
|
||||
if reason not in {"", "unknown"}:
|
||||
return reason
|
||||
if int(snapshot.get("worker_total") or 0) <= 0:
|
||||
return "no_workers"
|
||||
return ""
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
|
||||
def _routing_status_root(ctx: ToolContext) -> Path:
|
||||
return Path(str(getattr(ctx, "budget_drive_root", "") or ctx.drive_root))
|
||||
|
||||
|
||||
def _wait_for_promotion_admission(
|
||||
ctx: ToolContext,
|
||||
task_id: str,
|
||||
routing_token: str,
|
||||
*,
|
||||
client_message_id: str = "",
|
||||
timeout_sec: float = _PROMOTE_CONFIRM_TIMEOUT_SEC,
|
||||
) -> Dict[str, Any]:
|
||||
"""Wait for matching-token admission in the canonical task-result SSOT."""
|
||||
from ouroboros.task_results import load_task_result
|
||||
|
||||
root = _routing_status_root(ctx)
|
||||
deadline = time.monotonic() + max(0.0, float(timeout_sec))
|
||||
while True:
|
||||
result = load_task_result(root, task_id) or {}
|
||||
admission = result.get("promotion_admission")
|
||||
if (
|
||||
isinstance(admission, dict)
|
||||
and str(admission.get("routing_token") or "") == routing_token
|
||||
):
|
||||
status = str(admission.get("status") or "")
|
||||
if status in {"scheduled", "rejected", "unconfirmed"}:
|
||||
return {**admission, "task_status": str(result.get("status") or "")}
|
||||
# A duplicate id must never overwrite the existing task_result merely
|
||||
# to report the loser. The exact-token chat annotation is therefore a
|
||||
# negative-only fallback; positive scheduling authority stays solely in
|
||||
# the task-result admission record.
|
||||
if str(client_message_id or "").strip():
|
||||
from ouroboros.project_dialogue import chat_annotation_receipt
|
||||
|
||||
receipt = chat_annotation_receipt(
|
||||
root, str(client_message_id), routing_token
|
||||
)
|
||||
if str(receipt.get("status") or "") in {
|
||||
"needs_manual_target",
|
||||
"rejected",
|
||||
"unconfirmed",
|
||||
}:
|
||||
return receipt
|
||||
if time.monotonic() >= deadline:
|
||||
return {"status": "unconfirmed", "reason": "confirmation_timeout"}
|
||||
time.sleep(_PROMOTE_CONFIRM_POLL_SEC)
|
||||
|
||||
|
||||
def _wait_for_routing_annotation(
|
||||
ctx: ToolContext,
|
||||
client_message_id: str,
|
||||
routing_token: str,
|
||||
*,
|
||||
timeout_sec: float = _PROMOTE_CONFIRM_TIMEOUT_SEC,
|
||||
) -> Dict[str, Any]:
|
||||
"""Wait for an exact existing chat-annotation receipt (manual/steer)."""
|
||||
from ouroboros.project_dialogue import chat_annotation_receipt
|
||||
|
||||
if not str(client_message_id or "").strip():
|
||||
return {"status": "unconfirmed", "reason": "client_message_id_missing"}
|
||||
root = _routing_status_root(ctx)
|
||||
deadline = time.monotonic() + max(0.0, float(timeout_sec))
|
||||
while True:
|
||||
receipt = chat_annotation_receipt(root, client_message_id, routing_token)
|
||||
status = str(receipt.get("status") or "")
|
||||
if status in {"delivered", "needs_manual_target", "unconfirmed"}:
|
||||
return receipt
|
||||
if time.monotonic() >= deadline:
|
||||
return {"status": "unconfirmed", "reason": "confirmation_timeout"}
|
||||
time.sleep(_PROMOTE_CONFIRM_POLL_SEC)
|
||||
|
||||
|
||||
def _emit_and_wait_for_routing(
|
||||
ctx: ToolContext,
|
||||
evt: Dict[str, Any],
|
||||
) -> tuple[str, Dict[str, Any]]:
|
||||
"""Emit one routing event and return only its durable handler outcome."""
|
||||
mode = _emit_control_event(ctx, evt)
|
||||
if mode == "serialization_failed":
|
||||
return mode, {
|
||||
"status": "rejected",
|
||||
"reason": "event_serialization_failed",
|
||||
"detail": "The routing event was not emitted.",
|
||||
}
|
||||
timeout = _PROMOTE_CONFIRM_TIMEOUT_SEC if mode == "live" else 0.0
|
||||
if str(evt.get("type") or "") == "promote_chat_to_task":
|
||||
return mode, _wait_for_promotion_admission(
|
||||
ctx,
|
||||
str(evt.get("task_id") or ""),
|
||||
str(evt.get("routing_token") or ""),
|
||||
client_message_id=str(evt.get("client_message_id") or ""),
|
||||
timeout_sec=timeout,
|
||||
)
|
||||
return mode, _wait_for_routing_annotation(
|
||||
ctx,
|
||||
str(evt.get("client_message_id") or ""),
|
||||
str(evt.get("routing_token") or ""),
|
||||
timeout_sec=timeout,
|
||||
)
|
||||
|
||||
|
||||
def _evolution_restart_block_reason(ctx: ToolContext) -> str:
|
||||
if str(ctx.current_task_type or "") != "evolution":
|
||||
return ""
|
||||
|
|
@ -382,106 +551,6 @@ def _promote_to_stable(ctx: ToolContext, reason: str) -> str:
|
|||
return f"Promote to stable requested: {reason}"
|
||||
|
||||
|
||||
def _derive_source_project_id(src: str, is_git: bool) -> str:
|
||||
"""A filesystem-clean project id from the source itself (repo/folder name) —
|
||||
the `promote_chat_to_task(source=…)` one-liner registers a project even when
|
||||
the agent supplied no project_id/name (triad r2: a main-chat promote used to
|
||||
attach the folder but silently skip registration, breaking the documented
|
||||
capability). Deterministic-hash fallback for non-ASCII names."""
|
||||
import pathlib as _pl
|
||||
|
||||
from ouroboros.project_facts import project_id_from_display_name
|
||||
from ouroboros.project_sources import derive_repo_dir_name
|
||||
|
||||
base = derive_repo_dir_name(src) if is_git else _pl.Path(str(src).rstrip("/")).name
|
||||
return project_id_from_display_name(base or "project")
|
||||
|
||||
|
||||
def _resolve_promote_source(ctx: ToolContext, source: str, pid: str) -> tuple[str, str, str, str]:
|
||||
"""v6.59.0 (3.3): agent-side attach/clone through the SAME server primitives the
|
||||
New Project dialog uses. ``source`` is a git URL (cloned into the projects root)
|
||||
or an existing folder path (validated attach). Returns (workspace_root, note,
|
||||
error, effective_pid): on success the folder is registered on the project
|
||||
(working_dir + provenance + trusted_at at the canonical DATA_DIR) so the room
|
||||
and later tasks inherit it — a missing pid is DERIVED from the source name so
|
||||
registration never silently skips. Re-sourcing an existing project whose
|
||||
working_dir differs is refused (mirrors the gateway 409, triad r2 scope
|
||||
advisory); a folder-less existing project may gain its first folder. The agent
|
||||
reports loudly; no confirmation wait (owner quiz 13)."""
|
||||
from ouroboros.config import DATA_DIR
|
||||
from ouroboros.project_sources import clone_project_repo, valid_git_url, validate_attach_path
|
||||
|
||||
src = str(source or "").strip()
|
||||
if not src:
|
||||
return "", "", "", pid
|
||||
is_git = valid_git_url(src)
|
||||
pid = str(pid or "").strip() or _derive_source_project_id(src, is_git)
|
||||
try:
|
||||
from ouroboros.projects_registry import get_project
|
||||
|
||||
existing = get_project(DATA_DIR, pid)
|
||||
except Exception:
|
||||
existing = None
|
||||
if is_git:
|
||||
if str((existing or {}).get("working_dir") or "").strip():
|
||||
# Conflict BEFORE the clone side effect (triad r7): a git source always
|
||||
# creates a NEW folder, so it can never match an existing bound folder —
|
||||
# cloning first would leave a dangling directory behind the refusal.
|
||||
return "", "", (
|
||||
f"⚠️ PROJECT_SOURCE_ERROR (conflict): project {pid!r} already has folder "
|
||||
f"{existing.get('working_dir')} — re-sourcing an existing project is not "
|
||||
"supported; pass a different project_id/project_name or omit source to "
|
||||
"work in the existing folder."
|
||||
), pid
|
||||
cloned, code, detail = clone_project_repo(src, pid or "")
|
||||
if code:
|
||||
return "", "", f"⚠️ PROJECT_SOURCE_ERROR ({code}): {detail}", pid
|
||||
folder, provenance, clone_url = cloned, "cloned", src
|
||||
note = f" [cloned {src} -> {cloned}]"
|
||||
else:
|
||||
from ouroboros.project_sources import is_git_worktree_root
|
||||
|
||||
resolved, err = validate_attach_path(
|
||||
src, system_repo_dir=getattr(ctx, "repo_dir", ""), drive_root=DATA_DIR
|
||||
)
|
||||
if err:
|
||||
return "", "", f"⚠️ PROJECT_SOURCE_ERROR (attach): {err}", pid
|
||||
if not is_git_worktree_root(resolved):
|
||||
# Task admission requires a git worktree root; attaching a non-git
|
||||
# folder would register a project whose room tasks are born dead
|
||||
# (triad r5). Owner opt-in git-init lives in the New Project dialog.
|
||||
return "", "", (
|
||||
f"⚠️ PROJECT_SOURCE_ERROR (attach): {resolved} is not a git repository — "
|
||||
"ask the owner to create the project via New Project with init_git enabled, "
|
||||
"or have them git-init the folder first."
|
||||
), pid
|
||||
folder, provenance, clone_url = str(resolved), "attached", ""
|
||||
note = f" [attached {resolved} — I now have write+shell there for this project's tasks]"
|
||||
prior_wd = str((existing or {}).get("working_dir") or "").strip()
|
||||
if prior_wd and prior_wd != folder:
|
||||
return "", "", (
|
||||
f"⚠️ PROJECT_SOURCE_ERROR (conflict): project {pid!r} already has folder "
|
||||
f"{prior_wd} — re-sourcing an existing project is not supported; pass a "
|
||||
"different project_id/project_name or omit source to work in the existing folder."
|
||||
), pid
|
||||
if prior_wd == folder and str((existing or {}).get("provenance") or "").strip() not in ("", "none"):
|
||||
# Same folder, already stamped: idempotent — keep the original trusted_at.
|
||||
return folder, note, "", pid
|
||||
try:
|
||||
from ouroboros.projects_registry import create_project, update_project
|
||||
from ouroboros.utils import utc_now_iso
|
||||
|
||||
create_project(DATA_DIR, pid, origin="promote_chat_to_task")
|
||||
update_project(
|
||||
DATA_DIR, pid,
|
||||
working_dir=folder, provenance=provenance,
|
||||
clone_url=clone_url, trusted_at=utc_now_iso(),
|
||||
)
|
||||
except Exception as exc:
|
||||
return "", "", f"⚠️ PROJECT_SOURCE_ERROR (register): {type(exc).__name__}: {exc}", pid
|
||||
return folder, note, "", pid
|
||||
|
||||
|
||||
def _attach_origin_from_metadata(ctx: ToolContext, evt: Dict[str, Any]) -> None:
|
||||
"""Copy the ingress-captured owner-message origin (ref + full text) onto a
|
||||
promote-shaped event BY VALUE. The host built the ref at chat admission;
|
||||
|
|
@ -558,24 +627,28 @@ def _promote_chat_to_task(
|
|||
current_chat_id = int(getattr(ctx, "current_chat_id", None) or 0)
|
||||
except (TypeError, ValueError):
|
||||
current_chat_id = 0
|
||||
# v6.59.0 (3.3): attach/clone the source BEFORE emitting, so the event already
|
||||
# carries the resolved folder ("help me debug this GitHub project" one-liner).
|
||||
# The effective pid comes back: a source given without project_id/name derives
|
||||
# the project from the source name, so registration never silently skips.
|
||||
source_folder, source_note, source_error, pid = _resolve_promote_source(ctx, source, pid)
|
||||
if source_error:
|
||||
return source_error
|
||||
effective_workspace_root = str(workspace_root or "").strip() or source_folder
|
||||
tid = uuid.uuid4().hex[:8]
|
||||
tid = uuid.uuid4().hex[:16]
|
||||
routing_token = uuid.uuid4().hex
|
||||
disabled_reason = _promotion_pool_disabled_from_snapshot(ctx)
|
||||
if disabled_reason:
|
||||
return (
|
||||
f"PROMOTE_REJECTED: task {tid} was not scheduled "
|
||||
f"(worker_pool_unavailable: {disabled_reason}). No project/workspace "
|
||||
"admission side effects were started."
|
||||
)
|
||||
evt: Dict[str, Any] = {
|
||||
"type": "promote_chat_to_task",
|
||||
"task_id": tid,
|
||||
"routing_token": routing_token,
|
||||
"objective": goal,
|
||||
"expected_output": str(expected_output or "").strip(),
|
||||
"project_id": pid,
|
||||
"project_name": display_name,
|
||||
"title": str(title or "").strip()[:80],
|
||||
"workspace_root": effective_workspace_root,
|
||||
"workspace_root": str(workspace_root or "").strip(),
|
||||
# Source admission is intentionally supervisor-side, after the
|
||||
# authoritative worker-pool and duplicate-id gates.
|
||||
"source": str(source or "").strip(),
|
||||
# v6.58.0: "none" opts a project-room task OUT of the room's working_dir
|
||||
# default (a folder-less task in a folder-ful project stays possible).
|
||||
"workspace": str(workspace or "").strip().lower(),
|
||||
|
|
@ -591,18 +664,60 @@ def _promote_chat_to_task(
|
|||
"ts": utc_now_iso(),
|
||||
}
|
||||
_attach_origin_from_metadata(ctx, evt)
|
||||
mode = _emit_control_event(ctx, evt)
|
||||
mode, confirmation = _emit_and_wait_for_routing(ctx, evt)
|
||||
if display_name:
|
||||
scope_note = f" in new project '{display_name}'"
|
||||
elif pid:
|
||||
scope_note = f" in project '{pid}'"
|
||||
else:
|
||||
scope_note = ""
|
||||
confirmation_status = str(confirmation.get("status") or "unconfirmed")
|
||||
reason = str(confirmation.get("reason") or "")
|
||||
detail = str(confirmation.get("detail") or "")
|
||||
disabled_reason = str(confirmation.get("worker_pool_disabled_reason") or "")
|
||||
if confirmation_status == "scheduled":
|
||||
source_confirmation = f" [{detail}]" if detail else ""
|
||||
return (
|
||||
f"OK: task {tid}{scope_note} accepted and durably scheduled ({mode}).{source_confirmation} "
|
||||
"The conversation lane stays free; the owner sees a live task card and can "
|
||||
"steer the running task from chat. Use wait_task/get_task_result if its result "
|
||||
"is needed in this conversation."
|
||||
)
|
||||
if confirmation_status in {"rejected", "needs_manual_target"}:
|
||||
shown_reason = (
|
||||
f"{reason}: {disabled_reason}" if disabled_reason else reason
|
||||
)
|
||||
if detail:
|
||||
shown_reason = f"{shown_reason}: {detail}" if shown_reason else detail
|
||||
return (
|
||||
f"PROMOTE_REJECTED: task {tid} was not scheduled"
|
||||
f"{f' ({shown_reason})' if shown_reason else ''}. "
|
||||
"Do not report this task as created."
|
||||
)
|
||||
try:
|
||||
root = Path(str(getattr(ctx, "budget_drive_root", "") or ctx.drive_root))
|
||||
append_jsonl(
|
||||
root / "logs" / "supervisor.jsonl",
|
||||
{
|
||||
"ts": utc_now_iso(),
|
||||
"type": "promote_chat_to_task_unconfirmed",
|
||||
"task_id": tid,
|
||||
"transport_mode": mode,
|
||||
"reason": reason or "confirmation_timeout",
|
||||
"routing_token": routing_token,
|
||||
},
|
||||
)
|
||||
except Exception:
|
||||
log.debug("Failed to record unconfirmed promote", exc_info=True)
|
||||
confirmation_window = (
|
||||
f"within {int(_PROMOTE_CONFIRM_TIMEOUT_SEC)} seconds"
|
||||
if mode == "live"
|
||||
else f"because the event transport returned {mode}"
|
||||
)
|
||||
return (
|
||||
f"OK: promoted to supervised task {tid}{scope_note} ({mode}).{source_note} The conversation "
|
||||
"lane stays free; the owner sees a live task card and can steer the running "
|
||||
"task from chat (messages are delivered to its mailbox). Use wait_task/"
|
||||
"get_task_result to follow up if the result is needed in this conversation."
|
||||
f"PROMOTE_UNCONFIRMED: task {tid} admission was not confirmed {confirmation_window}. "
|
||||
"Do not report this task as "
|
||||
"created and do not retry automatically; keep this task id for reconciliation."
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -671,8 +786,10 @@ def _route_to_project(
|
|||
else "invalid_project_id" if not pid
|
||||
else "target_not_found"
|
||||
)
|
||||
mode = _emit_control_event(ctx, {
|
||||
routing_token = uuid.uuid4().hex
|
||||
mode, receipt = _emit_and_wait_for_routing(ctx, {
|
||||
"type": "routing_manual_target",
|
||||
"routing_token": routing_token,
|
||||
"chat_id": current_chat_id,
|
||||
"client_message_id": client_message_id,
|
||||
"requested_target": pid or requested_pid[:200],
|
||||
|
|
@ -680,15 +797,26 @@ def _route_to_project(
|
|||
"options": options,
|
||||
"ts": utc_now_iso(),
|
||||
})
|
||||
if str(receipt.get("status") or "") == "needs_manual_target":
|
||||
durable_options = (
|
||||
receipt.get("options") if isinstance(receipt.get("options"), list) else options
|
||||
)
|
||||
options_text = json.dumps(durable_options, ensure_ascii=False, default=str)
|
||||
return (
|
||||
f"⚠️ NEEDS_MANUAL_TARGET ({failure}, {mode}): no route was dispatched. "
|
||||
f"Host-validated options: {options_text}"
|
||||
)
|
||||
return (
|
||||
f"⚠️ NEEDS_MANUAL_TARGET ({failure}, {mode}): no route was dispatched. "
|
||||
"The owner received the concrete host-validated task/project options."
|
||||
f"⚠️ ROUTING_UNCONFIRMED ({failure}, {mode}): no route was dispatched and "
|
||||
"delivery of the manual target options was not confirmed."
|
||||
)
|
||||
tid = uuid.uuid4().hex[:8]
|
||||
tid = uuid.uuid4().hex[:16]
|
||||
routing_token = uuid.uuid4().hex
|
||||
objective = msg if not str(reason or "").strip() else f"{msg}\n\n(routing reason: {str(reason).strip()})"
|
||||
evt: Dict[str, Any] = {
|
||||
"type": "promote_chat_to_task",
|
||||
"task_id": tid,
|
||||
"routing_token": routing_token,
|
||||
"objective": objective,
|
||||
"project_id": pid,
|
||||
"chat_id": current_chat_id,
|
||||
|
|
@ -701,11 +829,24 @@ def _route_to_project(
|
|||
"ts": utc_now_iso(),
|
||||
}
|
||||
_attach_origin_from_metadata(ctx, evt)
|
||||
mode = _emit_control_event(ctx, evt)
|
||||
mode, receipt = _emit_and_wait_for_routing(ctx, evt)
|
||||
name = str(proj.get("name") or pid)
|
||||
status = str(receipt.get("status") or "unconfirmed")
|
||||
if status == "scheduled":
|
||||
return (
|
||||
f"✉️ Routed to project '{name}' ({pid}) as task {tid}; admission is durably "
|
||||
f"scheduled ({mode}). I'll continue there; this chat stays free for you."
|
||||
)
|
||||
reason_text = str(receipt.get("reason") or "confirmation_timeout")
|
||||
detail = str(receipt.get("detail") or "")
|
||||
if status in {"rejected", "needs_manual_target"}:
|
||||
return (
|
||||
f"⚠️ ROUTE_REJECTED: task {tid} was not routed to project '{name}' "
|
||||
f"({reason_text}{(': ' + detail) if detail else ''})."
|
||||
)
|
||||
return (
|
||||
f"✉️ Routed to project '{name}' ({pid}) as task {tid} ({mode}). I'll continue there; "
|
||||
"this chat stays free for you. Follow-ups you send reach the project task's mailbox."
|
||||
f"⚠️ ROUTE_UNCONFIRMED: task {tid} routing to project '{name}' was not durably "
|
||||
"confirmed. Do not report it as routed and do not retry automatically."
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -746,6 +887,7 @@ def _steer_task(ctx: ToolContext, task_id: str, message: str) -> str:
|
|||
)
|
||||
evt: Dict[str, Any] = {
|
||||
"type": "steer_task",
|
||||
"routing_token": uuid.uuid4().hex,
|
||||
"target_task_id": target,
|
||||
"message": msg,
|
||||
"chat_id": current_chat_id,
|
||||
|
|
@ -758,11 +900,21 @@ def _steer_task(ctx: ToolContext, task_id: str, message: str) -> str:
|
|||
if isinstance(_md, dict) else [],
|
||||
"ts": utc_now_iso(),
|
||||
}
|
||||
mode = _emit_control_event(ctx, evt)
|
||||
mode, receipt = _emit_and_wait_for_routing(ctx, evt)
|
||||
status = str(receipt.get("status") or "unconfirmed")
|
||||
if status == "delivered":
|
||||
return (
|
||||
f"✉️ Steering task {target}: mailbox delivery is durably confirmed ({mode}). "
|
||||
"The task receives it at its next checkpoint."
|
||||
)
|
||||
if status in {"rejected", "needs_manual_target"}:
|
||||
return (
|
||||
f"⚠️ STEER_REJECTED: task {target} was not steered "
|
||||
f"({str(receipt.get('reason') or 'target_not_steerable')})."
|
||||
)
|
||||
return (
|
||||
f"✉️ Steering task {target} ({mode}): the message reaches its mailbox at the task's next "
|
||||
"checkpoint. If that task has already finished, you'll get a notice — then answer inline "
|
||||
"or promote_chat_to_task instead."
|
||||
f"⚠️ STEER_UNCONFIRMED: mailbox delivery to task {target} was not durably confirmed "
|
||||
f"({mode}). Do not report the message as delivered."
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -1798,7 +1950,9 @@ _PROMOTE_CHAT_DESCRIPTION = (
|
|||
"the project's working folder as its ACTIVE WORKSPACE by default (its file/"
|
||||
"shell/git tools operate there, not on the Ouroboros repo); pass "
|
||||
"workspace='none' for a folder-less task. Owner follow-ups reach the "
|
||||
"running task via its mailbox."
|
||||
"running task via its mailbox. Report creation only when this tool returns "
|
||||
"OK; PROMOTE_REJECTED or PROMOTE_UNCONFIRMED means the task must not be "
|
||||
"claimed as created, and UNCONFIRMED must not be retried automatically."
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -403,7 +403,11 @@ def _planning_swarm_timing(ctx: ToolContext) -> tuple[float, float]:
|
|||
remaining = (deadline - _planning_now()).total_seconds()
|
||||
max_wait = 0.0 if remaining <= 0 else min(max_wait, remaining / 4.0)
|
||||
event_queue = getattr(ctx, "event_queue", None)
|
||||
live = event_queue is not None and event_queue.__class__.__module__ in {"queue", "multiprocessing.queues"}
|
||||
live = event_queue is not None and event_queue.__class__.__module__ in {
|
||||
"queue",
|
||||
"multiprocessing.queues",
|
||||
"multiprocessing.managers",
|
||||
}
|
||||
if not live:
|
||||
wait_timeout = min(wait_timeout, 0.25)
|
||||
max_wait = min(max_wait, wait_timeout)
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|||
|
||||
[project]
|
||||
name = "ouroboros"
|
||||
version = "6.87.2"
|
||||
version = "6.87.3"
|
||||
description = "Self-creating AI agent with constitution, background consciousness, and persistent identity"
|
||||
readme = "README.md"
|
||||
license = {text = "MIT"}
|
||||
|
|
|
|||
41
server.py
41
server.py
|
|
@ -444,7 +444,10 @@ def _route_project_chat_to_running_task(
|
|||
active_fence = ACCEPTANCE_FENCES.get(fence_root)
|
||||
if isinstance(active_fence, dict) and str(active_fence.get("status") or "") == "sealed":
|
||||
return ""
|
||||
write_owner_message(task_drive, f"{message}{attachment_note}", tid, msg_id=msg_id)
|
||||
if not write_owner_message(
|
||||
task_drive, f"{message}{attachment_note}", tid, msg_id=msg_id
|
||||
):
|
||||
return ""
|
||||
if direct_lock_held:
|
||||
direct_agent._owner_message_generation = int(
|
||||
getattr(direct_agent, "_owner_message_generation", 0) or 0
|
||||
|
|
@ -923,15 +926,16 @@ def _route_owner_message(bridge: Any, ctx: Any, incoming: Dict[str, Any]) -> Non
|
|||
# through the conversation decision lane would combine skill_repair with
|
||||
# _ephemeral_turn: ephemeral hides the repair mutators while heal mode
|
||||
# blocks promotion. Promote it directly without weakening either policy.
|
||||
from supervisor.workers import promote_chat_to_task
|
||||
from supervisor.events import _handle_promote_chat_to_task
|
||||
|
||||
ctx.consciousness.inject_observation(
|
||||
f"Message from my human: {incoming.get('log_text') or ''}"
|
||||
)
|
||||
task_id = uuid.uuid4().hex[:8]
|
||||
task_id = uuid.uuid4().hex[:16]
|
||||
event = {
|
||||
"type": "promote_chat_to_task",
|
||||
"task_id": task_id,
|
||||
"routing_token": uuid.uuid4().hex,
|
||||
"objective": text or image_caption,
|
||||
"chat_id": chat_id,
|
||||
"client_message_id": client_message_id,
|
||||
|
|
@ -945,7 +949,7 @@ def _route_owner_message(bridge: Any, ctx: Any, incoming: Dict[str, Any]) -> Non
|
|||
else:
|
||||
event["origin_suppressed"] = True
|
||||
try:
|
||||
outcome = promote_chat_to_task(event, ctx)
|
||||
outcome = _handle_promote_chat_to_task(event, ctx)
|
||||
except Exception:
|
||||
log.warning("Direct skill-repair promotion failed", exc_info=True)
|
||||
outcome = {
|
||||
|
|
@ -955,7 +959,15 @@ def _route_owner_message(bridge: Any, ctx: Any, incoming: Dict[str, Any]) -> Non
|
|||
}
|
||||
outcome = outcome if isinstance(outcome, dict) else {"status": "scheduled", "task_id": task_id}
|
||||
outcome_status = str(outcome.get("status") or "needs_manual_target")
|
||||
if outcome_status != "scheduled":
|
||||
if outcome_status == "scheduled":
|
||||
try:
|
||||
ctx.send_with_budget(
|
||||
chat_id,
|
||||
f"✅ Repair task {task_id} was accepted and durably scheduled.",
|
||||
)
|
||||
except Exception:
|
||||
log.debug("Repair promotion success notification failed", exc_info=True)
|
||||
else:
|
||||
reason = str(outcome.get("reason") or outcome_status)
|
||||
try:
|
||||
ctx.send_with_budget(
|
||||
|
|
@ -964,15 +976,6 @@ def _route_owner_message(bridge: Any, ctx: Any, incoming: Dict[str, Any]) -> Non
|
|||
)
|
||||
except Exception:
|
||||
log.debug("Repair promotion refusal notification failed", exc_info=True)
|
||||
_record_routing_receipt(
|
||||
bridge,
|
||||
ctx,
|
||||
chat_id=chat_id,
|
||||
client_message_id=client_message_id,
|
||||
action="promote_chat_to_task",
|
||||
target=str(outcome.get("task_id") or task_id),
|
||||
status=outcome_status,
|
||||
)
|
||||
return
|
||||
reserved_project = _reserved_project_for_chat(ctx, chat_id)
|
||||
project_id = (
|
||||
|
|
@ -1903,6 +1906,15 @@ def _shutdown_task_cleanup_args(restart_requested: bool) -> tuple[str, str]:
|
|||
return "cancelled", reason
|
||||
|
||||
|
||||
def _shutdown_supervisor_event_bus() -> None:
|
||||
try:
|
||||
from supervisor.workers import shutdown_event_q
|
||||
|
||||
shutdown_event_q()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def _execute_panic_stop(consciousness, kill_workers_fn) -> None:
|
||||
_execute_panic_stop_impl(
|
||||
consciousness,
|
||||
|
|
@ -2276,6 +2288,7 @@ async def lifespan(app):
|
|||
get_bridge().shutdown()
|
||||
except Exception:
|
||||
pass
|
||||
_shutdown_supervisor_event_bus()
|
||||
|
||||
|
||||
app = NetworkAuthGate(Starlette(routes=routes, lifespan=lifespan))
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ import logging
|
|||
import os
|
||||
import pathlib
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from typing import Any, Dict, Optional
|
||||
|
|
@ -47,25 +48,78 @@ def _emit_routing_receipt(
|
|||
action: str,
|
||||
target: str = "",
|
||||
status: str,
|
||||
reason: str = "",
|
||||
detail: str = "",
|
||||
options: Optional[list] = None,
|
||||
) -> None:
|
||||
"""Latest-only UI annotation plus typed non-bubble transport receipt."""
|
||||
publish: bool = True,
|
||||
) -> Dict[str, Any]:
|
||||
"""Persist and publish one token-bound routing annotation receipt."""
|
||||
client_message_id = str(evt.get("client_message_id") or "").strip()
|
||||
if not client_message_id:
|
||||
return
|
||||
try:
|
||||
from ouroboros.project_dialogue import append_chat_annotation
|
||||
routing_token = str(evt.get("routing_token") or "").strip()
|
||||
annotation_status = "not_applicable"
|
||||
if client_message_id:
|
||||
try:
|
||||
from ouroboros.project_dialogue import append_chat_annotation
|
||||
|
||||
append_chat_annotation(
|
||||
ctx.DRIVE_ROOT,
|
||||
client_message_id,
|
||||
annotation_status = (
|
||||
"persisted"
|
||||
if append_chat_annotation(
|
||||
ctx.DRIVE_ROOT,
|
||||
client_message_id,
|
||||
action=action,
|
||||
target=target,
|
||||
status=status,
|
||||
routing_token=routing_token,
|
||||
reason=reason,
|
||||
detail=detail,
|
||||
options=options,
|
||||
)
|
||||
else "failed"
|
||||
)
|
||||
except Exception:
|
||||
annotation_status = "failed"
|
||||
log.debug("Routing annotation append failed", exc_info=True)
|
||||
|
||||
effective_status = str(status or "needs_manual_target")
|
||||
effective_reason = str(reason or "")
|
||||
if annotation_status == "failed" and effective_status in {"scheduled", "delivered"}:
|
||||
effective_status = "unconfirmed"
|
||||
effective_reason = "routing_annotation_persist_failed"
|
||||
|
||||
receipt: Dict[str, Any] = {
|
||||
"persisted": annotation_status in {"persisted", "not_applicable"},
|
||||
"status": effective_status,
|
||||
"reason": effective_reason,
|
||||
"detail": str(detail or ""),
|
||||
"annotation_status": annotation_status,
|
||||
"routing_token": routing_token,
|
||||
}
|
||||
if not receipt["persisted"]:
|
||||
return receipt
|
||||
if publish:
|
||||
_publish_routing_ack(
|
||||
ctx,
|
||||
evt,
|
||||
action=action,
|
||||
target=target,
|
||||
status=status,
|
||||
status=effective_status,
|
||||
options=options,
|
||||
)
|
||||
except Exception:
|
||||
log.debug("Routing annotation append failed", exc_info=True)
|
||||
return receipt
|
||||
|
||||
|
||||
def _publish_routing_ack(
|
||||
ctx: Any,
|
||||
evt: Dict[str, Any],
|
||||
*,
|
||||
action: str,
|
||||
target: str,
|
||||
status: str,
|
||||
options: Optional[list] = None,
|
||||
) -> None:
|
||||
"""Publish a live non-bubble acknowledgement after durable authority exists."""
|
||||
try:
|
||||
client_message_id = str(evt.get("client_message_id") or "").strip()
|
||||
try:
|
||||
chat_id = int(evt.get("chat_id") or 0)
|
||||
except (TypeError, ValueError):
|
||||
|
|
@ -1984,57 +2038,383 @@ def _handle_project_digest(evt: Dict[str, Any], ctx: Any) -> None:
|
|||
log.debug("project_digest consciousness injection failed", exc_info=True)
|
||||
|
||||
|
||||
def _handle_promote_chat_to_task(evt: Dict[str, Any], ctx: Any) -> None:
|
||||
def _rollback_promoted_pending(
|
||||
ctx: Any, task_id: str, admission_token: str, *, reason: str,
|
||||
) -> bool:
|
||||
"""Remove an unconfirmed promote before the supervisor can assign it."""
|
||||
from supervisor import queue as supervisor_queue
|
||||
|
||||
removed = False
|
||||
with supervisor_queue._queue_lock:
|
||||
pending = getattr(ctx, "PENDING", supervisor_queue.PENDING)
|
||||
survivors = [
|
||||
task for task in pending
|
||||
if not (
|
||||
str(task.get("id") or "") == task_id
|
||||
and str(
|
||||
task.get("_admission_owner_token")
|
||||
or task.get("promotion_admission_token")
|
||||
or ""
|
||||
) == admission_token
|
||||
)
|
||||
]
|
||||
removed = len(survivors) != len(pending)
|
||||
if removed:
|
||||
pending[:] = survivors
|
||||
if removed:
|
||||
persist = getattr(ctx, "persist_queue_snapshot", None)
|
||||
if callable(persist):
|
||||
try:
|
||||
persist(reason=reason)
|
||||
except Exception:
|
||||
log.warning("Failed to persist promote rollback for %s", task_id, exc_info=True)
|
||||
return removed
|
||||
|
||||
|
||||
def _persist_promote_rejection(
|
||||
ctx: Any,
|
||||
evt: Dict[str, Any],
|
||||
outcome: Dict[str, Any],
|
||||
*,
|
||||
status: str = "rejected",
|
||||
) -> None:
|
||||
task_id = str(outcome.get("task_id") or evt.get("task_id") or "")
|
||||
reason = str(outcome.get("reason") or "admission_rejected")
|
||||
write_task_result(
|
||||
ctx.DRIVE_ROOT,
|
||||
task_id,
|
||||
STATUS_FAILED,
|
||||
reason_code=reason,
|
||||
project_id=str(evt.get("project_id") or ""),
|
||||
description=str(evt.get("objective") or ""),
|
||||
expected_output=str(evt.get("expected_output") or ""),
|
||||
promotion_admission={
|
||||
"status": status,
|
||||
"routing_token": str(evt.get("routing_token") or ""),
|
||||
"reason": reason,
|
||||
"detail": str(outcome.get("detail") or ""),
|
||||
"worker_pool_disabled_reason": str(
|
||||
outcome.get("worker_pool_disabled_reason") or ""
|
||||
),
|
||||
"confirmed_at": utc_now_iso(),
|
||||
},
|
||||
result=(
|
||||
f"Promotion was not scheduled: {reason}. "
|
||||
f"{str(outcome.get('detail') or '')}"
|
||||
).strip(),
|
||||
)
|
||||
|
||||
|
||||
def _prepare_promote_source_off_loop(evt: Dict[str, Any], ctx: Any) -> None:
|
||||
"""Resolve a potentially 900s clone away from the supervisor drain loop."""
|
||||
continuation = dict(evt)
|
||||
continuation["_source_prepared"] = True
|
||||
try:
|
||||
from ouroboros.promotion_source import resolve_promote_source
|
||||
|
||||
folder, note, error, project_id = resolve_promote_source(
|
||||
ctx,
|
||||
str(evt.get("source") or ""),
|
||||
str(evt.get("project_id") or ""),
|
||||
)
|
||||
continuation["project_id"] = project_id
|
||||
continuation["_source_note"] = note
|
||||
continuation["_source_error"] = error
|
||||
if folder and not str(continuation.get("workspace_root") or "").strip():
|
||||
continuation["workspace_root"] = folder
|
||||
except Exception as exc:
|
||||
continuation["_source_error"] = f"{type(exc).__name__}: {exc}"
|
||||
try:
|
||||
from supervisor.workers import get_event_q
|
||||
|
||||
get_event_q().put(continuation)
|
||||
except Exception as exc:
|
||||
log.exception("Failed to publish promote source continuation")
|
||||
from supervisor import queue as supervisor_queue
|
||||
|
||||
task_id = str(evt.get("task_id") or "")
|
||||
routing_token = str(evt.get("routing_token") or "")
|
||||
supervisor_queue.release_task_admission(task_id, routing_token)
|
||||
failed = {
|
||||
"status": "unconfirmed",
|
||||
"reason": "source_continuation_publish_failed",
|
||||
"detail": f"{type(exc).__name__}: {exc}",
|
||||
"task_id": task_id,
|
||||
}
|
||||
try:
|
||||
_persist_promote_rejection(ctx, evt, failed, status="unconfirmed")
|
||||
_emit_routing_receipt(
|
||||
ctx,
|
||||
evt,
|
||||
action=(
|
||||
"route_to_project"
|
||||
if bool(evt.get("routed_from_main"))
|
||||
else "promote_chat_to_task"
|
||||
),
|
||||
target=task_id,
|
||||
status="unconfirmed",
|
||||
reason=failed["reason"],
|
||||
detail=failed["detail"],
|
||||
)
|
||||
except Exception:
|
||||
log.exception("Failed to persist promote source continuation failure")
|
||||
|
||||
|
||||
def _handle_promote_chat_to_task(evt: Dict[str, Any], ctx: Any) -> Dict[str, Any]:
|
||||
"""Spawn a first-class pooled owner task from a conversation-lane promote.
|
||||
|
||||
Unlike ``schedule_subagent`` the child is NOT a subagent: it is a normal
|
||||
owner task (live card, canonical drive, project lease participation). The
|
||||
conversation lane that emitted the event stays free.
|
||||
"""
|
||||
from supervisor.workers import promote_chat_to_task
|
||||
from supervisor.workers import (
|
||||
_broadcast_task_named,
|
||||
promote_chat_to_task,
|
||||
worker_pool_admission_state,
|
||||
)
|
||||
receipt_action = (
|
||||
"route_to_project" if bool(evt.get("routed_from_main"))
|
||||
else "promote_chat_to_task"
|
||||
)
|
||||
|
||||
task_id = str(evt.get("task_id") or "")
|
||||
routing_token = str(evt.get("routing_token") or "")
|
||||
try:
|
||||
outcome = promote_chat_to_task(evt, ctx)
|
||||
outcome = outcome if isinstance(outcome, dict) else {"status": "scheduled"}
|
||||
_emit_routing_receipt(
|
||||
ctx,
|
||||
evt,
|
||||
action=receipt_action,
|
||||
target=str(outcome.get("task_id") or evt.get("task_id") or ""),
|
||||
status=str(outcome.get("status") or "needs_manual_target"),
|
||||
from supervisor import queue as supervisor_queue
|
||||
|
||||
reservation = supervisor_queue.reserve_task_admission(
|
||||
task_id,
|
||||
routing_token,
|
||||
require_worker_pool=True,
|
||||
drive_root=ctx.DRIVE_ROOT,
|
||||
worker_pool=getattr(ctx, "WORKERS", None),
|
||||
)
|
||||
if str(outcome.get("status") or "") != "scheduled":
|
||||
reservation_status = str(reservation.get("status") or "")
|
||||
if reservation_status == "already_reserved" and evt.get("_admission_reserved"):
|
||||
reservation_status = "reserved"
|
||||
if reservation_status != "reserved":
|
||||
if reservation_status == "existing_same_token":
|
||||
admission = reservation.get("promotion_admission")
|
||||
return {
|
||||
"status": str((admission or {}).get("status") or "unconfirmed"),
|
||||
"task_id": task_id,
|
||||
"reason": str((admission or {}).get("reason") or ""),
|
||||
}
|
||||
if reservation_status == "already_reserved":
|
||||
return {"status": "preparing", "task_id": task_id}
|
||||
blocked = {
|
||||
"status": "needs_manual_target",
|
||||
"reason": str(reservation.get("reason") or "admission_reservation_failed"),
|
||||
"worker_pool_disabled_reason": str(
|
||||
reservation.get("worker_pool_disabled_reason") or ""
|
||||
),
|
||||
"task_id": task_id,
|
||||
"reservation_owned": False,
|
||||
}
|
||||
if blocked["reason"] != "duplicate_task_id":
|
||||
_persist_promote_rejection(ctx, evt, blocked)
|
||||
_emit_routing_receipt(
|
||||
ctx,
|
||||
evt,
|
||||
action=receipt_action,
|
||||
target=task_id,
|
||||
status="needs_manual_target",
|
||||
reason=blocked["reason"],
|
||||
)
|
||||
ctx.append_jsonl(
|
||||
ctx.DRIVE_ROOT / "logs" / "supervisor.jsonl",
|
||||
{
|
||||
"ts": utc_now_iso(),
|
||||
"type": "promote_chat_to_task_rejected",
|
||||
"task_id": str(outcome.get("task_id") or evt.get("task_id") or ""),
|
||||
"reason": str(outcome.get("reason") or "admission_rejected"),
|
||||
"project_lifecycle": str(outcome.get("project_lifecycle") or ""),
|
||||
"task_id": task_id,
|
||||
"reason": blocked["reason"],
|
||||
"worker_pool_disabled_reason": blocked[
|
||||
"worker_pool_disabled_reason"
|
||||
],
|
||||
},
|
||||
)
|
||||
except Exception:
|
||||
return blocked
|
||||
evt = {**evt, "_admission_reserved": True}
|
||||
if str(evt.get("source") or "").strip() and not evt.get("_source_prepared"):
|
||||
threading.Thread(
|
||||
target=_prepare_promote_source_off_loop,
|
||||
args=(dict(evt), ctx),
|
||||
daemon=True,
|
||||
name=f"promote-source-{task_id[:12]}",
|
||||
).start()
|
||||
return {"status": "preparing", "task_id": task_id}
|
||||
source_error = str(evt.get("_source_error") or "")
|
||||
if source_error:
|
||||
outcome = {
|
||||
"status": "needs_manual_target",
|
||||
"reason": "project_source_error",
|
||||
"detail": source_error,
|
||||
"task_id": task_id,
|
||||
"reservation_owned": True,
|
||||
}
|
||||
else:
|
||||
outcome = None
|
||||
pool_state = worker_pool_admission_state(ctx)
|
||||
if outcome is None and not pool_state["available"]:
|
||||
outcome = {
|
||||
"status": "needs_manual_target",
|
||||
"reason": "worker_pool_unavailable",
|
||||
"worker_pool_disabled_reason": str(pool_state.get("disabled_reason") or ""),
|
||||
"task_id": task_id,
|
||||
}
|
||||
elif outcome is None:
|
||||
outcome = promote_chat_to_task(evt, ctx)
|
||||
outcome = outcome if isinstance(outcome, dict) else {"status": "scheduled"}
|
||||
if str(outcome.get("status") or "") == "scheduled":
|
||||
title = str(evt.get("title") or "").strip()[:80]
|
||||
receipt = _emit_routing_receipt(
|
||||
ctx,
|
||||
evt,
|
||||
action=receipt_action,
|
||||
target=str(outcome.get("task_id") or task_id),
|
||||
status="scheduled",
|
||||
detail=str(outcome.get("source_note") or ""),
|
||||
publish=False,
|
||||
)
|
||||
admission_status = (
|
||||
"scheduled"
|
||||
if receipt.get("persisted") and str(receipt.get("status") or "") == "scheduled"
|
||||
else "unconfirmed"
|
||||
)
|
||||
stored = write_task_result(
|
||||
ctx.DRIVE_ROOT,
|
||||
str(outcome.get("task_id") or task_id),
|
||||
STATUS_SCHEDULED,
|
||||
project_id=str(outcome.get("project_id") or evt.get("project_id") or ""),
|
||||
description=str(evt.get("objective") or ""),
|
||||
expected_output=str(evt.get("expected_output") or ""),
|
||||
suggested_name=title,
|
||||
promotion_admission={
|
||||
"status": admission_status,
|
||||
"routing_token": str(evt.get("routing_token") or ""),
|
||||
"reason": str(receipt.get("reason") or ""),
|
||||
"confirmed_at": utc_now_iso(),
|
||||
"queue_snapshot_persisted": True,
|
||||
"routing_receipt_required": bool(str(evt.get("client_message_id") or "")),
|
||||
"routing_receipt_status": str(receipt.get("annotation_status") or ""),
|
||||
"source_note": str(outcome.get("source_note") or ""),
|
||||
},
|
||||
result=(
|
||||
"Task accepted and durably scheduled."
|
||||
if admission_status == "scheduled"
|
||||
else "Task is scheduled, but its owner-facing routing receipt was not confirmed."
|
||||
),
|
||||
)
|
||||
admission = stored.get("promotion_admission") if isinstance(stored, dict) else {}
|
||||
if (
|
||||
str((admission or {}).get("status") or "") != admission_status
|
||||
or str((admission or {}).get("routing_token") or "")
|
||||
!= str(evt.get("routing_token") or "")
|
||||
):
|
||||
raise RuntimeError("scheduled promotion result was not persisted")
|
||||
supervisor_queue.release_task_admission(task_id, routing_token)
|
||||
if admission_status != "scheduled":
|
||||
return {
|
||||
**outcome,
|
||||
"status": "unconfirmed",
|
||||
"reason": str(receipt.get("reason") or "routing_receipt_persist_failed"),
|
||||
}
|
||||
_publish_routing_ack(
|
||||
ctx,
|
||||
evt,
|
||||
action=receipt_action,
|
||||
target=str(outcome.get("task_id") or task_id),
|
||||
status="scheduled",
|
||||
)
|
||||
if title:
|
||||
_broadcast_task_named(
|
||||
{"type": "task_named", "task_id": str(outcome.get("task_id") or task_id),
|
||||
"suggested_name": title}
|
||||
)
|
||||
try:
|
||||
ctx.append_jsonl(
|
||||
ctx.DRIVE_ROOT / "logs" / "supervisor.jsonl",
|
||||
{
|
||||
"ts": utc_now_iso(),
|
||||
"type": "promote_chat_to_task_admitted",
|
||||
"task_id": str(outcome.get("task_id") or task_id),
|
||||
},
|
||||
)
|
||||
except Exception:
|
||||
log.warning("Failed to record admitted promote %s", task_id, exc_info=True)
|
||||
return outcome
|
||||
|
||||
_rollback_promoted_pending(
|
||||
ctx,
|
||||
str(outcome.get("task_id") or task_id),
|
||||
routing_token,
|
||||
reason="promote_chat_to_task_rejected",
|
||||
)
|
||||
supervisor_queue.release_task_admission(task_id, routing_token)
|
||||
_persist_promote_rejection(ctx, evt, outcome)
|
||||
_emit_routing_receipt(
|
||||
ctx,
|
||||
evt,
|
||||
action=receipt_action,
|
||||
target=str(outcome.get("task_id") or task_id),
|
||||
status="needs_manual_target",
|
||||
reason=str(outcome.get("reason") or "admission_rejected"),
|
||||
detail=str(outcome.get("detail") or ""),
|
||||
)
|
||||
ctx.append_jsonl(
|
||||
ctx.DRIVE_ROOT / "logs" / "supervisor.jsonl",
|
||||
{
|
||||
"ts": utc_now_iso(),
|
||||
"type": "promote_chat_to_task_rejected",
|
||||
"task_id": str(outcome.get("task_id") or evt.get("task_id") or ""),
|
||||
"reason": str(outcome.get("reason") or "admission_rejected"),
|
||||
"project_lifecycle": str(outcome.get("project_lifecycle") or ""),
|
||||
"worker_pool_disabled_reason": str(
|
||||
outcome.get("worker_pool_disabled_reason") or ""
|
||||
),
|
||||
},
|
||||
)
|
||||
return outcome
|
||||
except Exception as exc:
|
||||
log.warning("promote_chat_to_task event failed", exc_info=True)
|
||||
_rollback_promoted_pending(
|
||||
ctx, task_id, routing_token, reason="promote_chat_to_task_failed",
|
||||
)
|
||||
try:
|
||||
from supervisor import queue as supervisor_queue
|
||||
|
||||
supervisor_queue.release_task_admission(task_id, routing_token)
|
||||
except Exception:
|
||||
pass
|
||||
failed_outcome = {
|
||||
"status": "unconfirmed",
|
||||
"reason": "promotion_persistence_failed",
|
||||
"task_id": task_id,
|
||||
"detail": f"{type(exc).__name__}: {exc}",
|
||||
}
|
||||
try:
|
||||
_persist_promote_rejection(ctx, evt, failed_outcome, status="unconfirmed")
|
||||
except Exception:
|
||||
log.warning("Failed to persist promote failure for %s", task_id, exc_info=True)
|
||||
_emit_routing_receipt(
|
||||
ctx,
|
||||
evt,
|
||||
action=receipt_action,
|
||||
target=str(evt.get("task_id") or ""),
|
||||
status="needs_manual_target",
|
||||
status="unconfirmed",
|
||||
reason="promotion_persistence_failed",
|
||||
detail=f"{type(exc).__name__}: {exc}",
|
||||
)
|
||||
ctx.append_jsonl(
|
||||
ctx.DRIVE_ROOT / "logs" / "supervisor.jsonl",
|
||||
{
|
||||
"ts": utc_now_iso(),
|
||||
"type": "promote_chat_to_task_failed",
|
||||
"event_repr": repr(evt)[:500],
|
||||
"task_id": task_id,
|
||||
"error": f"{type(exc).__name__}: {exc}",
|
||||
},
|
||||
)
|
||||
return failed_outcome
|
||||
|
||||
|
||||
def _handle_ensure_project_scope(evt: Dict[str, Any], ctx: Any) -> None:
|
||||
|
|
@ -2060,6 +2440,7 @@ def _handle_routing_manual_target(evt: Dict[str, Any], ctx: Any) -> None:
|
|||
action="route_decision",
|
||||
target=str(evt.get("requested_target") or evt.get("reason") or "")[:200],
|
||||
status="needs_manual_target",
|
||||
reason=str(evt.get("reason") or "target_unspecified"),
|
||||
options=options,
|
||||
)
|
||||
|
||||
|
|
@ -2143,11 +2524,11 @@ def _handle_steer_task(evt: Dict[str, Any], ctx: Any) -> None:
|
|||
# Fail visibly: the chosen task is no longer a steerable running task in
|
||||
# this chat. Tell the owner so the agent/owner can answer or spawn instead.
|
||||
client_message_id = str(evt.get("client_message_id") or "").strip()
|
||||
if client_message_id:
|
||||
_emit_routing_receipt(
|
||||
ctx, evt, action="steer_task", target=target, status="needs_manual_target",
|
||||
)
|
||||
elif chat_id:
|
||||
_emit_routing_receipt(
|
||||
ctx, evt, action="steer_task", target=target, status="needs_manual_target",
|
||||
reason="target_not_steerable",
|
||||
)
|
||||
if not client_message_id and chat_id:
|
||||
try:
|
||||
ctx.send_with_budget(
|
||||
chat_id,
|
||||
|
|
@ -2180,6 +2561,7 @@ def _handle_steer_task(evt: Dict[str, Any], ctx: Any) -> None:
|
|||
):
|
||||
_emit_routing_receipt(
|
||||
ctx, evt, action="steer_task", target=target, status="needs_manual_target",
|
||||
reason="target_closed",
|
||||
)
|
||||
return
|
||||
drive = pathlib.Path(ctx.DRIVE_ROOT) if direct_active else _task_drive_for_task(task, target)
|
||||
|
|
@ -2203,6 +2585,7 @@ def _handle_steer_task(evt: Dict[str, Any], ctx: Any) -> None:
|
|||
if live_meta is None and not still_pending:
|
||||
_emit_routing_receipt(
|
||||
ctx, evt, action="steer_task", target=target, status="needs_manual_target",
|
||||
reason="target_finished",
|
||||
)
|
||||
return
|
||||
fence_root = str(task.get("root_task_id") or target)
|
||||
|
|
@ -2210,15 +2593,17 @@ def _handle_steer_task(evt: Dict[str, Any], ctx: Any) -> None:
|
|||
if isinstance(active_fence, dict) and str(active_fence.get("status") or "") == "sealed":
|
||||
_emit_routing_receipt(
|
||||
ctx, evt, action="steer_task", target=target, status="needs_manual_target",
|
||||
reason="acceptance_fence_sealed",
|
||||
)
|
||||
return
|
||||
write_owner_message(
|
||||
if not write_owner_message(
|
||||
drive,
|
||||
f"{message}{attachment_note}",
|
||||
target,
|
||||
msg_id=msg_id,
|
||||
kind=KIND_OWNER_TEXT,
|
||||
)
|
||||
):
|
||||
raise OSError("owner mailbox append was not durable")
|
||||
if direct_active:
|
||||
direct_agent._owner_message_generation = int(
|
||||
getattr(direct_agent, "_owner_message_generation", 0) or 0
|
||||
|
|
@ -2234,6 +2619,7 @@ def _handle_steer_task(evt: Dict[str, Any], ctx: Any) -> None:
|
|||
log.warning("steer_task delivery failed for task %s", target, exc_info=True)
|
||||
_emit_routing_receipt(
|
||||
ctx, evt, action="steer_task", target=target, status="needs_manual_target",
|
||||
reason="mailbox_write_failed",
|
||||
)
|
||||
finally:
|
||||
if queue_lock_held:
|
||||
|
|
|
|||
|
|
@ -135,12 +135,18 @@ PENDING: List[Dict[str, Any]] = []
|
|||
RUNNING: Dict[str, Dict[str, Any]] = {}
|
||||
QUEUE_SEQ_COUNTER_REF: Dict[str, int] = {"value": 0}
|
||||
ACCEPTANCE_FENCES: Dict[str, Dict[str, Any]] = {}
|
||||
ADMISSION_RESERVATIONS: Dict[str, str] = {}
|
||||
|
||||
# Guards PENDING/RUNNING mutations across main loop, direct chat, watchdog.
|
||||
_queue_lock = threading.RLock()
|
||||
_last_skill_schedule_sync: float = 0.0
|
||||
_SKILL_SCHEDULE_SYNC_INTERVAL_SEC: float = 60.0
|
||||
|
||||
from supervisor.task_admission import ( # noqa: E402,F401 - public queue API
|
||||
release_task_admission,
|
||||
reserve_task_admission,
|
||||
)
|
||||
|
||||
# Variant A off-loop worker reaper lives in supervisor/task_reaper.py (extracted for
|
||||
# module size). Re-export the thin names the enforce path and tests use; monkeypatching
|
||||
# these queue-module names still works because the enforce path references them here.
|
||||
|
|
@ -158,6 +164,7 @@ def init_queue_refs(pending: List[Dict[str, Any]], running: Dict[str, Dict[str,
|
|||
PENDING = pending
|
||||
RUNNING = running
|
||||
QUEUE_SEQ_COUNTER_REF = seq_counter_ref
|
||||
ADMISSION_RESERVATIONS.clear()
|
||||
|
||||
|
||||
def _task_priority(task_type: str) -> int:
|
||||
|
|
@ -198,6 +205,59 @@ def enqueue_task(
|
|||
t = dict(task)
|
||||
attach_task_contract(t)
|
||||
with _queue_lock:
|
||||
require_unique_id = bool(t.pop("_require_unique_task_id", False))
|
||||
require_worker_pool = bool(t.pop("_require_worker_pool", False))
|
||||
admission_token = str(t.pop("_admission_token", "") or "")
|
||||
task_id = str(t.get("id") or "").strip()
|
||||
reserved_token = str(ADMISSION_RESERVATIONS.get(task_id) or "")
|
||||
if reserved_token and admission_token != reserved_token:
|
||||
# A reservation owns this id until its request either enqueues or
|
||||
# releases it. Tokenless internal callers and competing ingress
|
||||
# requests must not be able to consume/collide with that id.
|
||||
t["_admission_blocked"] = "admission_reservation_owned"
|
||||
return t
|
||||
if require_worker_pool:
|
||||
try:
|
||||
from supervisor import workers
|
||||
|
||||
disabled_reason = str(workers._WORKER_POOL_DISABLED_REASON or "")
|
||||
worker_count = len(workers.WORKERS)
|
||||
except Exception:
|
||||
disabled_reason = "state_unavailable"
|
||||
worker_count = 0
|
||||
if disabled_reason or worker_count <= 0:
|
||||
if ADMISSION_RESERVATIONS.get(task_id) == admission_token:
|
||||
ADMISSION_RESERVATIONS.pop(task_id, None)
|
||||
t["_admission_blocked"] = "worker_pool_unavailable"
|
||||
t["_worker_pool_disabled_reason"] = disabled_reason or "no_workers"
|
||||
return t
|
||||
if admission_token and reserved_token != admission_token:
|
||||
t["_admission_blocked"] = "admission_reservation_lost"
|
||||
return t
|
||||
if require_unique_id and task_id:
|
||||
live_duplicate = task_id in RUNNING or any(
|
||||
isinstance(row, dict) and str(row.get("id") or "") == task_id
|
||||
for row in PENDING
|
||||
)
|
||||
if live_duplicate:
|
||||
if ADMISSION_RESERVATIONS.get(task_id) == admission_token:
|
||||
ADMISSION_RESERVATIONS.pop(task_id, None)
|
||||
t["_admission_blocked"] = "duplicate_task_id"
|
||||
return t
|
||||
try:
|
||||
from ouroboros.task_results import load_task_result
|
||||
|
||||
if load_task_result(DRIVE_ROOT, task_id):
|
||||
if ADMISSION_RESERVATIONS.get(task_id) == admission_token:
|
||||
ADMISSION_RESERVATIONS.pop(task_id, None)
|
||||
t["_admission_blocked"] = "duplicate_task_id"
|
||||
return t
|
||||
except Exception:
|
||||
log.warning("Fresh task-id lookup failed for %s", task_id, exc_info=True)
|
||||
t["_admission_blocked"] = "task_id_lookup_failed"
|
||||
if ADMISSION_RESERVATIONS.get(task_id) == admission_token:
|
||||
ADMISSION_RESERVATIONS.pop(task_id, None)
|
||||
return t
|
||||
project_id = str(t.get("project_id") or "").strip()
|
||||
if project_id:
|
||||
try:
|
||||
|
|
@ -209,20 +269,28 @@ def enqueue_task(
|
|||
t["_admission_blocked"] = "project_routing_fence"
|
||||
t["_project_lifecycle"] = lifecycle
|
||||
t["_project_id"] = project_id
|
||||
if ADMISSION_RESERVATIONS.get(task_id) == admission_token:
|
||||
ADMISSION_RESERVATIONS.pop(task_id, None)
|
||||
return t
|
||||
except Exception:
|
||||
log.warning("Project admission check failed for %s", project_id, exc_info=True)
|
||||
t["_admission_blocked"] = "project_routing_fence_lookup_failed"
|
||||
t["_project_id"] = project_id
|
||||
if ADMISSION_RESERVATIONS.get(task_id) == admission_token:
|
||||
ADMISSION_RESERVATIONS.pop(task_id, None)
|
||||
return t
|
||||
root_id = str(t.get("root_task_id") or "").strip()
|
||||
if root_id and not restoring_snapshot and apply_budget_root_admission_fence(t, root_id):
|
||||
if ADMISSION_RESERVATIONS.get(task_id) == admission_token:
|
||||
ADMISSION_RESERVATIONS.pop(task_id, None)
|
||||
return t
|
||||
fence = ACCEPTANCE_FENCES.get(root_id) if root_id else None
|
||||
if isinstance(fence, dict) and str(fence.get("status") or "") in {"active", "sealed"}:
|
||||
t["_admission_blocked"] = "task_acceptance_fence"
|
||||
t["_acceptance_fence_token"] = str(fence.get("token") or "")
|
||||
t["_acceptance_fence_status"] = str(fence.get("status") or "active")
|
||||
if ADMISSION_RESERVATIONS.get(task_id) == admission_token:
|
||||
ADMISSION_RESERVATIONS.pop(task_id, None)
|
||||
return t
|
||||
QUEUE_SEQ_COUNTER_REF["value"] += 1
|
||||
seq = QUEUE_SEQ_COUNTER_REF["value"]
|
||||
|
|
@ -231,8 +299,12 @@ def enqueue_task(
|
|||
t.setdefault("_attempt", int(_att) if _att is not None else 1)
|
||||
t["_queue_seq"] = -seq if front else seq
|
||||
t["queued_at"] = utc_now_iso()
|
||||
if admission_token:
|
||||
t["_admission_owner_token"] = admission_token
|
||||
PENDING.append(t)
|
||||
sort_pending()
|
||||
if ADMISSION_RESERVATIONS.get(task_id) == admission_token:
|
||||
ADMISSION_RESERVATIONS.pop(task_id, None)
|
||||
return t
|
||||
|
||||
|
||||
|
|
@ -596,7 +668,7 @@ def _kept_service_pids() -> "set[int]":
|
|||
return set()
|
||||
|
||||
|
||||
def persist_queue_snapshot(reason: str = "") -> None:
|
||||
def persist_queue_snapshot(reason: str = "") -> bool:
|
||||
"""Persist queue snapshot for restart/recovery diagnostics.
|
||||
|
||||
Snapshots PENDING/RUNNING under the queue lock: iterating the live dicts
|
||||
|
|
@ -620,6 +692,9 @@ def persist_queue_snapshot(reason: str = "") -> None:
|
|||
|
||||
_ws = list(_workers_mod.WORKERS.values())
|
||||
worker_total = len(_ws)
|
||||
worker_pool_disabled_reason = str(
|
||||
getattr(_workers_mod, "_WORKER_POOL_DISABLED_REASON", "") or ""
|
||||
)
|
||||
reaping_count = sum(1 for _w in _ws if getattr(_w, "reaping", False))
|
||||
assignable_idle_workers = sum(
|
||||
1 for _w in _ws
|
||||
|
|
@ -627,6 +702,7 @@ def persist_queue_snapshot(reason: str = "") -> None:
|
|||
)
|
||||
except Exception:
|
||||
worker_total = 0
|
||||
worker_pool_disabled_reason = "unknown"
|
||||
reaping_count = 0
|
||||
assignable_idle_workers = 0
|
||||
pending_rows = []
|
||||
|
|
@ -687,6 +763,7 @@ def persist_queue_snapshot(reason: str = "") -> None:
|
|||
"pending_count": len(pending_items), "running_count": len(running_items),
|
||||
"reaping_count": reaping_count,
|
||||
"worker_total": worker_total,
|
||||
"worker_pool_disabled_reason": worker_pool_disabled_reason,
|
||||
"assignable_idle_workers": assignable_idle_workers,
|
||||
"acceptance_fences": acceptance_fences,
|
||||
"budget_root_fences": budget_root_fences,
|
||||
|
|
@ -694,9 +771,10 @@ def persist_queue_snapshot(reason: str = "") -> None:
|
|||
}
|
||||
try:
|
||||
atomic_write_text(QUEUE_SNAPSHOT_PATH, json.dumps(payload, ensure_ascii=False, indent=2))
|
||||
return True
|
||||
except Exception:
|
||||
log.warning("Failed to persist queue snapshot (reason=%s)", reason, exc_info=True)
|
||||
pass
|
||||
return False
|
||||
|
||||
|
||||
def parse_iso_to_ts(iso_ts: str) -> Optional[float]:
|
||||
|
|
|
|||
88
supervisor/task_admission.py
Normal file
88
supervisor/task_admission.py
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
"""Token-owned user-ingress reservations for the managed task queue."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pathlib
|
||||
from typing import Any, Dict
|
||||
|
||||
|
||||
def reserve_task_admission(
|
||||
task_id: str,
|
||||
admission_token: str,
|
||||
*,
|
||||
require_worker_pool: bool = False,
|
||||
drive_root: Any = None,
|
||||
worker_pool: Any = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Atomically reserve one fresh user-ingress id before side effects."""
|
||||
from supervisor import queue
|
||||
|
||||
tid = str(task_id or "").strip()
|
||||
token = str(admission_token or "").strip()
|
||||
if not tid or not token:
|
||||
return {"status": "blocked", "reason": "invalid_admission_reservation"}
|
||||
with queue._queue_lock:
|
||||
reserved = queue.ADMISSION_RESERVATIONS.get(tid)
|
||||
if reserved:
|
||||
if reserved == token:
|
||||
return {"status": "already_reserved", "reason": ""}
|
||||
return {"status": "blocked", "reason": "duplicate_task_id"}
|
||||
if tid in queue.RUNNING or any(
|
||||
isinstance(row, dict) and str(row.get("id") or "") == tid
|
||||
for row in queue.PENDING
|
||||
):
|
||||
return {"status": "blocked", "reason": "duplicate_task_id"}
|
||||
try:
|
||||
from ouroboros.task_results import load_task_result
|
||||
|
||||
existing = load_task_result(
|
||||
pathlib.Path(drive_root or queue.DRIVE_ROOT), tid
|
||||
) or {}
|
||||
except Exception:
|
||||
return {"status": "blocked", "reason": "task_id_lookup_failed"}
|
||||
if existing:
|
||||
admission = existing.get("promotion_admission")
|
||||
if (
|
||||
isinstance(admission, dict)
|
||||
and str(admission.get("routing_token") or "") == token
|
||||
):
|
||||
return {
|
||||
"status": "existing_same_token",
|
||||
"reason": "",
|
||||
"task_status": str(existing.get("status") or ""),
|
||||
"promotion_admission": dict(admission),
|
||||
}
|
||||
return {"status": "blocked", "reason": "duplicate_task_id"}
|
||||
if require_worker_pool:
|
||||
try:
|
||||
from supervisor import workers
|
||||
|
||||
disabled_reason = str(workers._WORKER_POOL_DISABLED_REASON or "")
|
||||
pool = workers.WORKERS if worker_pool is None else worker_pool
|
||||
worker_count = len(pool)
|
||||
except Exception:
|
||||
return {"status": "blocked", "reason": "worker_pool_state_unavailable"}
|
||||
if disabled_reason or worker_count <= 0:
|
||||
return {
|
||||
"status": "blocked",
|
||||
"reason": "worker_pool_unavailable",
|
||||
"worker_pool_disabled_reason": disabled_reason or "no_workers",
|
||||
}
|
||||
queue.ADMISSION_RESERVATIONS[tid] = token
|
||||
return {"status": "reserved", "reason": ""}
|
||||
|
||||
|
||||
def release_task_admission(task_id: str, admission_token: str) -> bool:
|
||||
"""Release only the reservation owned by the supplied token."""
|
||||
from supervisor import queue
|
||||
|
||||
tid = str(task_id or "").strip()
|
||||
token = str(admission_token or "").strip()
|
||||
with queue._queue_lock:
|
||||
if queue.ADMISSION_RESERVATIONS.get(tid) != token:
|
||||
return False
|
||||
queue.ADMISSION_RESERVATIONS.pop(tid, None)
|
||||
return True
|
||||
|
||||
|
||||
__all__ = ["release_task_admission", "reserve_task_admission"]
|
||||
|
|
@ -768,14 +768,12 @@ def _publish_cancelled_task(
|
|||
q._emit_cancel_task_done(task, task_id, cost_fields=cost_fields, status=settled_status)
|
||||
except Exception:
|
||||
log.warning("Failed to publish terminal event for %s", task_id, exc_info=True)
|
||||
# Respawn recovery is the REAPER'S canonical step 5, not a private variant:
|
||||
# membership check + respawn under the queue lock (mutually exclusive with
|
||||
# shutdown's kill_workers), and on failure the marker is cleared UNDER THE
|
||||
# LOCK so the crash detector can recover the slot on a later tick.
|
||||
# Respawn recovery is the REAPER'S canonical step 5, not a private variant.
|
||||
# The helper serializes against shutdown with the lifecycle lock and starts
|
||||
# the child outside the queue lock; on failure the marker is cleared under
|
||||
# the lock so the crash detector can recover the slot on a later tick.
|
||||
try:
|
||||
with q._queue_lock:
|
||||
if worker.wid in workers.WORKERS:
|
||||
workers.respawn_worker(worker.wid)
|
||||
workers.respawn_worker(worker.wid)
|
||||
except Exception:
|
||||
log.warning("Respawn after cancelling %s failed; clearing reaping for recovery", task_id, exc_info=True)
|
||||
try:
|
||||
|
|
|
|||
|
|
@ -514,16 +514,10 @@ def reap_timed_out_task(job: Dict[str, Any]) -> None:
|
|||
|
||||
# 5. Respawn a fresh worker for the slot; on failure, CLEAR reaping so the crash detector
|
||||
# can recover the slot on a later tick instead of stranding it permanently.
|
||||
# Hold _queue_lock across the membership check AND the respawn so it is mutually
|
||||
# exclusive with kill_workers (which clears WORKERS under the same lock at shutdown).
|
||||
# Otherwise the reaper could pass the check, start a replacement process, and insert it
|
||||
# into WORKERS only AFTER shutdown cleanup already cleared the pool — an orphan worker
|
||||
# surviving shutdown. _queue_lock is an RLock and respawn_worker re-acquires it
|
||||
# internally, so taking it here is safe (and a cleared pool makes the check fail closed).
|
||||
# respawn_worker owns the lifecycle race with shutdown and starts the child
|
||||
# outside _queue_lock, so a fork can never inherit the RLock from this thread.
|
||||
try:
|
||||
with _q._queue_lock:
|
||||
if worker_id in workers_mod.WORKERS:
|
||||
workers_mod.respawn_worker(worker_id)
|
||||
workers_mod.respawn_worker(worker_id)
|
||||
except Exception:
|
||||
log.warning("Reaper: respawn failed for worker %d; clearing reaping for recovery", worker_id, exc_info=True)
|
||||
try:
|
||||
|
|
|
|||
|
|
@ -434,7 +434,7 @@ def enqueue_assisted_resolution_task(tx: Dict[str, Any]) -> str:
|
|||
worker for it. Used by both the apply orchestration and boot recovery so the objective +
|
||||
structured metadata stay in one place. Returns the task id."""
|
||||
from supervisor.queue import enqueue_task
|
||||
from supervisor.workers import spawn_workers
|
||||
from supervisor.workers import ensure_worker_pool_started
|
||||
|
||||
task_id = str(tx.get("task_id") or "")
|
||||
task = {
|
||||
|
|
@ -452,9 +452,12 @@ def enqueue_assisted_resolution_task(tx: Dict[str, Any]) -> str:
|
|||
}
|
||||
enqueue_task(task, front=True)
|
||||
try:
|
||||
spawn_workers()
|
||||
if not ensure_worker_pool_started(allow_disabled_restart=True):
|
||||
_g.log.warning(
|
||||
"enqueue_assisted_resolution_task: worker pool remains explicitly disabled"
|
||||
)
|
||||
except Exception:
|
||||
_g.log.warning("enqueue_assisted_resolution_task: spawn_workers failed", exc_info=True)
|
||||
_g.log.warning("enqueue_assisted_resolution_task: worker pool start failed", exc_info=True)
|
||||
return task_id
|
||||
|
||||
|
||||
|
|
@ -713,11 +716,14 @@ def abort_orphaned_assisted_tx(task_id: str) -> Dict[str, Any]:
|
|||
ok, msg = rollback_managed_update("assisted_resolution_orphaned")
|
||||
_log_supervisor({"type": "managed_update_assisted_orphaned_rollback", "ok": ok, "msg": msg})
|
||||
try:
|
||||
from supervisor.workers import spawn_workers
|
||||
from supervisor.workers import ensure_worker_pool_started
|
||||
|
||||
spawn_workers()
|
||||
if not ensure_worker_pool_started(allow_disabled_restart=True):
|
||||
_g.log.warning(
|
||||
"abort_orphaned_assisted_tx: worker pool remains explicitly disabled"
|
||||
)
|
||||
except Exception:
|
||||
_g.log.warning("abort_orphaned_assisted_tx: spawn_workers failed", exc_info=True)
|
||||
_g.log.warning("abort_orphaned_assisted_tx: worker pool start failed", exc_info=True)
|
||||
return {"acted": True, "rolled_back": ok, "msg": msg}
|
||||
finally:
|
||||
if lock_fh is not None:
|
||||
|
|
|
|||
|
|
@ -92,16 +92,97 @@ class Worker:
|
|||
|
||||
|
||||
_EVENT_Q = None
|
||||
_EVENT_Q_MANAGER = None
|
||||
_EVENT_Q_GENERATION = ""
|
||||
_EVENT_Q_LOCK = threading.Lock()
|
||||
_EVENT_Q_SHUTDOWN = False
|
||||
_WORKER_POOL_DISABLED_REASON = ""
|
||||
_WORKER_LIFECYCLE_LOCK = threading.RLock()
|
||||
|
||||
|
||||
def _serialized_worker_lifecycle(fn):
|
||||
def wrapped(*args, **kwargs):
|
||||
with _WORKER_LIFECYCLE_LOCK:
|
||||
return fn(*args, **kwargs)
|
||||
|
||||
return wrapped
|
||||
|
||||
|
||||
def get_event_q():
|
||||
"""Return EVENT_Q, creating it lazily."""
|
||||
global _EVENT_Q
|
||||
if _EVENT_Q is None:
|
||||
_EVENT_Q = _get_ctx().Queue()
|
||||
"""Return the process-lifetime supervisor event bus, creating it lazily.
|
||||
|
||||
Worker-pool generations are replaceable; the producers that publish onto
|
||||
this bus (direct chat, consciousness, active turns, and workers) are not.
|
||||
Rotating the queue during a pool respawn strands those producers on an
|
||||
undrained queue, so only a new server process creates a new bus.
|
||||
"""
|
||||
global _EVENT_Q, _EVENT_Q_MANAGER, _EVENT_Q_GENERATION
|
||||
with _EVENT_Q_LOCK:
|
||||
if _EVENT_Q_SHUTDOWN:
|
||||
raise RuntimeError("supervisor event bus is shutting down")
|
||||
if _EVENT_Q is None:
|
||||
# A raw multiprocessing.Queue has an asynchronous feeder and its
|
||||
# pipe can be corrupted when a worker is force-killed mid-frame.
|
||||
# A manager-backed queue serializes synchronously in the producer
|
||||
# and isolates each producer connection, so replacing/killing a
|
||||
# worker generation cannot wedge the process-lifetime bus.
|
||||
_EVENT_Q_MANAGER = _get_ctx().Manager()
|
||||
_EVENT_Q = _EVENT_Q_MANAGER.Queue()
|
||||
_EVENT_Q_GENERATION = f"{os.getpid()}:{uuid.uuid4().hex[:12]}"
|
||||
try:
|
||||
from ouroboros.process_custody import record_process
|
||||
|
||||
manager_proc = getattr(_EVENT_Q_MANAGER, "_process", None)
|
||||
manager_pid = int(getattr(manager_proc, "pid", 0) or 0)
|
||||
if manager_pid:
|
||||
record_process(
|
||||
DRIVE_ROOT,
|
||||
pid=manager_pid,
|
||||
cmd="multiprocessing SyncManager",
|
||||
purpose="supervisor_event_queue_manager",
|
||||
scope="session",
|
||||
reap_process_group=False,
|
||||
)
|
||||
except Exception:
|
||||
log.warning("Failed to custody-track event queue manager", exc_info=True)
|
||||
try:
|
||||
append_jsonl(
|
||||
DRIVE_ROOT / "logs" / "supervisor.jsonl",
|
||||
{
|
||||
"ts": utc_now_iso(),
|
||||
"type": "event_queue_generation_started",
|
||||
"generation": _EVENT_Q_GENERATION,
|
||||
"server_pid": os.getpid(),
|
||||
"start_method": _WORKER_START_METHOD,
|
||||
},
|
||||
)
|
||||
except Exception:
|
||||
log.debug("Failed to record event queue generation", exc_info=True)
|
||||
return _EVENT_Q
|
||||
|
||||
|
||||
def shutdown_event_q() -> None:
|
||||
"""Stop the manager on graceful exit; custody reaps it after a hard exit."""
|
||||
global _EVENT_Q, _EVENT_Q_MANAGER, _EVENT_Q_GENERATION, _EVENT_Q_SHUTDOWN
|
||||
with _EVENT_Q_LOCK:
|
||||
_EVENT_Q_SHUTDOWN = True
|
||||
manager = _EVENT_Q_MANAGER
|
||||
_EVENT_Q = None
|
||||
_EVENT_Q_MANAGER = None
|
||||
_EVENT_Q_GENERATION = ""
|
||||
if manager is not None:
|
||||
try:
|
||||
manager.shutdown()
|
||||
except Exception:
|
||||
log.debug("Event queue manager shutdown failed", exc_info=True)
|
||||
|
||||
|
||||
def event_queue_generation() -> str:
|
||||
"""Stable diagnostic identity for the current server-process event bus."""
|
||||
get_event_q()
|
||||
return _EVENT_Q_GENERATION
|
||||
|
||||
|
||||
WORKERS: Dict[int, Worker] = {}
|
||||
PENDING: List[Dict[str, Any]] = []
|
||||
RUNNING: Dict[str, Dict[str, Any]] = {}
|
||||
|
|
@ -112,6 +193,38 @@ QUEUE_SEQ_COUNTER_REF: Dict[str, int] = {"value": 0}
|
|||
from supervisor.queue import _queue_lock
|
||||
|
||||
|
||||
def worker_pool_admission_state(ctx: Any = None) -> Dict[str, Any]:
|
||||
"""Return the user-facing managed-task executor admission state.
|
||||
|
||||
A busy or reaping pool is still a valid queue target. Only an explicitly
|
||||
disabled pool, or a genuinely absent pool after supervisor readiness, is
|
||||
unavailable. Internal boot/update recovery may enqueue before an initial
|
||||
spawn and therefore does not use this user-ingress predicate.
|
||||
"""
|
||||
pool = getattr(ctx, "WORKERS", WORKERS) if ctx is not None else WORKERS
|
||||
with _queue_lock:
|
||||
disabled_reason = str(_WORKER_POOL_DISABLED_REASON or "")
|
||||
worker_count = len(pool)
|
||||
available = worker_count > 0 and not disabled_reason
|
||||
return {
|
||||
"available": available,
|
||||
"reason_code": "" if available else "worker_pool_unavailable",
|
||||
"disabled_reason": disabled_reason or ("no_workers" if not worker_count else ""),
|
||||
"worker_count": worker_count,
|
||||
}
|
||||
|
||||
|
||||
def ensure_worker_pool_started(n: int = 0, *, allow_disabled_restart: bool = False) -> bool:
|
||||
"""Start an absent pool; only explicit internal recovery may clear disablement."""
|
||||
state = worker_pool_admission_state()
|
||||
if state["available"]:
|
||||
return True
|
||||
if state["disabled_reason"] not in {"", "no_workers"} and not allow_disabled_restart:
|
||||
return False
|
||||
spawn_workers(n)
|
||||
return True
|
||||
|
||||
|
||||
_chat_agent = None
|
||||
# Serializes every direct-chat caller; _chat_agent has mutable per-call state.
|
||||
import threading as _threading
|
||||
|
|
@ -227,6 +340,27 @@ def _canonical_promoted_repair_constraint(value: Any) -> tuple[Optional[dict], s
|
|||
}, ""
|
||||
|
||||
|
||||
def _promote_duplicate_reason(task_id: str, ctx: Any) -> str:
|
||||
"""Fail closed if a promoted id is already live, durable, or uncheckable."""
|
||||
pending = getattr(ctx, "PENDING", PENDING)
|
||||
running = getattr(ctx, "RUNNING", RUNNING)
|
||||
with _queue_lock:
|
||||
live_duplicate = any(
|
||||
isinstance(row, dict) and str(row.get("id") or "") == task_id
|
||||
for row in list(pending or [])
|
||||
) or task_id in (running or {})
|
||||
try:
|
||||
from ouroboros.task_results import load_task_result
|
||||
|
||||
stored_duplicate = bool(
|
||||
load_task_result(getattr(ctx, "DRIVE_ROOT", DRIVE_ROOT), task_id)
|
||||
)
|
||||
except Exception:
|
||||
log.warning("promote: duplicate-id lookup failed for %s", task_id, exc_info=True)
|
||||
return "task_id_lookup_failed"
|
||||
return "duplicate_task_id" if live_duplicate or stored_duplicate else ""
|
||||
|
||||
|
||||
def promote_chat_to_task(evt: dict, ctx: Any) -> dict:
|
||||
"""Enqueue a first-class pooled owner task from a conversation-lane promote.
|
||||
|
||||
|
|
@ -236,10 +370,24 @@ def promote_chat_to_task(evt: dict, ctx: Any) -> dict:
|
|||
"""
|
||||
from ouroboros.contracts.task_contract import attach_task_contract
|
||||
|
||||
tid = str(evt.get("task_id") or uuid.uuid4().hex[:8])
|
||||
tid = str(evt.get("task_id") or uuid.uuid4().hex[:16])
|
||||
admission_token = str(evt.get("routing_token") or "").strip()
|
||||
objective = str(evt.get("objective") or "").strip()
|
||||
if not objective:
|
||||
return {"status": "needs_manual_target", "reason": "empty_objective", "task_id": tid}
|
||||
# Reject before project/source/workspace side effects. enqueue_task repeats
|
||||
# the check atomically for the tiny race before queue insertion.
|
||||
duplicate_reason = _promote_duplicate_reason(tid, ctx)
|
||||
if duplicate_reason:
|
||||
return {
|
||||
"status": "needs_manual_target",
|
||||
"reason": duplicate_reason,
|
||||
"task_id": tid,
|
||||
}
|
||||
|
||||
evt = dict(evt)
|
||||
source_note = str(evt.get("_source_note") or "")
|
||||
effective_pid = str(evt.get("project_id") or "")
|
||||
repair_constraint, constraint_error = _canonical_promoted_repair_constraint(
|
||||
evt.get("task_constraint")
|
||||
)
|
||||
|
|
@ -274,6 +422,10 @@ def promote_chat_to_task(evt: dict, ctx: Any) -> dict:
|
|||
"expected_output": expected_output,
|
||||
"title": title,
|
||||
"source": "promote_chat_to_task",
|
||||
"_require_unique_task_id": True,
|
||||
"_require_worker_pool": True,
|
||||
"_admission_token": admission_token,
|
||||
"promotion_admission_token": admission_token,
|
||||
}
|
||||
if repair_constraint is not None:
|
||||
# Must be present before attach_task_contract so the managed root task
|
||||
|
|
@ -468,19 +620,34 @@ def promote_chat_to_task(evt: dict, ctx: Any) -> dict:
|
|||
"project_lifecycle": str(admitted.get("_project_lifecycle") or ""),
|
||||
"task_id": tid,
|
||||
}
|
||||
# v6.40 "name ANY task card": the agent already coined `title` here (zero extra LLM
|
||||
# call), so persist it as suggested_name + emit task_named so the promoted card shows
|
||||
# the human title up front exactly like a proactively-named direct-chat card, and a
|
||||
# later turn-into-project reuses it. Same-status (SCHEDULED) write — merges, never
|
||||
# regresses; fail-soft.
|
||||
if title:
|
||||
try:
|
||||
from ouroboros.task_results import STATUS_SCHEDULED, write_task_result
|
||||
|
||||
write_task_result(DRIVE_ROOT, tid, STATUS_SCHEDULED, suggested_name=title)
|
||||
_broadcast_task_named({"type": "task_named", "task_id": tid, "suggested_name": title})
|
||||
except Exception:
|
||||
log.debug("promote: suggested_name persist/broadcast failed for %s", tid, exc_info=True)
|
||||
# A positive promote confirmation is allowed only after the durable queue
|
||||
# projection exists. The event handler writes the scheduled task result
|
||||
# after the routing receipt; keeping that last step outside this function
|
||||
# makes the result itself the cross-process admission receipt.
|
||||
persist_snapshot = getattr(ctx, "persist_queue_snapshot", None)
|
||||
if not callable(persist_snapshot):
|
||||
return {
|
||||
"status": "needs_manual_target",
|
||||
"reason": "queue_snapshot_persist_unavailable",
|
||||
"task_id": tid,
|
||||
"admission_started": True,
|
||||
}
|
||||
try:
|
||||
if persist_snapshot(reason="promote_chat_to_task") is False:
|
||||
return {
|
||||
"status": "needs_manual_target",
|
||||
"reason": "queue_snapshot_persist_failed",
|
||||
"task_id": tid,
|
||||
"admission_started": True,
|
||||
}
|
||||
except Exception:
|
||||
log.warning("promote: queue snapshot persist failed for %s", tid, exc_info=True)
|
||||
return {
|
||||
"status": "needs_manual_target",
|
||||
"reason": "queue_snapshot_persist_failed",
|
||||
"task_id": tid,
|
||||
"admission_started": True,
|
||||
}
|
||||
# v6.82 (P5) disclosed residual: a PROMOTED root carries the host-attested
|
||||
# `cancelable` marker from its first RUNNING relay, not from enqueue — the
|
||||
# promote path emits no owner-facing progress frame of its own, and minting a
|
||||
|
|
@ -488,7 +655,12 @@ def promote_chat_to_task(evt: dict, ctx: Any) -> dict:
|
|||
# message seam (tests/test_heartbeat_presentation.py). While it is still
|
||||
# PENDING the Dashboard Activity row cancels it; the card action appears once
|
||||
# it starts.
|
||||
return {"status": "scheduled", "task_id": tid}
|
||||
outcome = {"status": "scheduled", "task_id": tid}
|
||||
if effective_pid:
|
||||
outcome["project_id"] = effective_pid
|
||||
if source_note:
|
||||
outcome["source_note"] = source_note
|
||||
return outcome
|
||||
|
||||
|
||||
def _fail_promoted_task_loudly(ctx: Any, task: dict, ws_error: str) -> None:
|
||||
|
|
@ -1366,14 +1538,22 @@ def reap_orphaned_workers() -> int:
|
|||
return len(killed)
|
||||
|
||||
|
||||
@_serialized_worker_lifecycle
|
||||
def spawn_workers(n: int = 0) -> None:
|
||||
global _CTX, _EVENT_Q
|
||||
# Reap any workers left orphaned by a prior/abrupt server exit before we
|
||||
# spawn fresh ones, so process groups do not accumulate across restarts.
|
||||
global _CTX, _WORKER_POOL_DISABLED_REASON
|
||||
global _LAST_SPAWN_TIME
|
||||
with _queue_lock:
|
||||
if WORKERS:
|
||||
raise RuntimeError(
|
||||
"spawn_workers requires an empty pool; stop the current workers "
|
||||
"or use respawn_worker for one slot"
|
||||
)
|
||||
# Never hold the queue's process-local threading.RLock across fork: a child
|
||||
# would inherit it owned by a vanished thread. The dedicated lifecycle lock
|
||||
# is not used by worker code and serializes competing full-pool starts/kills.
|
||||
reap_orphaned_workers()
|
||||
# Fresh context ensures workers use current code.
|
||||
_CTX = mp.get_context(_WORKER_START_METHOD)
|
||||
_EVENT_Q = _CTX.Queue()
|
||||
event_q = get_event_q()
|
||||
events_path = DRIVE_ROOT / "logs" / "events.jsonl"
|
||||
try:
|
||||
events_offset = int(events_path.stat().st_size)
|
||||
|
|
@ -1388,33 +1568,63 @@ def spawn_workers(n: int = 0) -> None:
|
|||
"type": "worker_spawn_start",
|
||||
"start_method": _WORKER_START_METHOD,
|
||||
"count": count,
|
||||
"event_queue_generation": event_queue_generation(),
|
||||
"event_queue_transport": "manager",
|
||||
},
|
||||
)
|
||||
WORKERS.clear()
|
||||
for i in range(count):
|
||||
in_q = _CTX.Queue()
|
||||
proc = _CTX.Process(target=worker_main,
|
||||
args=(i, in_q, _EVENT_Q, str(REPO_DIR), str(DRIVE_ROOT),
|
||||
_current_custody_session_id()))
|
||||
proc.daemon = True
|
||||
proc.start()
|
||||
WORKERS[i] = Worker(wid=i, proc=proc, in_q=in_q, busy_task_id=None)
|
||||
global _LAST_SPAWN_TIME
|
||||
_LAST_SPAWN_TIME = time.time()
|
||||
new_workers: Dict[int, Worker] = {}
|
||||
try:
|
||||
for i in range(count):
|
||||
in_q = _CTX.Queue()
|
||||
proc = _CTX.Process(target=worker_main,
|
||||
args=(i, in_q, event_q, str(REPO_DIR), str(DRIVE_ROOT),
|
||||
_current_custody_session_id()))
|
||||
proc.daemon = True
|
||||
proc.start()
|
||||
new_workers[i] = Worker(wid=i, proc=proc, in_q=in_q, busy_task_id=None)
|
||||
except Exception:
|
||||
for worker in new_workers.values():
|
||||
try:
|
||||
worker.proc.terminate()
|
||||
worker.proc.join(timeout=2)
|
||||
except Exception:
|
||||
pass
|
||||
raise
|
||||
with _queue_lock:
|
||||
if WORKERS:
|
||||
for worker in new_workers.values():
|
||||
try:
|
||||
worker.proc.terminate()
|
||||
worker.proc.join(timeout=2)
|
||||
except Exception:
|
||||
pass
|
||||
raise RuntimeError("worker pool appeared during serialized startup")
|
||||
WORKERS.update(new_workers)
|
||||
_WORKER_POOL_DISABLED_REASON = ""
|
||||
_LAST_SPAWN_TIME = time.time()
|
||||
_record_worker_pids()
|
||||
# Verify asynchronously so spawn does not block the supervisor loop.
|
||||
threading.Thread(target=_verify_worker_sha_after_spawn, args=(events_offset,), daemon=True).start()
|
||||
|
||||
|
||||
@_serialized_worker_lifecycle
|
||||
def kill_workers(
|
||||
force: bool = True,
|
||||
*,
|
||||
result_reason: str = "Worker process crashed (crash storm). Task was not completed.",
|
||||
terminal_status: str = "",
|
||||
archive_service_logs: bool = True,
|
||||
disable_reason: str = "",
|
||||
) -> None:
|
||||
global _WORKER_POOL_DISABLED_REASON
|
||||
from supervisor import queue
|
||||
with _queue_lock:
|
||||
if disable_reason:
|
||||
_WORKER_POOL_DISABLED_REASON = str(disable_reason)
|
||||
# Publish the admission fence before slow process-tree teardown so
|
||||
# concurrent ingress can refuse without starting project/workspace
|
||||
# side effects while workers are being joined.
|
||||
queue.persist_queue_snapshot(reason="worker_pool_disabling")
|
||||
cleared_running = len(RUNNING)
|
||||
from ouroboros.platform_layer import kill_pid_tree
|
||||
for w in WORKERS.values():
|
||||
|
|
@ -1497,19 +1707,55 @@ def _kill_survivors() -> None:
|
|||
w.proc.join(timeout=2)
|
||||
|
||||
|
||||
def respawn_worker(wid: int) -> None:
|
||||
@_serialized_worker_lifecycle
|
||||
def respawn_worker(wid: int) -> bool:
|
||||
"""Replace one owned slot without forking under the queue RLock.
|
||||
|
||||
The lifecycle lock makes the two-phase check/start/swap mutually exclusive
|
||||
with full-pool shutdown/start. The identity check after ``proc.start()``
|
||||
prevents a replacement from being installed if the slot was removed while
|
||||
the queue lock was released.
|
||||
"""
|
||||
with _queue_lock:
|
||||
old = WORKERS.get(wid)
|
||||
if old is None:
|
||||
return False
|
||||
ctx = _get_ctx()
|
||||
in_q = ctx.Queue()
|
||||
proc = ctx.Process(target=worker_main,
|
||||
args=(wid, in_q, get_event_q(), str(REPO_DIR), str(DRIVE_ROOT),
|
||||
_current_custody_session_id()))
|
||||
proc.daemon = True
|
||||
proc.start()
|
||||
# Swap under _queue_lock (an RLock — safe even when the caller already holds
|
||||
# it) so a concurrent assign_tasks cannot enqueue into the slot mid-swap.
|
||||
try:
|
||||
proc.start()
|
||||
except Exception:
|
||||
try:
|
||||
in_q.close()
|
||||
in_q.cancel_join_thread()
|
||||
except Exception:
|
||||
pass
|
||||
raise
|
||||
installed = False
|
||||
with _queue_lock:
|
||||
old = WORKERS.get(wid)
|
||||
WORKERS[wid] = Worker(wid=wid, proc=proc, in_q=in_q, busy_task_id=None)
|
||||
if WORKERS.get(wid) is old:
|
||||
WORKERS[wid] = Worker(wid=wid, proc=proc, in_q=in_q, busy_task_id=None)
|
||||
installed = True
|
||||
if not installed:
|
||||
try:
|
||||
from ouroboros.platform_layer import kill_pid_tree
|
||||
|
||||
if proc.pid:
|
||||
kill_pid_tree(proc.pid)
|
||||
elif proc.is_alive():
|
||||
proc.terminate()
|
||||
proc.join(timeout=2)
|
||||
finally:
|
||||
try:
|
||||
in_q.close()
|
||||
in_q.cancel_join_thread()
|
||||
except Exception:
|
||||
pass
|
||||
return False
|
||||
# Close the crashed worker's old queue now that nothing can route to it,
|
||||
# otherwise its file descriptors / semaphores leak on every respawn.
|
||||
if old is not None and getattr(old, "in_q", None) is not None:
|
||||
|
|
@ -1520,6 +1766,7 @@ def respawn_worker(wid: int) -> None:
|
|||
log.debug("Failed to close old worker queue on respawn", exc_info=True)
|
||||
_record_worker_pids()
|
||||
# Do not reset _LAST_SPAWN_TIME here; respawn grace would hide crash storms.
|
||||
return True
|
||||
|
||||
|
||||
def _drop_cancelled_pending() -> None:
|
||||
|
|
@ -1829,13 +2076,32 @@ def ensure_workers_healthy() -> None:
|
|||
if (time.time() - _LAST_SPAWN_TIME) < _SPAWN_GRACE_SEC:
|
||||
return
|
||||
with _queue_lock:
|
||||
_ensure_workers_healthy_locked(queue)
|
||||
respawn_ids, disable_pool = _ensure_workers_healthy_locked(queue)
|
||||
if disable_pool:
|
||||
# Every lifecycle operation takes lifecycle -> queue lock. Calling
|
||||
# kill_workers while still holding queue lock would invert that order
|
||||
# against a concurrent respawn and deadlock.
|
||||
kill_workers(disable_reason="worker_crash_storm")
|
||||
CRASH_TS.clear()
|
||||
return
|
||||
for wid in respawn_ids:
|
||||
try:
|
||||
respawn_worker(wid)
|
||||
except Exception:
|
||||
log.warning("Failed to respawn crashed worker %d", wid, exc_info=True)
|
||||
with _queue_lock:
|
||||
slot = WORKERS.get(wid)
|
||||
if slot is not None:
|
||||
slot.reaping = False
|
||||
if respawn_ids:
|
||||
queue.persist_queue_snapshot(reason="worker_respawn_after_crash")
|
||||
|
||||
|
||||
def _ensure_workers_healthy_locked(queue: Any) -> None:
|
||||
def _ensure_workers_healthy_locked(queue: Any) -> tuple[List[int], bool]:
|
||||
busy_crashes = 0
|
||||
dead_detections = 0
|
||||
crashed_tasks = []
|
||||
respawn_ids: List[int] = []
|
||||
for wid, w in list(WORKERS.items()):
|
||||
# Variant A: a slot marked `reaping` is owned end-to-end by the background reaper
|
||||
# (kill -> join -> archive -> respawn). Its proc is expected to die mid-reap, so the
|
||||
|
|
@ -1844,6 +2110,9 @@ def _ensure_workers_healthy_locked(queue: Any) -> None:
|
|||
if getattr(w, "reaping", False):
|
||||
continue
|
||||
if not w.proc.is_alive():
|
||||
# Reserve the dead slot before the main loop releases the queue lock
|
||||
# to start its replacement. assign_tasks skips reaping slots.
|
||||
w.reaping = True
|
||||
dead_detections += 1
|
||||
if w.busy_task_id is not None:
|
||||
busy_crashes += 1
|
||||
|
|
@ -2082,8 +2351,7 @@ def _ensure_workers_healthy_locked(queue: Any) -> None:
|
|||
reason_code=reason_code,
|
||||
**r_cost_fields,
|
||||
)
|
||||
respawn_worker(wid)
|
||||
queue.persist_queue_snapshot(reason="worker_respawn_after_crash")
|
||||
respawn_ids.append(wid)
|
||||
|
||||
now = time.time()
|
||||
alive_now = sum(1 for w in WORKERS.values() if w.proc.is_alive())
|
||||
|
|
@ -2095,7 +2363,8 @@ def _ensure_workers_healthy_locked(queue: Any) -> None:
|
|||
CRASH_TS.clear()
|
||||
|
||||
CRASH_TS[:] = [t for t in CRASH_TS if (now - t) < 60.0]
|
||||
if len(CRASH_TS) >= 3:
|
||||
disable_pool = len(CRASH_TS) >= 3
|
||||
if disable_pool:
|
||||
# Do not execv on crash storms; keep direct-chat mode alive.
|
||||
st = load_state()
|
||||
append_jsonl(
|
||||
|
|
@ -2119,5 +2388,4 @@ def _ensure_workers_healthy_locked(queue: Any) -> None:
|
|||
"toast_once": f"worker-crash-storm:{int(min(CRASH_TS) if CRASH_TS else now)}",
|
||||
},
|
||||
)
|
||||
kill_workers()
|
||||
CRASH_TS.clear()
|
||||
return respawn_ids, disable_pool
|
||||
|
|
|
|||
|
|
@ -269,6 +269,8 @@ _TRUNCATION_DECISIONS: dict[str, tuple[bool, str]] = {
|
|||
"delegation_constraint_require_lane": (False, "control_delegation.py:126 rejected call"),
|
||||
"deep_self_review_unavailable": (False, "agent.py:705 owner-config gap on a review task"),
|
||||
"deep_self_review_error": (False, "agent.py:743 review-stage error on a review task"),
|
||||
"worker_pool_unavailable": (False, "gateway/tasks.py managed-task admission refusal"),
|
||||
"worker_pool_state_unavailable": (False, "gateway/tasks.py fail-closed admission inspection"),
|
||||
}
|
||||
|
||||
_REASON_CODE_LITERAL = re.compile(
|
||||
|
|
|
|||
|
|
@ -54,8 +54,10 @@ def test_api_tasks_create_carries_disabled_tools(tmp_path, monkeypatch):
|
|||
(data / "memory").mkdir(parents=True)
|
||||
|
||||
captured = []
|
||||
monkeypatch.setattr("supervisor.workers.WORKERS", {0: object()})
|
||||
monkeypatch.setattr("supervisor.workers._WORKER_POOL_DISABLED_REASON", "")
|
||||
monkeypatch.setattr("supervisor.queue.enqueue_task", lambda task: captured.append(dict(task)) or task)
|
||||
monkeypatch.setattr("supervisor.queue.persist_queue_snapshot", lambda reason="": None)
|
||||
monkeypatch.setattr("supervisor.queue.persist_queue_snapshot", lambda reason="": True)
|
||||
monkeypatch.setattr("ouroboros.workspace_admission.bootstrap_process_path", lambda: [])
|
||||
|
||||
app = Starlette(routes=[Route("/api/tasks", endpoint=api_tasks_create, methods=["POST"])])
|
||||
|
|
@ -81,8 +83,10 @@ def test_api_tasks_create_carries_acceptance_claims(tmp_path, monkeypatch):
|
|||
data = tmp_path / "data"
|
||||
(data / "memory").mkdir(parents=True)
|
||||
captured = []
|
||||
monkeypatch.setattr("supervisor.workers.WORKERS", {0: object()})
|
||||
monkeypatch.setattr("supervisor.workers._WORKER_POOL_DISABLED_REASON", "")
|
||||
monkeypatch.setattr("supervisor.queue.enqueue_task", lambda task: captured.append(dict(task)) or task)
|
||||
monkeypatch.setattr("supervisor.queue.persist_queue_snapshot", lambda reason="": None)
|
||||
monkeypatch.setattr("supervisor.queue.persist_queue_snapshot", lambda reason="": True)
|
||||
monkeypatch.setattr("ouroboros.workspace_admission.bootstrap_process_path", lambda: [])
|
||||
|
||||
app = Starlette(routes=[Route("/api/tasks", endpoint=api_tasks_create, methods=["POST"])])
|
||||
|
|
|
|||
|
|
@ -45,6 +45,15 @@ from ouroboros.utils import utc_now_iso
|
|||
from ouroboros.workspace_preflight import _infer_tools_from_manifests
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _managed_worker_pool_available(monkeypatch):
|
||||
"""HTTP task tests model a ready server unless a case overrides the pool."""
|
||||
import supervisor.workers as workers
|
||||
|
||||
monkeypatch.setattr(workers, "WORKERS", {0: SimpleNamespace()})
|
||||
monkeypatch.setattr(workers, "_WORKER_POOL_DISABLED_REASON", "")
|
||||
|
||||
|
||||
def _init_repo_with_file(repo, name="tracked.txt", content="old\n"):
|
||||
repo.mkdir()
|
||||
subprocess.run(["git", "init"], cwd=repo, check=True, capture_output=True)
|
||||
|
|
@ -76,7 +85,7 @@ def test_task_api_enqueue_workspace_creates_child_drive(tmp_path, monkeypatch):
|
|||
return task
|
||||
|
||||
monkeypatch.setattr("supervisor.queue.enqueue_task", fake_enqueue)
|
||||
monkeypatch.setattr("supervisor.queue.persist_queue_snapshot", lambda reason="": None)
|
||||
monkeypatch.setattr("supervisor.queue.persist_queue_snapshot", lambda reason="": True)
|
||||
monkeypatch.setattr("ouroboros.workspace_admission.bootstrap_process_path", lambda: bootstrapped.append(True) or [])
|
||||
|
||||
app = Starlette(routes=[Route("/api/tasks", endpoint=api_tasks_create, methods=["POST"])])
|
||||
|
|
@ -198,6 +207,79 @@ def test_task_api_admission_refusal_is_terminal_not_scheduled_phantom(tmp_path,
|
|||
assert not (data / "state" / "headless_tasks" / "blocked-root").exists()
|
||||
|
||||
|
||||
def test_task_api_refuses_when_durable_queue_snapshot_fails(tmp_path, monkeypatch):
|
||||
import supervisor.queue as queue
|
||||
from ouroboros.task_results import STATUS_FAILED, load_task_result
|
||||
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir()
|
||||
data = tmp_path / "data"
|
||||
(data / "memory").mkdir(parents=True)
|
||||
pending = []
|
||||
monkeypatch.setattr(queue, "DRIVE_ROOT", data)
|
||||
monkeypatch.setattr(queue, "PENDING", pending)
|
||||
monkeypatch.setattr(queue, "RUNNING", {})
|
||||
calls = []
|
||||
|
||||
def persist(reason=""):
|
||||
calls.append(reason)
|
||||
return reason == "api_task_create_rollback"
|
||||
|
||||
monkeypatch.setattr(queue, "persist_queue_snapshot", persist)
|
||||
app = Starlette(routes=[Route("/api/tasks", endpoint=api_tasks_create, methods=["POST"])])
|
||||
app.state.drive_root = data
|
||||
app.state.repo_dir = repo
|
||||
response = TestClient(app).post(
|
||||
"/api/tasks",
|
||||
json={"description": "must be durable", "task_id": "snapshot-fail"},
|
||||
)
|
||||
|
||||
assert response.status_code == 503
|
||||
assert response.json()["admission"]["reason_code"] == "queue_snapshot_persist_failed"
|
||||
assert pending == []
|
||||
assert calls == ["api_task_create", "api_task_create_rollback"]
|
||||
assert load_task_result(data, "snapshot-fail")["status"] == STATUS_FAILED
|
||||
assert not (data / "state" / "headless_tasks" / "snapshot-fail").exists()
|
||||
|
||||
|
||||
def test_task_api_releases_reservation_when_payload_composition_fails(
|
||||
tmp_path, monkeypatch,
|
||||
):
|
||||
import supervisor.queue as queue
|
||||
from ouroboros.gateway import tasks
|
||||
|
||||
data = tmp_path / "data"
|
||||
repo = tmp_path / "repo"
|
||||
data.mkdir()
|
||||
repo.mkdir()
|
||||
task_id = "compose-failure"
|
||||
real_compose = tasks._compose_task_text
|
||||
monkeypatch.setattr(
|
||||
tasks,
|
||||
"_compose_task_text",
|
||||
lambda *_args, **_kwargs: (_ for _ in ()).throw(RuntimeError("compose failed")),
|
||||
)
|
||||
monkeypatch.setattr(queue, "enqueue_task", lambda task: task)
|
||||
monkeypatch.setattr(queue, "persist_queue_snapshot", lambda **_kwargs: True)
|
||||
app = Starlette(routes=[Route("/api/tasks", endpoint=api_tasks_create, methods=["POST"])])
|
||||
app.state.drive_root = data
|
||||
app.state.repo_dir = repo
|
||||
client = TestClient(app)
|
||||
|
||||
failed = client.post(
|
||||
"/api/tasks", json={"task_id": task_id, "description": "compose me"}
|
||||
)
|
||||
assert failed.status_code == 503
|
||||
assert task_id not in queue.ADMISSION_RESERVATIONS
|
||||
assert not task_artifacts_dir(data, task_id, create=False).exists()
|
||||
|
||||
monkeypatch.setattr(tasks, "_compose_task_text", real_compose)
|
||||
retried = client.post(
|
||||
"/api/tasks", json={"task_id": task_id, "description": "compose me"}
|
||||
)
|
||||
assert retried.status_code == 200, retried.text
|
||||
|
||||
|
||||
def test_api_tasks_create_requires_description_not_legacy_aliases(monkeypatch):
|
||||
captured = []
|
||||
monkeypatch.setattr("supervisor.queue.enqueue_task", lambda task: captured.append(task) or task)
|
||||
|
|
@ -223,7 +305,7 @@ def test_api_tasks_create_rejects_internal_task_types(tmp_path, monkeypatch):
|
|||
(data / "memory").mkdir(parents=True)
|
||||
|
||||
monkeypatch.setattr("supervisor.queue.enqueue_task", lambda task: task)
|
||||
monkeypatch.setattr("supervisor.queue.persist_queue_snapshot", lambda reason="": None)
|
||||
monkeypatch.setattr("supervisor.queue.persist_queue_snapshot", lambda reason="": True)
|
||||
monkeypatch.setattr("ouroboros.workspace_admission.bootstrap_process_path", lambda: [])
|
||||
|
||||
app = Starlette(routes=[Route("/api/tasks", endpoint=api_tasks_create, methods=["POST"])])
|
||||
|
|
@ -267,7 +349,7 @@ def test_task_api_rejects_unsafe_task_id_and_system_workspace(tmp_path, monkeypa
|
|||
data = tmp_path / "data"
|
||||
data.mkdir()
|
||||
monkeypatch.setattr("supervisor.queue.enqueue_task", lambda task: task)
|
||||
monkeypatch.setattr("supervisor.queue.persist_queue_snapshot", lambda reason="": None)
|
||||
monkeypatch.setattr("supervisor.queue.persist_queue_snapshot", lambda reason="": True)
|
||||
|
||||
app = Starlette(routes=[Route("/api/tasks", endpoint=api_tasks_create, methods=["POST"])])
|
||||
app.state.drive_root = data
|
||||
|
|
@ -324,7 +406,7 @@ def test_task_api_rejects_forged_subagent_without_child_drive_side_effect(tmp_pa
|
|||
data = tmp_path / "data"
|
||||
data.mkdir()
|
||||
monkeypatch.setattr("supervisor.queue.enqueue_task", lambda task: pytest.fail("forged subagent enqueued"))
|
||||
monkeypatch.setattr("supervisor.queue.persist_queue_snapshot", lambda reason="": None)
|
||||
monkeypatch.setattr("supervisor.queue.persist_queue_snapshot", lambda reason="": True)
|
||||
|
||||
app = Starlette(routes=[Route("/api/tasks", endpoint=api_tasks_create, methods=["POST"])])
|
||||
app.state.drive_root = data
|
||||
|
|
@ -357,7 +439,7 @@ def test_task_api_rejects_external_lineage_forgery(tmp_path, monkeypatch):
|
|||
data = tmp_path / "data"
|
||||
data.mkdir()
|
||||
monkeypatch.setattr("supervisor.queue.enqueue_task", lambda task: pytest.fail("forged lineage enqueued"))
|
||||
monkeypatch.setattr("supervisor.queue.persist_queue_snapshot", lambda reason="": None)
|
||||
monkeypatch.setattr("supervisor.queue.persist_queue_snapshot", lambda reason="": True)
|
||||
|
||||
app = Starlette(routes=[Route("/api/tasks", endpoint=api_tasks_create, methods=["POST"])])
|
||||
app.state.drive_root = data
|
||||
|
|
@ -389,7 +471,7 @@ def test_task_api_preserves_top_level_actor_id_after_metadata_sanitization(tmp_p
|
|||
data.mkdir()
|
||||
captured = []
|
||||
monkeypatch.setattr("supervisor.queue.enqueue_task", lambda task: captured.append(dict(task)) or task)
|
||||
monkeypatch.setattr("supervisor.queue.persist_queue_snapshot", lambda reason="": None)
|
||||
monkeypatch.setattr("supervisor.queue.persist_queue_snapshot", lambda reason="": True)
|
||||
|
||||
app = Starlette(routes=[Route("/api/tasks", endpoint=api_tasks_create, methods=["POST"])])
|
||||
app.state.drive_root = data
|
||||
|
|
@ -2463,7 +2545,7 @@ def test_queue_restore_accepts_headless_chat_zero(tmp_path, monkeypatch):
|
|||
monkeypatch.setattr(queue, "DRIVE_ROOT", tmp_path)
|
||||
monkeypatch.setattr(queue, "QUEUE_SNAPSHOT_PATH", tmp_path / "queue_snapshot.json")
|
||||
monkeypatch.setattr(queue, "append_jsonl", lambda *args, **kwargs: None)
|
||||
monkeypatch.setattr(queue, "persist_queue_snapshot", lambda reason="": None)
|
||||
monkeypatch.setattr(queue, "persist_queue_snapshot", lambda reason="": True)
|
||||
(tmp_path / "queue_snapshot.json").write_text(
|
||||
json.dumps({
|
||||
"ts": utc_now_iso(),
|
||||
|
|
|
|||
|
|
@ -221,6 +221,7 @@ def test_d5_project_scoped_shared_preserves_mode_but_isolates_drive(tmp_path, mo
|
|||
|
||||
from ouroboros.gateway import tasks
|
||||
from supervisor import queue
|
||||
from supervisor import workers
|
||||
|
||||
async def fake_request_json_or(_request, _default):
|
||||
return {"description": "x", "project_id": "proj_x", "memory_mode": "shared"}
|
||||
|
|
@ -240,8 +241,14 @@ def test_d5_project_scoped_shared_preserves_mode_but_isolates_drive(tmp_path, mo
|
|||
monkeypatch.setattr(tasks, "request_drive_root", lambda _r: tmp_path / "data")
|
||||
monkeypatch.setattr(tasks, "request_repo_dir", lambda _r: tmp_path / "repo")
|
||||
monkeypatch.setattr(tasks, "prepare_task_drive", fake_prepare)
|
||||
monkeypatch.setattr(queue, "enqueue_task", lambda task: captured.update(task))
|
||||
monkeypatch.setattr(queue, "persist_queue_snapshot", lambda *a, **k: None)
|
||||
monkeypatch.setattr(
|
||||
queue,
|
||||
"enqueue_task",
|
||||
lambda task: captured.update(task) or task,
|
||||
)
|
||||
monkeypatch.setattr(queue, "persist_queue_snapshot", lambda *a, **k: True)
|
||||
monkeypatch.setattr(workers, "WORKERS", {0: SimpleNamespace()})
|
||||
monkeypatch.setattr(workers, "_WORKER_POOL_DISABLED_REASON", "")
|
||||
|
||||
request = SimpleNamespace(app=SimpleNamespace(state=SimpleNamespace(supervisor_ready_event=None)))
|
||||
response = asyncio.run(tasks.api_tasks_create(request))
|
||||
|
|
|
|||
|
|
@ -440,7 +440,7 @@ def test_routing_ack_is_typed_and_never_broadcast_as_chat_bubble(monkeypatch):
|
|||
assert all(payload.get("type") != "chat" for payload in ws_payloads)
|
||||
|
||||
|
||||
def test_manual_target_event_emits_transient_options_but_sidecar_stays_annotation_only(
|
||||
def test_manual_target_event_persists_concrete_options_in_latest_annotation(
|
||||
tmp_path, monkeypatch,
|
||||
):
|
||||
from ouroboros.project_dialogue import latest_chat_annotations
|
||||
|
|
@ -471,4 +471,4 @@ def test_manual_target_event_emits_transient_options_but_sidecar_stays_annotatio
|
|||
assert payload["suppress_bubble"] is True
|
||||
sidecar = latest_chat_annotations(tmp_path)["owner-choice-1"]
|
||||
assert set(sidecar) >= {"client_message_id", "action", "target", "status"}
|
||||
assert "options" not in sidecar
|
||||
assert sidecar["options"] == event["options"]
|
||||
|
|
|
|||
|
|
@ -5,9 +5,17 @@ from __future__ import annotations
|
|||
import types
|
||||
|
||||
|
||||
def test_promote_tool_emits_event_with_chat_and_project(tmp_path):
|
||||
def _confirm_promote(monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
"ouroboros.tools.control._wait_for_promotion_admission",
|
||||
lambda *_args, **_kwargs: {"status": "scheduled"},
|
||||
)
|
||||
|
||||
|
||||
def test_promote_tool_emits_event_with_chat_and_project(tmp_path, monkeypatch):
|
||||
from ouroboros.tools.control import _promote_chat_to_task
|
||||
|
||||
_confirm_promote(monkeypatch)
|
||||
events = []
|
||||
ctx = types.SimpleNamespace(
|
||||
pending_events=events,
|
||||
|
|
@ -16,7 +24,8 @@ def test_promote_tool_emits_event_with_chat_and_project(tmp_path):
|
|||
drive_root=tmp_path,
|
||||
)
|
||||
out = _promote_chat_to_task(ctx, "Build the racer prototype", project_id="racer")
|
||||
assert out.startswith("OK: promoted to supervised task")
|
||||
assert out.startswith("OK: task")
|
||||
assert "accepted and durably scheduled" in out
|
||||
assert len(events) == 1
|
||||
evt = events[0]
|
||||
assert evt["type"] == "promote_chat_to_task"
|
||||
|
|
@ -38,11 +47,12 @@ def test_promote_tool_rejects_dirty_project_id(tmp_path):
|
|||
assert not ctx.pending_events
|
||||
|
||||
|
||||
def test_promote_tool_project_name_creates_named_project_event(tmp_path):
|
||||
def test_promote_tool_project_name_creates_named_project_event(tmp_path, monkeypatch):
|
||||
"""LLM-first 'create a named project and work there' (v6.33.0): project_name
|
||||
derives a clean id, carries the human display name, and rides title."""
|
||||
from ouroboros.tools.control import _promote_chat_to_task
|
||||
|
||||
_confirm_promote(monkeypatch)
|
||||
events = []
|
||||
ctx = types.SimpleNamespace(
|
||||
pending_events=events, event_queue=None, current_chat_id=1, drive_root=tmp_path,
|
||||
|
|
@ -51,7 +61,7 @@ def test_promote_tool_project_name_creates_named_project_event(tmp_path):
|
|||
ctx, "research everything about the airi institute",
|
||||
project_name="Airi Research", title="Airi Research",
|
||||
)
|
||||
assert out.startswith("OK: promoted to supervised task")
|
||||
assert out.startswith("OK: task")
|
||||
assert "new project 'Airi Research'" in out
|
||||
evt = events[0]
|
||||
assert evt["project_name"] == "Airi Research"
|
||||
|
|
@ -71,19 +81,20 @@ def test_project_id_from_display_name_handles_non_ascii():
|
|||
assert project_id_from_display_name("") == ""
|
||||
|
||||
|
||||
def test_promote_tool_cyrillic_project_name_still_creates(tmp_path):
|
||||
def test_promote_tool_cyrillic_project_name_still_creates(tmp_path, monkeypatch):
|
||||
"""promote_chat_to_task(project_name=<cyrillic>) must NOT fail — it derives a
|
||||
hash id while keeping the Cyrillic display name (Workflow-caught regression)."""
|
||||
from ouroboros.project_facts import project_id_from_display_name
|
||||
from ouroboros.tools.control import _promote_chat_to_task
|
||||
|
||||
_confirm_promote(monkeypatch)
|
||||
events = []
|
||||
ctx = types.SimpleNamespace(
|
||||
pending_events=events, event_queue=None, current_chat_id=1, drive_root=tmp_path,
|
||||
)
|
||||
out = _promote_chat_to_task(ctx, "исследуй динозавров", project_name="динозавры", title="динозавры")
|
||||
assert "TOOL_ARG_ERROR" not in out
|
||||
assert out.startswith("OK: promoted")
|
||||
assert out.startswith("OK: task")
|
||||
evt = events[0]
|
||||
assert evt["project_name"] == "динозавры"
|
||||
assert evt["project_id"] == project_id_from_display_name("динозавры")
|
||||
|
|
@ -100,6 +111,7 @@ def test_promote_event_names_project_from_display_name(tmp_path, monkeypatch):
|
|||
enqueued = []
|
||||
ctx = types.SimpleNamespace(
|
||||
enqueue_task=lambda task: enqueued.append(task),
|
||||
persist_queue_snapshot=lambda **_kwargs: True,
|
||||
load_state=lambda: {"owner_chat_id": 1},
|
||||
)
|
||||
workers.promote_chat_to_task({
|
||||
|
|
@ -141,6 +153,7 @@ def test_promote_event_enqueues_first_class_task(tmp_path, monkeypatch):
|
|||
enqueued = []
|
||||
ctx = types.SimpleNamespace(
|
||||
enqueue_task=lambda task: enqueued.append(task),
|
||||
persist_queue_snapshot=lambda **_kwargs: True,
|
||||
load_state=lambda: {"owner_chat_id": 1},
|
||||
)
|
||||
evt = {
|
||||
|
|
@ -201,6 +214,7 @@ def test_route_to_project_event_emits_route_receipt_action(tmp_path, monkeypatch
|
|||
WORKERS={0: types.SimpleNamespace()},
|
||||
bridge=Bridge(),
|
||||
enqueue_task=lambda task: enqueued.append(task),
|
||||
persist_queue_snapshot=lambda **_kwargs: True,
|
||||
load_state=lambda: {"owner_chat_id": 1},
|
||||
append_jsonl=lambda *args, **kwargs: None,
|
||||
)
|
||||
|
|
@ -208,6 +222,7 @@ def test_route_to_project_event_emits_route_receipt_action(tmp_path, monkeypatch
|
|||
_handle_promote_chat_to_task({
|
||||
"type": "promote_chat_to_task",
|
||||
"task_id": "route01",
|
||||
"routing_token": "route-token-01",
|
||||
"objective": "Continue the racer",
|
||||
"project_id": "racer",
|
||||
"chat_id": 1,
|
||||
|
|
@ -229,6 +244,7 @@ def test_promoted_skill_repair_is_canonical_confined_managed_task(tmp_path, monk
|
|||
enqueued = []
|
||||
ctx = types.SimpleNamespace(
|
||||
enqueue_task=lambda task: enqueued.append(task),
|
||||
persist_queue_snapshot=lambda **_kwargs: True,
|
||||
load_state=lambda: {"owner_chat_id": 1},
|
||||
)
|
||||
|
||||
|
|
@ -268,6 +284,7 @@ def test_promoted_skill_repair_rejects_missing_payload(tmp_path, monkeypatch):
|
|||
enqueued = []
|
||||
ctx = types.SimpleNamespace(
|
||||
enqueue_task=lambda task: enqueued.append(task),
|
||||
persist_queue_snapshot=lambda **_kwargs: True,
|
||||
load_state=lambda: {"owner_chat_id": 1},
|
||||
)
|
||||
|
||||
|
|
@ -314,6 +331,7 @@ def test_promote_route_persists_source_ref_and_fails_closed_on_binding_error(tmp
|
|||
enqueued = []
|
||||
ctx = types.SimpleNamespace(
|
||||
enqueue_task=lambda task: enqueued.append(task),
|
||||
persist_queue_snapshot=lambda **_kwargs: True,
|
||||
load_state=lambda: {"owner_chat_id": 1},
|
||||
)
|
||||
from ouroboros.project_dialogue import _text_sha256
|
||||
|
|
@ -379,6 +397,7 @@ def test_promote_chat_to_task_broadcasts_projects_changed(tmp_path, monkeypatch)
|
|||
monkeypatch.setattr(mbus, "get_bridge", lambda: fake_bridge)
|
||||
ctx = types.SimpleNamespace(
|
||||
enqueue_task=lambda task: None,
|
||||
persist_queue_snapshot=lambda **_kwargs: True,
|
||||
load_state=lambda: {"owner_chat_id": 1},
|
||||
)
|
||||
workers.promote_chat_to_task({
|
||||
|
|
@ -631,7 +650,9 @@ def test_route_project_chat_defers_when_multiple_running_tasks(tmp_path, monkeyp
|
|||
project_chat = int(proj["chat_id"])
|
||||
|
||||
delivered = []
|
||||
monkeypatch.setattr(omb, "write_owner_message", lambda *a, **k: delivered.append(a))
|
||||
monkeypatch.setattr(
|
||||
omb, "write_owner_message", lambda *a, **k: delivered.append(a) or True
|
||||
)
|
||||
|
||||
ctx = types.SimpleNamespace(
|
||||
DRIVE_ROOT=tmp_path,
|
||||
|
|
@ -659,7 +680,7 @@ def test_route_project_chat_1to1_delivery_is_idempotent(tmp_path, monkeypatch):
|
|||
|
||||
msg_ids = []
|
||||
monkeypatch.setattr(omb, "write_owner_message",
|
||||
lambda drive, text, tid, msg_id=None, **k: msg_ids.append(msg_id))
|
||||
lambda drive, text, tid, msg_id=None, **k: msg_ids.append(msg_id) or True)
|
||||
|
||||
ctx = types.SimpleNamespace(
|
||||
DRIVE_ROOT=tmp_path,
|
||||
|
|
@ -671,6 +692,33 @@ def test_route_project_chat_1to1_delivery_is_idempotent(tmp_path, monkeypatch):
|
|||
assert msg_ids == ["cmid-7:pr", "cmid-7:pr"]
|
||||
|
||||
|
||||
def test_route_project_chat_does_not_confirm_failed_mailbox_write(tmp_path, monkeypatch):
|
||||
import types
|
||||
|
||||
import ouroboros.owner_mailbox as omb
|
||||
import server
|
||||
from ouroboros.projects_registry import create_project
|
||||
|
||||
project_chat = int(create_project(tmp_path, "racer")["chat_id"])
|
||||
monkeypatch.setattr(omb, "write_owner_message", lambda *_a, **_k: False)
|
||||
ctx = types.SimpleNamespace(
|
||||
DRIVE_ROOT=tmp_path,
|
||||
RUNNING={
|
||||
"pr": {
|
||||
"task": {"id": "pr", "chat_id": project_chat},
|
||||
"last_heartbeat_at": 1.0,
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
assert (
|
||||
server._route_project_chat_to_running_task(
|
||||
ctx, project_chat, "must be durable", "owner-msg"
|
||||
)
|
||||
== ""
|
||||
)
|
||||
|
||||
|
||||
def test_busy_project_chat_routes_to_ephemeral_decision_turn(tmp_path, monkeypatch):
|
||||
"""WS1/P5 (v6.34.0): a busy PROJECT chat is NOT mechanically auto-enqueued into a
|
||||
duplicate pooled task. It runs the ephemeral decision turn (project-scoped, seeing
|
||||
|
|
@ -1127,10 +1175,11 @@ def test_steer_task_tool_emits_event_with_target_and_client_id(tmp_path):
|
|||
events = []
|
||||
ctx = types.SimpleNamespace(
|
||||
pending_events=events, event_queue=None, current_chat_id=1,
|
||||
drive_root=tmp_path,
|
||||
task_metadata={"client_message_id": "cm-42"},
|
||||
)
|
||||
out = _steer_task(ctx, "abc12345", "also add the benchmarks slide")
|
||||
assert out.startswith("✉️ Steering task abc12345")
|
||||
assert out.startswith("⚠️ STEER_UNCONFIRMED")
|
||||
assert len(events) == 1
|
||||
evt = events[0]
|
||||
assert evt["type"] == "steer_task"
|
||||
|
|
@ -1154,6 +1203,7 @@ def test_main_steer_can_address_project_bound_root_from_host_manifest(tmp_path,
|
|||
pending_events=emitted,
|
||||
event_queue=None,
|
||||
current_chat_id=1,
|
||||
drive_root=tmp_path,
|
||||
task_metadata={
|
||||
"client_message_id": "main-42",
|
||||
"routing_contract": {"source_lane": "main"},
|
||||
|
|
@ -1210,6 +1260,7 @@ def test_busy_direct_main_root_is_manifested_and_steerable_without_promotion(tmp
|
|||
pending_events=emitted,
|
||||
event_queue=None,
|
||||
current_chat_id=1,
|
||||
drive_root=tmp_path,
|
||||
task_metadata={
|
||||
"client_message_id": "followup-1",
|
||||
"routing_contract": metadata["routing_contract"],
|
||||
|
|
|
|||
848
tests/test_promote_event_transport.py
Normal file
848
tests/test_promote_event_transport.py
Normal file
|
|
@ -0,0 +1,848 @@
|
|||
"""Cross-process transport and durable admission regressions for chat promotion."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import multiprocessing as mp
|
||||
import threading
|
||||
import types
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
pytestmark = pytest.mark.serial
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _isolate_event_bus_shutdown_latch():
|
||||
"""A prior TestClient lifespan must not leak its shutdown latch into tests."""
|
||||
import supervisor.workers as workers
|
||||
|
||||
workers._EVENT_Q_SHUTDOWN = False
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
workers._EVENT_Q_SHUTDOWN = False
|
||||
|
||||
|
||||
def _child_put(queue, payload):
|
||||
queue.put(payload)
|
||||
|
||||
|
||||
def test_manager_event_bus_accepts_real_spawn_child_after_generation_setup(
|
||||
monkeypatch, tmp_path,
|
||||
):
|
||||
import supervisor.workers as workers
|
||||
|
||||
monkeypatch.setattr(workers, "DRIVE_ROOT", tmp_path)
|
||||
monkeypatch.setattr(workers, "_CTX", None)
|
||||
monkeypatch.setattr(workers, "_EVENT_Q", None)
|
||||
monkeypatch.setattr(workers, "_EVENT_Q_MANAGER", None)
|
||||
monkeypatch.setattr(workers, "_EVENT_Q_GENERATION", "")
|
||||
monkeypatch.setattr(workers, "_EVENT_Q_SHUTDOWN", False)
|
||||
queue = workers.get_event_q()
|
||||
manager = workers._EVENT_Q_MANAGER
|
||||
ctx = mp.get_context(workers._WORKER_START_METHOD)
|
||||
child = ctx.Process(target=_child_put, args=(queue, {"type": "probe", "value": 7}))
|
||||
try:
|
||||
child.start()
|
||||
child.join(10)
|
||||
assert child.exitcode == 0
|
||||
assert queue.get(timeout=2) == {"type": "probe", "value": 7}
|
||||
ledger = [
|
||||
json.loads(line)
|
||||
for line in (tmp_path / "state" / "process_ledger.jsonl")
|
||||
.read_text(encoding="utf-8")
|
||||
.splitlines()
|
||||
]
|
||||
assert ledger[-1]["pid"] == manager._process.pid
|
||||
assert ledger[-1]["purpose"] == "supervisor_event_queue_manager"
|
||||
assert ledger[-1]["scope"] == "session"
|
||||
assert ledger[-1]["pgid"] == 0
|
||||
finally:
|
||||
if child.is_alive():
|
||||
child.terminate()
|
||||
child.join(2)
|
||||
workers.shutdown_event_q()
|
||||
assert not manager._process.is_alive()
|
||||
|
||||
|
||||
def test_concurrent_pool_start_cannot_orphan_a_generation(monkeypatch, tmp_path):
|
||||
import supervisor.workers as workers
|
||||
|
||||
event_q = object()
|
||||
fake_ctx = MagicMock()
|
||||
fake_ctx.Queue.return_value = object()
|
||||
created = []
|
||||
|
||||
def make_process(*_args, **_kwargs):
|
||||
proc = MagicMock(pid=1000 + len(created))
|
||||
created.append(proc)
|
||||
return proc
|
||||
|
||||
fake_ctx.Process.side_effect = make_process
|
||||
monkeypatch.setattr(workers, "DRIVE_ROOT", tmp_path)
|
||||
monkeypatch.setattr(workers, "_CTX", fake_ctx)
|
||||
monkeypatch.setattr(workers.mp, "get_context", lambda _method: fake_ctx)
|
||||
monkeypatch.setattr(workers, "_EVENT_Q", event_q)
|
||||
monkeypatch.setattr(workers, "_EVENT_Q_GENERATION", "test-generation")
|
||||
monkeypatch.setattr(workers, "WORKERS", {})
|
||||
monkeypatch.setattr(workers, "reap_orphaned_workers", lambda: 0)
|
||||
monkeypatch.setattr(workers, "_record_worker_pids", lambda: None)
|
||||
monkeypatch.setattr(workers, "_verify_worker_sha_after_spawn", lambda *_args: None)
|
||||
barrier = threading.Barrier(2)
|
||||
|
||||
def start():
|
||||
barrier.wait()
|
||||
try:
|
||||
workers.spawn_workers(1)
|
||||
return "started"
|
||||
except RuntimeError:
|
||||
return "refused"
|
||||
|
||||
with ThreadPoolExecutor(max_workers=2) as executor:
|
||||
futures = [executor.submit(start) for _ in range(2)]
|
||||
outcomes = sorted(future.result() for future in futures)
|
||||
assert outcomes == ["refused", "started"]
|
||||
assert len(created) == 1
|
||||
assert len(workers.WORKERS) == 1
|
||||
|
||||
|
||||
def test_single_slot_respawn_starts_child_without_queue_lock(monkeypatch, tmp_path):
|
||||
import supervisor.workers as workers
|
||||
|
||||
lock_was_free = []
|
||||
|
||||
class ProbeProcess:
|
||||
pid = 4321
|
||||
daemon = False
|
||||
|
||||
def start(self):
|
||||
def probe():
|
||||
with workers._queue_lock:
|
||||
lock_was_free.append(True)
|
||||
|
||||
thread = threading.Thread(target=probe)
|
||||
thread.start()
|
||||
thread.join(1)
|
||||
assert not thread.is_alive(), "proc.start() ran while queue lock was held"
|
||||
|
||||
def is_alive(self):
|
||||
return True
|
||||
|
||||
fake_ctx = MagicMock()
|
||||
fake_ctx.Queue.return_value = MagicMock()
|
||||
fake_ctx.Process.return_value = ProbeProcess()
|
||||
old = workers.Worker(
|
||||
wid=0,
|
||||
proc=MagicMock(pid=111),
|
||||
in_q=MagicMock(),
|
||||
busy_task_id=None,
|
||||
reaping=True,
|
||||
)
|
||||
monkeypatch.setattr(workers, "DRIVE_ROOT", tmp_path)
|
||||
monkeypatch.setattr(workers, "WORKERS", {0: old})
|
||||
monkeypatch.setattr(workers, "_get_ctx", lambda: fake_ctx)
|
||||
monkeypatch.setattr(workers, "get_event_q", lambda: object())
|
||||
monkeypatch.setattr(workers, "_record_worker_pids", lambda: None)
|
||||
|
||||
assert workers.respawn_worker(0) is True
|
||||
assert lock_was_free == [True]
|
||||
assert workers.WORKERS[0] is not old
|
||||
assert workers.WORKERS[0].reaping is False
|
||||
|
||||
|
||||
def test_worker_pool_respawn_reuses_process_event_bus_and_refuses_live_pool(monkeypatch, tmp_path):
|
||||
import supervisor.workers as workers
|
||||
|
||||
event_q = object()
|
||||
fake_ctx = MagicMock()
|
||||
fake_ctx.Queue.side_effect = [object(), object()]
|
||||
processes = [MagicMock(pid=101), MagicMock(pid=102)]
|
||||
fake_ctx.Process.side_effect = processes
|
||||
monkeypatch.setattr(workers, "DRIVE_ROOT", tmp_path)
|
||||
monkeypatch.setattr(workers, "_CTX", fake_ctx)
|
||||
monkeypatch.setattr(workers.mp, "get_context", lambda _method: fake_ctx)
|
||||
monkeypatch.setattr(workers, "_EVENT_Q", event_q)
|
||||
monkeypatch.setattr(workers, "_EVENT_Q_GENERATION", "test-generation")
|
||||
monkeypatch.setattr(workers, "_WORKER_POOL_DISABLED_REASON", "worker_crash_storm")
|
||||
monkeypatch.setattr(workers, "WORKERS", {})
|
||||
monkeypatch.setattr(workers, "reap_orphaned_workers", lambda: 0)
|
||||
monkeypatch.setattr(workers, "_record_worker_pids", lambda: None)
|
||||
monkeypatch.setattr(workers.threading, "Thread", MagicMock())
|
||||
|
||||
workers.spawn_workers(1)
|
||||
assert fake_ctx.Process.call_args_list[0].kwargs["args"][2] is event_q
|
||||
assert workers._EVENT_Q is event_q
|
||||
assert workers._WORKER_POOL_DISABLED_REASON == ""
|
||||
|
||||
try:
|
||||
workers.spawn_workers(1)
|
||||
except RuntimeError as exc:
|
||||
assert "requires an empty pool" in str(exc)
|
||||
else:
|
||||
raise AssertionError("spawn_workers replaced a live pool")
|
||||
assert fake_ctx.Process.call_count == 1
|
||||
|
||||
workers.WORKERS.clear()
|
||||
workers.spawn_workers(1)
|
||||
assert fake_ctx.Process.call_args_list[1].kwargs["args"][2] is event_q
|
||||
assert workers._EVENT_Q is event_q
|
||||
|
||||
|
||||
def test_promote_rejects_before_side_effects_when_pool_disabled(tmp_path):
|
||||
from ouroboros.task_results import load_task_result
|
||||
from supervisor.events import _handle_promote_chat_to_task
|
||||
|
||||
rows = []
|
||||
ctx = types.SimpleNamespace(
|
||||
DRIVE_ROOT=tmp_path,
|
||||
WORKERS={},
|
||||
PENDING=[],
|
||||
bridge=None,
|
||||
append_jsonl=lambda _path, row: rows.append(row),
|
||||
persist_queue_snapshot=lambda **_kwargs: True,
|
||||
enqueue_task=lambda _task: (_ for _ in ()).throw(AssertionError("must not enqueue")),
|
||||
load_state=lambda: {"owner_chat_id": 1},
|
||||
)
|
||||
|
||||
outcome = _handle_promote_chat_to_task(
|
||||
{
|
||||
"type": "promote_chat_to_task",
|
||||
"task_id": "nowork01",
|
||||
"routing_token": "nowork-token",
|
||||
"objective": "Build it",
|
||||
"project_id": "must-not-exist",
|
||||
"project_name": "Must Not Exist",
|
||||
"chat_id": 1,
|
||||
},
|
||||
ctx,
|
||||
)
|
||||
|
||||
assert outcome["reason"] == "worker_pool_unavailable"
|
||||
assert not (tmp_path / "state" / "projects.json").exists()
|
||||
stored = load_task_result(tmp_path, "nowork01")
|
||||
assert stored["status"] == "failed"
|
||||
assert stored["promotion_admission"]["status"] == "rejected"
|
||||
assert stored["promotion_admission"]["reason"] == "worker_pool_unavailable"
|
||||
assert any(row["type"] == "promote_chat_to_task_rejected" for row in rows)
|
||||
|
||||
|
||||
def test_tool_snapshot_precheck_skips_source_side_effects(monkeypatch, tmp_path):
|
||||
from ouroboros.tools import control
|
||||
|
||||
state = tmp_path / "state"
|
||||
state.mkdir(parents=True)
|
||||
(state / "queue_snapshot.json").write_text(
|
||||
json.dumps({"worker_pool_disabled_reason": "worker_crash_storm"}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
ctx = types.SimpleNamespace(
|
||||
event_queue=None,
|
||||
pending_events=[],
|
||||
current_chat_id=1,
|
||||
drive_root=tmp_path,
|
||||
)
|
||||
out = control._promote_chat_to_task(
|
||||
ctx,
|
||||
"Build",
|
||||
project_name="Must Not Exist",
|
||||
source="/tmp/must-not-attach",
|
||||
)
|
||||
assert out.startswith("PROMOTE_REJECTED:")
|
||||
assert "worker_crash_storm" in out
|
||||
assert ctx.pending_events == []
|
||||
|
||||
|
||||
def test_busy_or_reaping_workers_still_allow_queue_admission():
|
||||
from supervisor.workers import worker_pool_admission_state
|
||||
|
||||
ctx = types.SimpleNamespace(
|
||||
WORKERS={0: types.SimpleNamespace(busy_task_id="other", reaping=True)},
|
||||
)
|
||||
assert worker_pool_admission_state(ctx)["available"] is True
|
||||
|
||||
|
||||
def test_real_event_queue_reaches_dispatch_and_confirms_durable_admission(
|
||||
monkeypatch, tmp_path,
|
||||
):
|
||||
import supervisor.workers as workers
|
||||
from ouroboros.task_results import load_task_result
|
||||
from ouroboros.tools import control
|
||||
from supervisor.events import _handle_promote_chat_to_task
|
||||
|
||||
monkeypatch.setattr(workers, "DRIVE_ROOT", tmp_path)
|
||||
monkeypatch.setattr(control, "_PROMOTE_CONFIRM_TIMEOUT_SEC", 2.0)
|
||||
monkeypatch.setattr(control, "_PROMOTE_CONFIRM_POLL_SEC", 0.01)
|
||||
pending = []
|
||||
|
||||
def enqueue(task):
|
||||
pending.append(dict(task))
|
||||
return task
|
||||
|
||||
def persist_snapshot(**_kwargs):
|
||||
path = tmp_path / "state" / "queue_snapshot.json"
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(
|
||||
json.dumps({"pending": pending, "pending_count": len(pending)}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
return True
|
||||
|
||||
handler_ctx = types.SimpleNamespace(
|
||||
DRIVE_ROOT=tmp_path,
|
||||
WORKERS={0: types.SimpleNamespace(busy_task_id=None, reaping=False)},
|
||||
PENDING=pending,
|
||||
bridge=None,
|
||||
append_jsonl=lambda *_args, **_kwargs: None,
|
||||
persist_queue_snapshot=persist_snapshot,
|
||||
enqueue_task=enqueue,
|
||||
load_state=lambda: {"owner_chat_id": 1},
|
||||
)
|
||||
queue = mp.get_context("spawn").Queue()
|
||||
tool_ctx = types.SimpleNamespace(
|
||||
event_queue=queue,
|
||||
pending_events=[],
|
||||
current_chat_id=1,
|
||||
drive_root=tmp_path,
|
||||
task_metadata={},
|
||||
)
|
||||
try:
|
||||
with ThreadPoolExecutor(max_workers=1) as executor:
|
||||
future = executor.submit(control._promote_chat_to_task, tool_ctx, "Build the racer")
|
||||
event = queue.get(timeout=2)
|
||||
assert event["type"] == "promote_chat_to_task"
|
||||
outcome = _handle_promote_chat_to_task(event, handler_ctx)
|
||||
result_text = future.result(timeout=3)
|
||||
task_id = event["task_id"]
|
||||
assert outcome == {"status": "scheduled", "task_id": task_id}
|
||||
assert result_text.startswith(f"OK: task {task_id}")
|
||||
assert "accepted and durably scheduled" in result_text
|
||||
assert any(task["id"] == task_id for task in pending)
|
||||
snapshot = json.loads((tmp_path / "state" / "queue_snapshot.json").read_text())
|
||||
assert snapshot["pending_count"] == 1
|
||||
stored = load_task_result(tmp_path, task_id)
|
||||
assert stored["status"] == "scheduled"
|
||||
assert stored["promotion_admission"] == {
|
||||
"status": "scheduled",
|
||||
"routing_token": event["routing_token"],
|
||||
"confirmed_at": stored["promotion_admission"]["confirmed_at"],
|
||||
"queue_snapshot_persisted": True,
|
||||
"source_note": "",
|
||||
"reason": "",
|
||||
"routing_receipt_required": False,
|
||||
"routing_receipt_status": "not_applicable",
|
||||
}
|
||||
finally:
|
||||
queue.close()
|
||||
queue.cancel_join_thread()
|
||||
|
||||
|
||||
def test_route_to_project_waits_for_same_durable_admission(monkeypatch, tmp_path):
|
||||
import supervisor.workers as workers
|
||||
from ouroboros.projects_registry import create_project
|
||||
from ouroboros.project_dialogue import chat_annotation_receipt
|
||||
from ouroboros.tools import control
|
||||
from supervisor.events import _handle_promote_chat_to_task
|
||||
|
||||
monkeypatch.setattr(workers, "DRIVE_ROOT", tmp_path)
|
||||
monkeypatch.setattr(control, "_PROMOTE_CONFIRM_TIMEOUT_SEC", 2.0)
|
||||
monkeypatch.setattr(control, "_PROMOTE_CONFIRM_POLL_SEC", 0.01)
|
||||
create_project(tmp_path, "racer", name="Racer")
|
||||
pending = []
|
||||
|
||||
def enqueue(task):
|
||||
pending.append(dict(task))
|
||||
return task
|
||||
|
||||
handler_ctx = types.SimpleNamespace(
|
||||
DRIVE_ROOT=tmp_path,
|
||||
WORKERS={0: types.SimpleNamespace()},
|
||||
PENDING=pending,
|
||||
bridge=None,
|
||||
append_jsonl=lambda *_args, **_kwargs: None,
|
||||
persist_queue_snapshot=lambda **_kwargs: True,
|
||||
enqueue_task=enqueue,
|
||||
load_state=lambda: {"owner_chat_id": 1},
|
||||
)
|
||||
queue = mp.get_context("spawn").Queue()
|
||||
tool_ctx = types.SimpleNamespace(
|
||||
event_queue=queue,
|
||||
pending_events=[],
|
||||
current_chat_id=1,
|
||||
drive_root=tmp_path,
|
||||
task_metadata={"client_message_id": "route-owner-1"},
|
||||
)
|
||||
try:
|
||||
with ThreadPoolExecutor(max_workers=1) as executor:
|
||||
future = executor.submit(
|
||||
control._route_to_project,
|
||||
tool_ctx,
|
||||
"racer",
|
||||
"Continue the racer",
|
||||
"belongs there",
|
||||
)
|
||||
event = queue.get(timeout=2)
|
||||
outcome = _handle_promote_chat_to_task(event, handler_ctx)
|
||||
text = future.result(timeout=3)
|
||||
assert outcome["status"] == "scheduled"
|
||||
assert text.startswith("✉️ Routed to project 'Racer'")
|
||||
assert "durably scheduled" in text
|
||||
receipt = chat_annotation_receipt(
|
||||
tmp_path, "route-owner-1", event["routing_token"]
|
||||
)
|
||||
assert receipt["action"] == "route_to_project"
|
||||
assert receipt["status"] == "scheduled"
|
||||
finally:
|
||||
queue.close()
|
||||
queue.cancel_join_thread()
|
||||
|
||||
|
||||
def test_manual_target_tool_waits_for_durable_handler_receipt(monkeypatch, tmp_path):
|
||||
from ouroboros.tools import control
|
||||
from supervisor.events import _handle_routing_manual_target
|
||||
|
||||
monkeypatch.setattr(control, "_PROMOTE_CONFIRM_TIMEOUT_SEC", 2.0)
|
||||
monkeypatch.setattr(control, "_PROMOTE_CONFIRM_POLL_SEC", 0.01)
|
||||
queue = mp.get_context("spawn").Queue()
|
||||
tool_ctx = types.SimpleNamespace(
|
||||
event_queue=queue,
|
||||
pending_events=[],
|
||||
current_chat_id=1,
|
||||
drive_root=tmp_path,
|
||||
task_metadata={
|
||||
"client_message_id": "manual-owner-1",
|
||||
"routing_contract": {"manual_options": [{"kind": "new_task"}]},
|
||||
},
|
||||
)
|
||||
handler_ctx = types.SimpleNamespace(DRIVE_ROOT=tmp_path, bridge=None)
|
||||
try:
|
||||
with ThreadPoolExecutor(max_workers=1) as executor:
|
||||
future = executor.submit(
|
||||
control._route_to_project,
|
||||
tool_ctx,
|
||||
"missing-project",
|
||||
"Continue",
|
||||
"uncertain target",
|
||||
)
|
||||
event = queue.get(timeout=2)
|
||||
_handle_routing_manual_target(event, handler_ctx)
|
||||
text = future.result(timeout=3)
|
||||
assert text.startswith("⚠️ NEEDS_MANUAL_TARGET")
|
||||
assert 'Host-validated options: [{"kind": "new_task"}]' in text
|
||||
finally:
|
||||
queue.close()
|
||||
queue.cancel_join_thread()
|
||||
|
||||
|
||||
def test_steer_tool_reports_delivery_only_after_mailbox_receipt(
|
||||
monkeypatch, tmp_path,
|
||||
):
|
||||
import supervisor.queue as supervisor_queue
|
||||
from ouroboros.owner_mailbox import drain_owner_messages
|
||||
from ouroboros.tools import control
|
||||
from supervisor.events import _handle_steer_task
|
||||
|
||||
monkeypatch.setattr(supervisor_queue, "DRIVE_ROOT", tmp_path)
|
||||
monkeypatch.setattr(control, "_PROMOTE_CONFIRM_TIMEOUT_SEC", 2.0)
|
||||
monkeypatch.setattr(control, "_PROMOTE_CONFIRM_POLL_SEC", 0.01)
|
||||
target = {"id": "target01", "chat_id": 1}
|
||||
queue = mp.get_context("spawn").Queue()
|
||||
tool_ctx = types.SimpleNamespace(
|
||||
event_queue=queue,
|
||||
pending_events=[],
|
||||
current_chat_id=1,
|
||||
drive_root=tmp_path,
|
||||
task_metadata={"client_message_id": "steer-owner-1"},
|
||||
)
|
||||
handler_ctx = types.SimpleNamespace(
|
||||
DRIVE_ROOT=tmp_path,
|
||||
RUNNING={},
|
||||
PENDING=[target],
|
||||
bridge=None,
|
||||
get_chat_agent=lambda: None,
|
||||
persist_queue_snapshot=lambda **_kwargs: True,
|
||||
)
|
||||
try:
|
||||
with ThreadPoolExecutor(max_workers=1) as executor:
|
||||
future = executor.submit(control._steer_task, tool_ctx, "target01", "Use the new data")
|
||||
event = queue.get(timeout=2)
|
||||
_handle_steer_task(event, handler_ctx)
|
||||
text = future.result(timeout=3)
|
||||
assert text.startswith("✉️ Steering task target01")
|
||||
assert "durably confirmed" in text
|
||||
assert drain_owner_messages(tmp_path, "target01") == ["Use the new data"]
|
||||
finally:
|
||||
queue.close()
|
||||
queue.cancel_join_thread()
|
||||
|
||||
|
||||
def test_stale_live_transport_returns_unconfirmed_not_ok(monkeypatch, tmp_path):
|
||||
from ouroboros.tools import control
|
||||
|
||||
monkeypatch.setattr(control, "_PROMOTE_CONFIRM_TIMEOUT_SEC", 0.05)
|
||||
monkeypatch.setattr(control, "_PROMOTE_CONFIRM_POLL_SEC", 0.005)
|
||||
stale_queue = mp.get_context("spawn").Queue()
|
||||
ctx = types.SimpleNamespace(
|
||||
event_queue=stale_queue,
|
||||
pending_events=[],
|
||||
current_chat_id=1,
|
||||
drive_root=tmp_path,
|
||||
task_metadata={},
|
||||
)
|
||||
try:
|
||||
out = control._promote_chat_to_task(ctx, "Never drained")
|
||||
event = stale_queue.get(timeout=2)
|
||||
assert event["type"] == "promote_chat_to_task"
|
||||
assert out.startswith("PROMOTE_UNCONFIRMED:")
|
||||
assert "Do not report this task as created" in out
|
||||
assert not (tmp_path / "task_results" / f"{event['task_id']}.json").exists()
|
||||
finally:
|
||||
stale_queue.close()
|
||||
stale_queue.cancel_join_thread()
|
||||
|
||||
|
||||
def test_stale_task_result_and_receipt_cannot_confirm_new_admission_token(tmp_path):
|
||||
from ouroboros.task_results import STATUS_SCHEDULED, write_task_result
|
||||
from ouroboros.tools.control import _wait_for_promotion_admission
|
||||
|
||||
old_token = "a" * 32
|
||||
new_token = "b" * 32
|
||||
write_task_result(
|
||||
tmp_path,
|
||||
"deadbeef",
|
||||
STATUS_SCHEDULED,
|
||||
promotion_admission={"status": "scheduled", "routing_token": old_token},
|
||||
)
|
||||
ctx = types.SimpleNamespace(drive_root=tmp_path)
|
||||
assert _wait_for_promotion_admission(
|
||||
ctx, "deadbeef", new_token, timeout_sec=0.0,
|
||||
) == {
|
||||
"status": "unconfirmed",
|
||||
"reason": "confirmation_timeout",
|
||||
}
|
||||
|
||||
|
||||
def test_unpicklable_control_event_fails_before_feeder_thread(tmp_path):
|
||||
from ouroboros.tools.control import _emit_control_event
|
||||
|
||||
ctx = types.SimpleNamespace(
|
||||
event_queue=mp.get_context("spawn").Queue(),
|
||||
pending_events=[],
|
||||
drive_root=tmp_path,
|
||||
)
|
||||
try:
|
||||
mode = _emit_control_event(
|
||||
ctx,
|
||||
{"type": "promote_chat_to_task", "task_id": "pickle01", "bad": lambda: None},
|
||||
)
|
||||
assert mode == "serialization_failed"
|
||||
assert ctx.pending_events == []
|
||||
rows = [
|
||||
json.loads(line)
|
||||
for line in (tmp_path / "logs" / "supervisor.jsonl").read_text().splitlines()
|
||||
]
|
||||
assert rows[-1]["type"] == "control_event_serialization_failed"
|
||||
assert rows[-1]["task_id"] == "pickle01"
|
||||
finally:
|
||||
ctx.event_queue.close()
|
||||
ctx.event_queue.cancel_join_thread()
|
||||
|
||||
|
||||
def test_snapshot_persistence_failure_rolls_back_pending(monkeypatch, tmp_path):
|
||||
import supervisor.workers as workers
|
||||
from ouroboros.task_results import load_task_result
|
||||
from supervisor.events import _handle_promote_chat_to_task
|
||||
|
||||
monkeypatch.setattr(workers, "DRIVE_ROOT", tmp_path)
|
||||
pending = []
|
||||
|
||||
def enqueue(task):
|
||||
pending.append(dict(task))
|
||||
return task
|
||||
|
||||
ctx = types.SimpleNamespace(
|
||||
DRIVE_ROOT=tmp_path,
|
||||
WORKERS={0: types.SimpleNamespace()},
|
||||
PENDING=pending,
|
||||
bridge=None,
|
||||
enqueue_task=enqueue,
|
||||
persist_queue_snapshot=lambda **_kwargs: False,
|
||||
load_state=lambda: {"owner_chat_id": 1},
|
||||
append_jsonl=lambda *_args, **_kwargs: None,
|
||||
)
|
||||
outcome = _handle_promote_chat_to_task(
|
||||
{
|
||||
"type": "promote_chat_to_task",
|
||||
"task_id": "snapfail",
|
||||
"routing_token": "snapfail-token",
|
||||
"objective": "Build",
|
||||
},
|
||||
ctx,
|
||||
)
|
||||
assert outcome["reason"] == "queue_snapshot_persist_failed"
|
||||
assert pending == []
|
||||
stored = load_task_result(tmp_path, "snapfail")
|
||||
assert stored["promotion_admission"]["status"] == "rejected"
|
||||
|
||||
|
||||
def test_missing_snapshot_persister_fails_closed(monkeypatch, tmp_path):
|
||||
import supervisor.workers as workers
|
||||
|
||||
monkeypatch.setattr(workers, "DRIVE_ROOT", tmp_path)
|
||||
pending = []
|
||||
ctx = types.SimpleNamespace(
|
||||
enqueue_task=lambda task: pending.append(dict(task)) or task,
|
||||
load_state=lambda: {"owner_chat_id": 1},
|
||||
)
|
||||
outcome = workers.promote_chat_to_task(
|
||||
{"task_id": "no-persister", "objective": "Build"},
|
||||
ctx,
|
||||
)
|
||||
assert outcome["status"] == "needs_manual_target"
|
||||
assert outcome["reason"] == "queue_snapshot_persist_unavailable"
|
||||
assert len(pending) == 1
|
||||
|
||||
|
||||
def test_routing_receipt_failure_cannot_produce_positive_confirmation(
|
||||
monkeypatch, tmp_path,
|
||||
):
|
||||
import supervisor.workers as workers
|
||||
from ouroboros.task_results import load_task_result
|
||||
from supervisor.events import _handle_promote_chat_to_task
|
||||
|
||||
monkeypatch.setattr(workers, "DRIVE_ROOT", tmp_path)
|
||||
monkeypatch.setattr(
|
||||
"ouroboros.project_dialogue.append_chat_annotation",
|
||||
lambda *_args, **_kwargs: False,
|
||||
)
|
||||
pending = []
|
||||
|
||||
def enqueue(task):
|
||||
pending.append(dict(task))
|
||||
return task
|
||||
|
||||
ctx = types.SimpleNamespace(
|
||||
DRIVE_ROOT=tmp_path,
|
||||
WORKERS={0: types.SimpleNamespace()},
|
||||
PENDING=pending,
|
||||
bridge=None,
|
||||
enqueue_task=enqueue,
|
||||
persist_queue_snapshot=lambda **_kwargs: True,
|
||||
load_state=lambda: {"owner_chat_id": 1},
|
||||
append_jsonl=lambda *_args, **_kwargs: None,
|
||||
)
|
||||
outcome = _handle_promote_chat_to_task(
|
||||
{
|
||||
"type": "promote_chat_to_task",
|
||||
"task_id": "receiptfail",
|
||||
"routing_token": "receiptfail-token",
|
||||
"objective": "Build",
|
||||
"client_message_id": "owner-receipt-fail",
|
||||
"chat_id": 1,
|
||||
},
|
||||
ctx,
|
||||
)
|
||||
assert outcome["reason"] == "routing_annotation_persist_failed"
|
||||
assert outcome["status"] == "unconfirmed"
|
||||
assert len(pending) == 1
|
||||
stored = load_task_result(tmp_path, "receiptfail")
|
||||
assert stored["promotion_admission"]["status"] == "unconfirmed"
|
||||
|
||||
|
||||
def test_gateway_refuses_explicitly_disabled_pool_before_task_side_effects(
|
||||
monkeypatch, tmp_path,
|
||||
):
|
||||
import supervisor.workers as workers
|
||||
from ouroboros.gateway.tasks import _supervisor_ready_error
|
||||
|
||||
ready = threading.Event()
|
||||
ready.set()
|
||||
request = types.SimpleNamespace(
|
||||
app=types.SimpleNamespace(
|
||||
state=types.SimpleNamespace(supervisor_ready_event=ready),
|
||||
)
|
||||
)
|
||||
monkeypatch.setattr(workers, "WORKERS", {})
|
||||
monkeypatch.setattr(workers, "_WORKER_POOL_DISABLED_REASON", "worker_crash_storm")
|
||||
response = _supervisor_ready_error(request)
|
||||
payload = json.loads(response.body)
|
||||
assert response.status_code == 503
|
||||
assert payload["reason_code"] == "worker_pool_unavailable"
|
||||
assert payload["worker_pool_disabled_reason"] == "worker_crash_storm"
|
||||
assert not (tmp_path / "task_results").exists()
|
||||
|
||||
|
||||
def test_admission_reservation_rejects_tokenless_competing_enqueue(
|
||||
monkeypatch, tmp_path,
|
||||
):
|
||||
import supervisor.queue as supervisor_queue
|
||||
|
||||
pending = []
|
||||
monkeypatch.setattr(supervisor_queue, "DRIVE_ROOT", tmp_path)
|
||||
monkeypatch.setattr(supervisor_queue, "PENDING", pending)
|
||||
monkeypatch.setattr(supervisor_queue, "RUNNING", {})
|
||||
monkeypatch.setattr(supervisor_queue, "ADMISSION_RESERVATIONS", {})
|
||||
assert supervisor_queue.reserve_task_admission(
|
||||
"owned-task", "owner-token", drive_root=tmp_path
|
||||
)["status"] == "reserved"
|
||||
|
||||
loser = supervisor_queue.enqueue_task({"id": "owned-task", "type": "task"})
|
||||
|
||||
assert loser["_admission_blocked"] == "admission_reservation_owned"
|
||||
assert pending == []
|
||||
assert supervisor_queue.ADMISSION_RESERVATIONS == {
|
||||
"owned-task": "owner-token"
|
||||
}
|
||||
|
||||
|
||||
def test_project_registry_lookup_failure_prevents_clone(monkeypatch, tmp_path):
|
||||
from ouroboros.promotion_source import resolve_promote_source
|
||||
|
||||
monkeypatch.setattr(
|
||||
"ouroboros.projects_registry.get_reserved_project",
|
||||
lambda *_args, **_kwargs: (_ for _ in ()).throw(OSError("registry unreadable")),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"ouroboros.project_sources.clone_project_repo",
|
||||
lambda *_args, **_kwargs: (_ for _ in ()).throw(
|
||||
AssertionError("clone must not start")
|
||||
),
|
||||
)
|
||||
ctx = types.SimpleNamespace(DRIVE_ROOT=tmp_path, REPO_DIR=tmp_path / "repo")
|
||||
|
||||
folder, note, error, project_id = resolve_promote_source(
|
||||
ctx, "https://github.com/example/project.git", "project"
|
||||
)
|
||||
|
||||
assert folder == ""
|
||||
assert note == ""
|
||||
assert error.startswith("project_lookup_failed: OSError: registry unreadable")
|
||||
assert project_id == "project"
|
||||
|
||||
|
||||
def test_duplicate_promote_uses_negative_annotation_without_overwriting_result(
|
||||
monkeypatch, tmp_path,
|
||||
):
|
||||
import supervisor.queue as supervisor_queue
|
||||
import supervisor.workers as workers
|
||||
from ouroboros.task_results import STATUS_SCHEDULED, load_task_result, write_task_result
|
||||
from ouroboros.tools.control import _wait_for_promotion_admission
|
||||
from supervisor.events import _handle_promote_chat_to_task
|
||||
|
||||
old_token = "old-token"
|
||||
new_token = "new-token"
|
||||
write_task_result(
|
||||
tmp_path,
|
||||
"duplicate-task",
|
||||
STATUS_SCHEDULED,
|
||||
promotion_admission={"status": "scheduled", "routing_token": old_token},
|
||||
)
|
||||
monkeypatch.setattr(workers, "DRIVE_ROOT", tmp_path)
|
||||
monkeypatch.setattr(supervisor_queue, "DRIVE_ROOT", tmp_path)
|
||||
monkeypatch.setattr(supervisor_queue, "PENDING", [])
|
||||
monkeypatch.setattr(supervisor_queue, "RUNNING", {})
|
||||
monkeypatch.setattr(supervisor_queue, "ADMISSION_RESERVATIONS", {})
|
||||
rows = []
|
||||
ctx = types.SimpleNamespace(
|
||||
DRIVE_ROOT=tmp_path,
|
||||
WORKERS={0: types.SimpleNamespace()},
|
||||
bridge=None,
|
||||
append_jsonl=lambda _path, row: rows.append(row),
|
||||
)
|
||||
outcome = _handle_promote_chat_to_task(
|
||||
{
|
||||
"type": "promote_chat_to_task",
|
||||
"task_id": "duplicate-task",
|
||||
"routing_token": new_token,
|
||||
"objective": "Competing request",
|
||||
"chat_id": 1,
|
||||
"client_message_id": "duplicate-owner-message",
|
||||
},
|
||||
ctx,
|
||||
)
|
||||
|
||||
assert outcome["reason"] == "duplicate_task_id"
|
||||
confirmation = _wait_for_promotion_admission(
|
||||
types.SimpleNamespace(drive_root=tmp_path),
|
||||
"duplicate-task",
|
||||
new_token,
|
||||
client_message_id="duplicate-owner-message",
|
||||
timeout_sec=0.0,
|
||||
)
|
||||
assert confirmation["status"] == "needs_manual_target"
|
||||
assert confirmation["reason"] == "duplicate_task_id"
|
||||
assert load_task_result(tmp_path, "duplicate-task")["promotion_admission"] == {
|
||||
"status": "scheduled",
|
||||
"routing_token": old_token,
|
||||
}
|
||||
|
||||
|
||||
def test_source_resolution_runs_off_supervisor_loop_and_continues_once(
|
||||
monkeypatch, tmp_path,
|
||||
):
|
||||
import queue as thread_queue
|
||||
|
||||
import supervisor.queue as supervisor_queue
|
||||
import supervisor.workers as workers
|
||||
from ouroboros.task_results import load_task_result
|
||||
from supervisor.events import _handle_promote_chat_to_task
|
||||
|
||||
pending = []
|
||||
started = threading.Event()
|
||||
release = threading.Event()
|
||||
continuation_bus = thread_queue.Queue()
|
||||
|
||||
def slow_resolve(_ctx, _source, project_id):
|
||||
started.set()
|
||||
assert release.wait(2)
|
||||
return "", "source checked", "", project_id
|
||||
|
||||
monkeypatch.setattr(
|
||||
"ouroboros.promotion_source.resolve_promote_source", slow_resolve
|
||||
)
|
||||
monkeypatch.setattr(workers, "get_event_q", lambda: continuation_bus)
|
||||
monkeypatch.setattr(workers, "DRIVE_ROOT", tmp_path)
|
||||
monkeypatch.setattr(workers, "WORKERS", {0: types.SimpleNamespace()})
|
||||
monkeypatch.setattr(workers, "_WORKER_POOL_DISABLED_REASON", "")
|
||||
monkeypatch.setattr(supervisor_queue, "DRIVE_ROOT", tmp_path)
|
||||
monkeypatch.setattr(supervisor_queue, "PENDING", pending)
|
||||
monkeypatch.setattr(supervisor_queue, "RUNNING", {})
|
||||
monkeypatch.setattr(supervisor_queue, "ADMISSION_RESERVATIONS", {})
|
||||
ctx = types.SimpleNamespace(
|
||||
DRIVE_ROOT=tmp_path,
|
||||
WORKERS={0: types.SimpleNamespace()},
|
||||
PENDING=pending,
|
||||
bridge=None,
|
||||
enqueue_task=supervisor_queue.enqueue_task,
|
||||
persist_queue_snapshot=lambda **_kwargs: True,
|
||||
load_state=lambda: {"owner_chat_id": 1},
|
||||
append_jsonl=lambda *_args, **_kwargs: None,
|
||||
)
|
||||
event = {
|
||||
"type": "promote_chat_to_task",
|
||||
"task_id": "source-task",
|
||||
"routing_token": "source-token",
|
||||
"objective": "Inspect source",
|
||||
"source": "https://github.com/example/project.git",
|
||||
"chat_id": 1,
|
||||
}
|
||||
|
||||
first = _handle_promote_chat_to_task(event, ctx)
|
||||
assert first == {"status": "preparing", "task_id": "source-task"}
|
||||
assert started.wait(0.5)
|
||||
assert pending == []
|
||||
duplicate = _handle_promote_chat_to_task(event, ctx)
|
||||
assert duplicate == {"status": "preparing", "task_id": "source-task"}
|
||||
|
||||
release.set()
|
||||
continuation = continuation_bus.get(timeout=2)
|
||||
final = _handle_promote_chat_to_task(continuation, ctx)
|
||||
|
||||
assert final["status"] == "scheduled"
|
||||
assert [row["id"] for row in pending] == ["source-task"]
|
||||
assert load_task_result(tmp_path, "source-task")["promotion_admission"][
|
||||
"routing_token"
|
||||
] == "source-token"
|
||||
|
|
@ -36,7 +36,8 @@ def test_route_to_existing_project_emits_event_and_receipt(tmp_path):
|
|||
"origin_message_text": "continue the engine tuning",
|
||||
})
|
||||
out = _route_to_project(ctx, "racer", "paraphrased: keep tuning the engine", reason="follow-up")
|
||||
assert out.startswith("✉️ Routed to project 'Racer' (racer)")
|
||||
assert out.startswith("⚠️ ROUTE_UNCONFIRMED:")
|
||||
assert "do not retry automatically" in out.lower()
|
||||
assert len(events) == 1
|
||||
evt = events[0]
|
||||
assert evt["type"] == "promote_chat_to_task"
|
||||
|
|
@ -46,6 +47,7 @@ def test_route_to_existing_project_emits_event_and_receipt(tmp_path):
|
|||
assert "routing reason: follow-up" in evt["objective"]
|
||||
assert evt["chat_id"] == 1
|
||||
assert evt["task_id"]
|
||||
assert evt["routing_token"]
|
||||
assert evt["source_ref"] == origin_ref
|
||||
assert evt["source_text"] == "continue the engine tuning"
|
||||
assert ctx._typed_routing_action_emitted == "route_to_project"
|
||||
|
|
@ -59,23 +61,23 @@ def test_route_to_missing_project_emits_typed_manual_target(tmp_path):
|
|||
}
|
||||
ctx = _ctx(tmp_path, events, task_metadata=metadata)
|
||||
out = _route_to_project(ctx, "ghost", "do the thing")
|
||||
assert "NEEDS_MANUAL_TARGET" in out
|
||||
assert events == [{
|
||||
"type": "routing_manual_target",
|
||||
"chat_id": 1,
|
||||
"client_message_id": "owner-1",
|
||||
"requested_target": "ghost",
|
||||
"reason": "target_not_found",
|
||||
"options": [{"task_id": "task-1", "title": "Fix it"}],
|
||||
"ts": events[0]["ts"],
|
||||
}]
|
||||
assert "ROUTING_UNCONFIRMED" in out
|
||||
assert len(events) == 1
|
||||
assert events[0]["type"] == "routing_manual_target"
|
||||
assert events[0]["routing_token"]
|
||||
assert events[0]["chat_id"] == 1
|
||||
assert events[0]["client_message_id"] == "owner-1"
|
||||
assert events[0]["requested_target"] == "ghost"
|
||||
assert events[0]["reason"] == "target_not_found"
|
||||
assert events[0]["options"] == [{"task_id": "task-1", "title": "Fix it"}]
|
||||
assert ctx._typed_routing_action_emitted == "routing_manual_target"
|
||||
|
||||
|
||||
def test_route_rejects_dirty_project_id(tmp_path):
|
||||
events = []
|
||||
out = _route_to_project(_ctx(tmp_path, events), "Bad Name!", "msg")
|
||||
assert "NEEDS_MANUAL_TARGET" in out
|
||||
assert "ROUTING_UNCONFIRMED" in out
|
||||
assert events[0]["routing_token"]
|
||||
assert events[0]["reason"] == "invalid_project_id"
|
||||
|
||||
|
||||
|
|
@ -88,7 +90,8 @@ def test_route_empty_target_is_the_typed_abstention_path(tmp_path):
|
|||
},
|
||||
}
|
||||
out = _route_to_project(_ctx(tmp_path, events, task_metadata=metadata), "", "ambiguous follow-up")
|
||||
assert "NEEDS_MANUAL_TARGET" in out
|
||||
assert "ROUTING_UNCONFIRMED" in out
|
||||
assert events[0]["routing_token"]
|
||||
assert events[0]["reason"] == "target_unspecified"
|
||||
assert events[0]["options"][0]["label"] == "New task in Project"
|
||||
|
||||
|
|
|
|||
|
|
@ -37,7 +37,7 @@ def test_constrained_repair_promotes_managed_task_before_busy_ephemeral_lane(mon
|
|||
handle_chat_ephemeral=lambda cid, txt, img, task_constraint=None, task_metadata=None: calls["ephemeral"].append(task_constraint),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"supervisor.workers.promote_chat_to_task",
|
||||
"supervisor.events._handle_promote_chat_to_task",
|
||||
lambda event, _ctx: (
|
||||
calls["promote"].append(event)
|
||||
or {"status": "scheduled", "task_id": event["task_id"]}
|
||||
|
|
@ -71,7 +71,9 @@ def test_constrained_repair_promotes_managed_task_before_busy_ephemeral_lane(mon
|
|||
"payload_root": "skills/external/alpha",
|
||||
}
|
||||
assert event["origin_suppressed"] is True
|
||||
assert calls["sent"] == []
|
||||
assert len(calls["sent"]) == 1
|
||||
assert calls["sent"][0][0] == 1
|
||||
assert "accepted and durably scheduled" in calls["sent"][0][1]
|
||||
|
||||
|
||||
def test_constrained_repair_refusal_is_reported_to_owner(monkeypatch):
|
||||
|
|
@ -81,7 +83,7 @@ def test_constrained_repair_refusal_is_reported_to_owner(monkeypatch):
|
|||
send_with_budget=lambda chat_id, text: sent.append((chat_id, text)),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"supervisor.workers.promote_chat_to_task",
|
||||
"supervisor.events._handle_promote_chat_to_task",
|
||||
lambda event, _ctx: {
|
||||
"status": "needs_manual_target",
|
||||
"reason": "skill_repair_payload_missing",
|
||||
|
|
|
|||
|
|
@ -334,7 +334,7 @@ def test_promote_source_registers_derived_project_and_mirrors_conflict(tmp_path,
|
|||
|
||||
import ouroboros.config as config
|
||||
from ouroboros.projects_registry import get_project as get_reg_project
|
||||
from ouroboros.tools.control import _resolve_promote_source
|
||||
from ouroboros.promotion_source import resolve_promote_source
|
||||
|
||||
data = tmp_path / "data"
|
||||
data.mkdir()
|
||||
|
|
@ -349,11 +349,11 @@ def test_promote_source_registers_derived_project_and_mirrors_conflict(tmp_path,
|
|||
# admission requires a git worktree root — no born-dead project rooms).
|
||||
nogit = tmp_path / "plain_folder"
|
||||
nogit.mkdir()
|
||||
ws0, _, err0, _ = _resolve_promote_source(ctx, str(nogit), "")
|
||||
ws0, _, err0, _ = resolve_promote_source(ctx, str(nogit), "")
|
||||
assert ws0 == "" and "not a git repository" in err0
|
||||
|
||||
# No pid given: derived from the folder name, registered with provenance facts.
|
||||
ws, note, err, pid = _resolve_promote_source(ctx, str(folder), "")
|
||||
ws, note, err, pid = resolve_promote_source(ctx, str(folder), "")
|
||||
assert err == "" and pid == "myrepo" and ws
|
||||
entry = get_reg_project(data, "myrepo")
|
||||
assert entry is not None
|
||||
|
|
@ -363,7 +363,7 @@ def test_promote_source_registers_derived_project_and_mirrors_conflict(tmp_path,
|
|||
assert trusted_first
|
||||
|
||||
# Same folder again: idempotent, original trusted_at preserved.
|
||||
ws2, _, err2, pid2 = _resolve_promote_source(ctx, str(folder), "myrepo")
|
||||
ws2, _, err2, pid2 = resolve_promote_source(ctx, str(folder), "myrepo")
|
||||
assert err2 == "" and pid2 == "myrepo" and ws2 == ws
|
||||
assert get_reg_project(data, "myrepo")["trusted_at"] == trusted_first
|
||||
|
||||
|
|
@ -371,7 +371,7 @@ def test_promote_source_registers_derived_project_and_mirrors_conflict(tmp_path,
|
|||
other = tmp_path / "other"
|
||||
other.mkdir()
|
||||
subprocess.run(["git", "init", "-q"], cwd=str(other), check=True)
|
||||
ws3, _, err3, _ = _resolve_promote_source(ctx, str(other), "myrepo")
|
||||
ws3, _, err3, _ = resolve_promote_source(ctx, str(other), "myrepo")
|
||||
assert ws3 == "" and "conflict" in err3
|
||||
assert get_reg_project(data, "myrepo")["working_dir"] == ws
|
||||
|
||||
|
|
@ -387,5 +387,5 @@ def test_promote_source_registers_derived_project_and_mirrors_conflict(tmp_path,
|
|||
monkeypatch.setattr(
|
||||
"ouroboros.project_sources.clone_project_repo", _never_clone
|
||||
)
|
||||
ws4, _, err4, _ = _resolve_promote_source(ctx, "https://example.com/myrepo.git", "myrepo")
|
||||
ws4, _, err4, _ = resolve_promote_source(ctx, "https://example.com/myrepo.git", "myrepo")
|
||||
assert ws4 == "" and "conflict" in err4
|
||||
|
|
|
|||
|
|
@ -143,9 +143,13 @@ def _tool_ctx(tmp_path, events, metadata):
|
|||
)
|
||||
|
||||
|
||||
def test_promote_tool_passes_origin_by_value_despite_rewritten_objective(tmp_path):
|
||||
def test_promote_tool_passes_origin_by_value_despite_rewritten_objective(tmp_path, monkeypatch):
|
||||
from ouroboros.tools.control import _promote_chat_to_task
|
||||
|
||||
monkeypatch.setattr(
|
||||
"ouroboros.tools.control._wait_for_promotion_admission",
|
||||
lambda *_args, **_kwargs: {"status": "scheduled"},
|
||||
)
|
||||
events = []
|
||||
ctx = _tool_ctx(tmp_path, events, {
|
||||
"client_message_id": "owner-msg-1",
|
||||
|
|
@ -157,7 +161,7 @@ def test_promote_tool_passes_origin_by_value_despite_rewritten_objective(tmp_pat
|
|||
objective="Create a standalone 3D browser game about a robot (LLM-rewritten)",
|
||||
project_name="Robot City Adventure",
|
||||
)
|
||||
assert out.startswith("OK: promoted")
|
||||
assert out.startswith("OK: task")
|
||||
evt = events[0]
|
||||
assert evt["source_ref"] == _ref()
|
||||
assert evt["source_text"] == OWNER_TEXT
|
||||
|
|
@ -191,6 +195,7 @@ def test_promote_worker_absence_reason_follows_provenance(tmp_path, monkeypatch)
|
|||
create_project(tmp_path, "racer")
|
||||
ctx = SimpleNamespace(
|
||||
enqueue_task=lambda task: None,
|
||||
persist_queue_snapshot=lambda **_kwargs: True,
|
||||
load_state=lambda: {"owner_chat_id": 1},
|
||||
)
|
||||
result = workers.promote_chat_to_task({
|
||||
|
|
@ -568,7 +573,11 @@ def test_suppressed_message_promote_is_designed_absence(tmp_path, monkeypatch):
|
|||
"type": "promote_chat_to_task", "task_id": "sup-1",
|
||||
"objective": "Continue", "project_id": "supproj", "chat_id": 1,
|
||||
"client_message_id": "owner-sup-1", "origin_suppressed": True,
|
||||
}, SimpleNamespace(enqueue_task=lambda t: None, load_state=lambda: {"owner_chat_id": 1}))
|
||||
}, SimpleNamespace(
|
||||
enqueue_task=lambda t: None,
|
||||
persist_queue_snapshot=lambda **_kwargs: True,
|
||||
load_state=lambda: {"owner_chat_id": 1},
|
||||
))
|
||||
assert project_binding_for_task(tmp_path, "sup-1")["origin_absent"] == "mid_task_no_origin"
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -13,6 +13,20 @@ from __future__ import annotations
|
|||
import time
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _isolate_worker_crash_state():
|
||||
"""Crash history is process-global and must not leak between serial tests."""
|
||||
import supervisor.workers as workers
|
||||
|
||||
workers.CRASH_TS.clear()
|
||||
workers._WORKER_POOL_DISABLED_REASON = ""
|
||||
yield
|
||||
workers.CRASH_TS.clear()
|
||||
workers._WORKER_POOL_DISABLED_REASON = ""
|
||||
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
|
|||
|
|
@ -1400,6 +1400,7 @@ def test_docker_executor_rejects_network_none_when_container_has_network(tmp_pat
|
|||
def test_api_task_metadata_accepts_normalized_executor_ref(tmp_path, monkeypatch):
|
||||
from ouroboros.gateway import tasks
|
||||
import supervisor.queue as queue
|
||||
import supervisor.workers as workers
|
||||
|
||||
captured: dict[str, object] = {}
|
||||
|
||||
|
|
@ -1419,6 +1420,7 @@ def test_api_task_metadata_accepts_normalized_executor_ref(tmp_path, monkeypatch
|
|||
|
||||
def fake_enqueue(task):
|
||||
captured.update(task)
|
||||
return task
|
||||
|
||||
_init_repo(tmp_path / "workspace")
|
||||
(tmp_path / "data").mkdir()
|
||||
|
|
@ -1426,7 +1428,9 @@ def test_api_task_metadata_accepts_normalized_executor_ref(tmp_path, monkeypatch
|
|||
monkeypatch.setattr(tasks, "request_drive_root", lambda _request: tmp_path / "data")
|
||||
monkeypatch.setattr(tasks, "request_repo_dir", lambda _request: tmp_path / "repo")
|
||||
monkeypatch.setattr(queue, "enqueue_task", fake_enqueue)
|
||||
monkeypatch.setattr(queue, "persist_queue_snapshot", lambda *a, **k: None)
|
||||
monkeypatch.setattr(queue, "persist_queue_snapshot", lambda *a, **k: True)
|
||||
monkeypatch.setattr(workers, "WORKERS", {0: SimpleNamespace()})
|
||||
monkeypatch.setattr(workers, "_WORKER_POOL_DISABLED_REASON", "")
|
||||
|
||||
request = SimpleNamespace(app=SimpleNamespace(state=SimpleNamespace(supervisor_ready_event=None)))
|
||||
response = asyncio.run(tasks.api_tasks_create(request))
|
||||
|
|
|
|||
|
|
@ -510,4 +510,4 @@
|
|||
* @property {boolean=} ok
|
||||
*/
|
||||
|
||||
export const GATEWAY_CONTRACT_VERSION = '6.87.2';
|
||||
export const GATEWAY_CONTRACT_VERSION = '6.87.3';
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
{
|
||||
"name": "ouroboros-web",
|
||||
"version": "6.87.2",
|
||||
"version": "6.87.3",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"description": "Ouroboros browser UI package boundary",
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue