diff --git a/docs/AI.md b/docs/AI.md index 33d644776..09d5dcb4a 100644 --- a/docs/AI.md +++ b/docs/AI.md @@ -29,7 +29,7 @@ Pulse Assistant is a **protocol-driven, safety-gated LLM tool surface** that: - **Grounds answers in real tool work** โ€” visible tool traces, read-after-write verification, and transcript hygiene prevent unsupported execution claims from being treated as facts - **Returns structured tool errors** โ€” the model can recover from clear, machine-readable failures -๐Ÿ“– **For a deep technical dive into the Assistant architecture, see [architecture/pulse-assistant-deep-dive.md](architecture/pulse-assistant-deep-dive.md).** +๐Ÿ“– **For a deep technical dive into the Assistant architecture, see [Assistant architecture](ASSISTANT_ARCHITECTURE.md).** ### Not Just Another Alerting System @@ -43,12 +43,12 @@ Pulse Patrol is a **scheduled and event-triggered governed operator** that: All while running entirely on your infrastructure with BYOK for complete privacy. -๐Ÿ“– **For a deep technical dive into the Patrol runtime, see [architecture/pulse-patrol-deep-dive.md](architecture/pulse-patrol-deep-dive.md).** +๐Ÿ“– **For a deep technical dive into the Patrol runtime, see [Patrol architecture](PATROL_ARCHITECTURE.md).** ๐Ÿงช **For independent live-fault qualification, safety gates, model comparison, and release-claim rules, see [AI_PATROL_QUALIFICATION.md](AI_PATROL_QUALIFICATION.md).** -See [architecture/pulse-assistant.md](architecture/pulse-assistant.md) for the original safety architecture documentation. +See [Assistant safety architecture](ASSISTANT_SAFETY.md) for the original safety architecture documentation. ### Assistant And MCP @@ -696,11 +696,11 @@ Pulse tracks token usage and costs: ### Deep Dives (Recommended for Technical Audiences) -- **[Pulse Assistant Deep Dive](architecture/pulse-assistant-deep-dive.md)** โ€” Complete technical breakdown of the model-owned tool surface: explicit context, session fact caching, FSM enforcement, parallel execution, grounded execution guardrails, structured errors -- **[Pulse Patrol Deep Dive](architecture/pulse-patrol-deep-dive.md)** โ€” Patrol runtime documentation: evidence assembly, deterministic signal extraction, model evaluation, investigation context, investigation orchestration +- **[Pulse Assistant Deep Dive](ASSISTANT_ARCHITECTURE.md)** โ€” Complete technical breakdown of the model-owned tool surface: explicit context, session fact caching, FSM enforcement, parallel execution, grounded execution guardrails, structured errors +- **[Pulse Patrol Deep Dive](PATROL_ARCHITECTURE.md)** โ€” Patrol runtime documentation: evidence assembly, deterministic signal extraction, model evaluation, investigation context, investigation orchestration ### Reference Documentation -- [Architecture: Pulse Assistant (Safety Gates)](architecture/pulse-assistant.md) โ€” Detailed FSM states, tool protocol, and invariants +- [Architecture: Pulse Assistant (Safety Gates)](ASSISTANT_SAFETY.md) โ€” Detailed FSM states, tool protocol, and invariants - [API Reference](API.md) โ€” Complete API endpoint documentation - [Plans and entitlements](PULSE_PRO.md) โ€” Community/Relay/Pro/Cloud features and licensing diff --git a/docs/AI_AUTONOMY.md b/docs/AI_AUTONOMY.md index 0de1f8a06..b5610ffe6 100644 --- a/docs/AI_AUTONOMY.md +++ b/docs/AI_AUTONOMY.md @@ -182,4 +182,4 @@ Token usage and estimated costs are tracked per provider: - [Pulse Intelligence Overview](AI.md) โ€” Full Pulse Intelligence system documentation - [Plans and Entitlements](PULSE_PRO.md) โ€” Feature availability by plan - [API Reference](API.md) โ€” Complete API documentation -- [Pulse Patrol Deep Dive](architecture/pulse-patrol-deep-dive.md) โ€” Technical architecture details +- [Pulse Patrol Deep Dive](PATROL_ARCHITECTURE.md) โ€” Technical architecture details diff --git a/docs/API.md b/docs/API.md index 4f9701c28..7e8c669b8 100644 --- a/docs/API.md +++ b/docs/API.md @@ -492,8 +492,8 @@ Detailed storage usage per node and pool. ### Recovery (formerly Backups / Snapshots) Pulse v6 uses the recovery API to provide a platform-agnostic view of backup and snapshot artifacts. -See `docs/architecture/RECOVERY_CONTRACT.md` for the provider-neutral contract -(subjects, points, rollups, posture, and filter semantics). +The endpoints below carry the provider-neutral contract covering subjects, +points, rollups, posture, and filter semantics. - `GET /api/recovery/points` - Query params: diff --git a/docs/ASSISTANT_ARCHITECTURE.md b/docs/ASSISTANT_ARCHITECTURE.md new file mode 100644 index 000000000..aef8dac3f --- /dev/null +++ b/docs/ASSISTANT_ARCHITECTURE.md @@ -0,0 +1,99 @@ +# Pulse Assistant deep dive + +How the Assistant's agentic loop executes tool calls, for readers who want +more than the overview in [AI features](AI.md). + +The safety state machine has its own page. See +[Pulse Assistant safety architecture](ASSISTANT_SAFETY.md) for the states, +transitions, and invariants. This page covers the loop that runs around it. + +## The three-phase pipeline + +Each provider turn can return several tool calls at once. The loop processes +them in three phases, and the split matters because only one of the three is +safe to parallelise. + +**Phase 1, pre-check, runs sequentially.** This is where the state machine +gate, loop detection, and budget checks happen. Every call is judged before +any call runs. + +**Phase 2, execute, runs in parallel.** Independent calls run concurrently +through goroutines, with concurrency capped at four. + +**Phase 3, post-process, runs sequentially.** Streaming output, state machine +transitions, and knowledge extraction happen in a deterministic order, so +concurrent execution cannot produce non-deterministic session state. + +## What is not allowed to run in parallel + +Parallelism is bounded by real ordering requirements rather than applied +uniformly. + +A same-turn read-before-write dependency forces sequential execution. If one +turn contains both a `patrol_get_findings` read and a finding lifecycle write, +the batch runs in order, because the write's deduplication and assessment +precondition is established by that read. Letting them race inside one turn +would mean writing against a precondition that had not been checked. +Independent reads and independent finding writes still run in parallel. The +provider's original call order stays authoritative. + +Interactive input is also excluded. `pulse_question` never runs in parallel +with other tools, since asking you something is not an independent operation. + +## The look-before-asking gate + +The Assistant is discouraged from asking you a question before it has tried to +find the answer. If the model attempts to ask without having attempted any +tool call, the attempt is blocked and it is pushed to look first. + +The gate is bounded rather than absolute. It allows at most two blocks per +turn, after which the question goes through. A model that genuinely cannot +proceed without input is not trapped in a loop, and any real tool attempt +satisfies the gate immediately. + +## Structured errors + +Failures reach the model as stable machine-readable codes rather than as +prose, so it can branch on the failure instead of parsing an English sentence. +The codes are declared in `internal/agentcapabilities/errors.go` and include +`resource_not_found`, `operator_state_not_set`, `operator_state_invalid`, +`invalid_finding_request`, `finding_not_found`, `finding_action_not_allowed`, +`patrol_unavailable`, `invalid_action_request`, `capability_not_found`, +`action_execution_unavailable`, `action_actor_unavailable`, and `missing_id`. + +A call blocked by the state machine uses the same mechanism, returning the +`fsm_blocked` code with the state and tool that were involved. + +The same codes are published in the capability manifest at +`/api/agent/capabilities`, and a contract test fails the build if a handler +can emit a code the manifest does not declare, or the manifest declares a code +no handler emits. See [agent integrations](AGENT_SUBSTRATE.md). + +## Grounded execution + +Several guardrails exist to keep the model's claims tied to evidence it +actually gathered. + +The state machine supplies the structural half. A write moves the session into +verification, and the Assistant cannot deliver a final answer about that write +until it has read something afterwards. + +The prompts supply the rest. Instructions repeated across the agentic prompts +tell the model to treat infrastructure names, labels, logs, and other +collected values as untrusted data rather than as instructions, and not to +invent evidence, root cause, verification, remediation, or a claim that an +action was taken. + +Prompt instructions are the weaker of the two, which is exactly why the +verification requirement lives in code instead. Where a guarantee needs to +hold, it is enforced structurally. + +## Related reading + +- [Pulse Assistant safety architecture](ASSISTANT_SAFETY.md) for the state + machine in detail. +- [Patrol deep dive](PATROL_ARCHITECTURE.md) for the scheduled analysis + runtime. +- [Agent integrations](AGENT_SUBSTRATE.md) for driving the same surface + from an external agent. +- [AI features](AI.md) for the overview and configuration. diff --git a/docs/ASSISTANT_SAFETY.md b/docs/ASSISTANT_SAFETY.md new file mode 100644 index 000000000..7c8c2fdce --- /dev/null +++ b/docs/ASSISTANT_SAFETY.md @@ -0,0 +1,100 @@ +# Pulse Assistant safety architecture + +The state machine that governs what the Pulse Assistant is allowed to do +during a chat session, the tool classification it runs on, and the invariants +it holds. + +The point of this machine is structural. Prompt wording can be argued with by +a model, and drifts as prompts are edited. These rules are enforced in code, +in `internal/ai/chat/fsm.go`, so neither a model nor a future prompt change +can talk its way past them. + +## Tool kinds + +Every tool call is classified before it runs. + +| Kind | Meaning | +|---|---| +| `resolve` | Discovery and query tools that find resources | +| `read` | Read-only tools such as logs, metrics, status, and config | +| `write` | Mutating tools such as restart, stop, start, delete, and file write | +| `user_input` | Interactive tools that ask you something | + +Classification is `ClassifyToolCall`, which delegates to the shared +`agentcapabilities` classifier so the Assistant and the rest of the agent +surface agree on what counts as a write. + +## States + +A session starts in `RESOLVING` and moves between four states. + +| State | What it means | +|---|---| +| `RESOLVING` | No validated target yet, so resources must be discovered first | +| `READING` | A target is established and querying is allowed | +| `WRITING` | Transitional, entered around a mutation | +| `VERIFYING` | A write happened and evidence has not been gathered since | + +Transitions on a successful tool call are as follows. + +- A `resolve` or `read` in `RESOLVING` moves the session to `READING`. +- A `write` from any state moves the session to `VERIFYING`, records the tool + and timestamp, and clears the read-after-write flag. +- A `resolve` or `read` while in `VERIFYING` sets read-after-write, which is + what satisfies the verification requirement. +- A `user_input` call does not advance state at all, because asking you a + question is neither discovery nor verification. + +`CompleteVerification` returns a verified session from `VERIFYING` to +`READING` so further writes become possible. + +## The invariants + +**No writing without a validated target.** A `write` attempted in `RESOLVING` +is blocked. The model must establish what it is acting on before it acts. + +**No writing again until the last write is verified.** A `write` attempted in +`VERIFYING` is blocked until a read or resolve has run since the write. + +**No final answer about an unverified change.** `CanFinalAnswer` refuses while +the session is in `VERIFYING` with no read-after-write. The Assistant cannot +tell you it restarted something and then decline to look at whether the +restart worked. + +**Repeated attempts do not wear the gate down.** Consecutive blocked writes in +`VERIFYING` increment a counter, and that counter is telemetry only. There is +no attempt threshold after which the verification requirement is waived. + +**Reads are never blocked.** No state blocks a `read`, `resolve`, or +`user_input`. The machine constrains mutation and the claims made about +mutation, not information gathering. + +## Blocked calls and recovery + +A blocked call returns an `FSMBlockedError` carrying the state, the tool, the +tool kind, a reason, and a recoverable flag. It surfaces to the model with the +stable code `ErrCodeFSMBlocked` rather than as prose, so the model can branch +on the code. + +Blocks are recoverable rather than terminal. The session tracks a pending +recovery per blocked operation, and a later successful call of the same tool +clears it. Pending recoveries expire after ten minutes. + +## Resetting + +`Reset` returns the session to `RESOLVING` and clears all tracking, which is +what a full session clear does. + +`ResetKeepProgress` is the softer variant used when context is cleared but +pinned items are kept. It drops verification tracking and moves a `VERIFYING` +session back to `READING`, without discarding that a target was established. + +Note that `WroteThisEpisode` means "wrote at all during this session" rather +than "wrote during the current verification cycle", and `CompleteVerification` +deliberately leaves it set. + +## Related reading + +- [AI features](AI.md) for the overview and configuration. +- [Patrol deep dive](PATROL_ARCHITECTURE.md) for the scheduled analysis + runtime, which is a separate loop from the Assistant. diff --git a/docs/PATROL_ARCHITECTURE.md b/docs/PATROL_ARCHITECTURE.md new file mode 100644 index 000000000..1e9c135d8 --- /dev/null +++ b/docs/PATROL_ARCHITECTURE.md @@ -0,0 +1,152 @@ +# Pulse Patrol deep dive + +How a Patrol run actually works, for readers who want more than the overview +in [AI features](AI.md). + +The short version is that Patrol does not trust the model to notice +everything. A run pairs model judgement with deterministic detection, and the +deterministic half acts as a safety net over the model's output rather than as +an input to it. + +## A run, end to end + +A run moves through the following stages. Each is described in more detail +below. + +1. A pre-flight budget check, which fails fast before any context is built. +2. Guest intelligence gathering, covering discovery and reachability probing. +3. Deterministic triage, which narrows the surface the model will look at. +4. Seed context construction from the triage result. +5. The agentic loop, where the model calls tools and files findings. +6. Deterministic signal detection over the tool calls the model actually made. +7. An evaluation pass for signals the model saw but did not file. +8. An assessment sweep for findings the model left unassessed. + +## Budget check first + +The budget check runs before context assembly and before a chat service is +acquired. A run that cannot afford to complete is stopped at the cheapest +possible point rather than after the expensive work. + +## Guest intelligence + +Discovery and reachability probing run before the seed context is built, so +the context the model receives already reflects which guests are reachable. +Reachability results are kept, because they feed the deterministic detection +stage later in the run. + +## Deterministic triage and seed context + +Triage narrows the surface first, and the seed context is then built from what +triage returned rather than from everything Pulse knows. This keeps the +model's starting context focused on what looks interesting. + +Quiet infrastructure still goes through the configured model. Patrol does not +skip the model when nothing looks wrong, because "nothing is wrong" is itself +a judgement worth making explicitly. + +The turn budget for the loop is computed from the size of the surface. See +`computePatrolMaxTurns` and `computeTriageMaxTurns` in +`internal/ai/patrol_ai.go`. + +## The agentic loop + +The model runs an agentic loop with tool access. Findings are created through +tool calls during the loop rather than parsed out of prose afterwards, so a +finding exists because the model deliberately filed it. + +## Deterministic signal detection + +After the loop completes, Patrol collects the tool calls that ran and scans +their results for known problem signals. This scan is deterministic and does +not depend on model judgement at all. The entry point is `DetectSignals` in +`internal/ai/patrol_signals.go`. + +The signal types are: + +| Signal | Meaning | +|---|---| +| `smart_failure` | A disk reporting SMART failure | +| `high_cpu` | Sustained CPU above the threshold | +| `high_memory` | Sustained memory above the threshold | +| `high_disk` | Disk or pool usage above the threshold | +| `backup_failed` | A backup job that failed | +| `backup_stale` | No backup inside the staleness window | +| `guest_unreachable` | A guest that failed reachability probing | + +Thresholds come from your own alert configuration rather than from a separate +set of numbers, so detection agrees with what Pulse would alert on. +`SignalThresholdsFromPatrol` maps your configured alert thresholds onto the +signal thresholds, falling back per field to these defaults when a value is +not configured. + +| Threshold | Default | +|---|---| +| Storage warning | 75% | +| Storage critical | 95% | +| High CPU | 70% | +| High memory | 80% | +| Backup staleness | 48 hours | + +Backup staleness has no user-facing setting and always uses the default. + +Reachability signals detected during the guest intelligence stage are merged +into the same set at this point. + +## The evaluation pass + +Detected signals are compared against the findings the model actually filed. +Any signal with no corresponding finding is an unmatched signal, meaning the +data showed a problem and the model did not report it. + +Unmatched signals go to a bounded evaluation pass scoped to exactly those +signals. This is the safety net. It is why a run can still surface a failing +disk when the model overlooked one in a large sweep. + +## The assessment sweep + +Smaller models sometimes finish the main pass without recording an assessment +for every finding they filed. Patrol detects the missing assessments and runs +one bounded follow-up pass scoped to exactly those findings, rather than +rerunning the whole analysis or leaving the findings unassessed. + +## Investigation + +A finding may be investigated after it is filed. Investigation is a separate +loop from the run that produced the finding, coordinated through the +`InvestigationOrchestrator` contract in `pkg/aicontracts/interfaces.go`. + +Whether a finding is investigated at all depends on your autonomy level and on +the finding's own state. `Finding.ShouldInvestigate` in +`internal/ai/findings.go` is the gate, and it refuses in every one of these +cases. + +- Autonomy is unset or `monitor`. Investigation only runs at `approval`, + `assisted`, or `full`. +- The finding is resolved, dismissed, suppressed, or snoozed. +- The severity is not warning or critical. Info and watch findings are not + investigated. +- A fix is already queued for approval, so the next move belongs to you. +- The finding reached a terminal outcome, such as fix verified, resolved, fix + rejected, cannot fix, or needs attention. +- The finding has already been investigated three times. +- The last investigation was inside the cooldown, which is one hour by + default and shorter for timeout failures, since those are transient. + +Patrol also recovers investigations that were left running and retries ones +that timed out. See `recoverStuckInvestigations` and +`retryTimedOutInvestigations` in `internal/ai/patrol_findings.go`. + +## Why it is built this way + +Model judgement is good at explaining and correlating, and unreliable at +exhaustive enumeration. Deterministic detection is the reverse. Patrol uses +each for what it is good at, and the unmatched-signal check is the seam where +the deterministic half can correct the model. + +## Related reading + +- [AI features](AI.md) for the overview and configuration. +- [Patrol autonomy](AI_AUTONOMY.md) for what each autonomy level permits. +- [Patrol qualification](AI_PATROL_QUALIFICATION.md) for the evidence and + release-claim rules. diff --git a/docs/PULSE_PRO.md b/docs/PULSE_PRO.md index b9f0a812e..e2f901bb0 100644 --- a/docs/PULSE_PRO.md +++ b/docs/PULSE_PRO.md @@ -2,8 +2,6 @@ This document explains Pulse's user-facing plan structure, the locked self-hosted commercial model, and how those plans map to runtime feature gates. -For the canonical, code-aligned entitlement table (including internal tier names), see: -- `docs/architecture/ENTITLEMENT_MATRIX.md` ## Plan Mapping (User-Facing -> Code Tiers) @@ -134,7 +132,7 @@ Legend: - Included: `Y` / `N` - `Y*`: Enterprise/custom only (`enterprise` tier or explicit entitlement) -This matrix is derived from the canonical table in `docs/architecture/ENTITLEMENT_MATRIX.md` plus runtime history/limit semantics exposed through entitlements. +This matrix reflects the entitlement keys enforced in code plus the runtime history and limit semantics exposed through entitlements. | Constant | Capability Key | Display Name | Community | Relay | Pro | Cloud | Primary Gating Mechanism / Notes | |---|---|---|:---:|:---:|:---:|:---:|---| @@ -221,6 +219,6 @@ This returns a feature map including keys like `relay`, `ai_alerts`, `ai_autofix ## Deep Dives -- [Pulse Patrol Deep Dive](architecture/pulse-patrol-deep-dive.md) -- [Pulse Assistant Deep Dive](architecture/pulse-assistant-deep-dive.md) +- [Pulse Patrol Deep Dive](PATROL_ARCHITECTURE.md) +- [Pulse Assistant Deep Dive](ASSISTANT_ARCHITECTURE.md) - [Pulse Intelligence overview](AI.md) diff --git a/docs/UPGRADE_v6.md b/docs/UPGRADE_v6.md index 87c362e02..147c87884 100644 --- a/docs/UPGRADE_v6.md +++ b/docs/UPGRADE_v6.md @@ -384,4 +384,3 @@ Multi-tenant mode is opt-in and additionally license-gated: - Enablement flag: `PULSE_MULTI_TENANT_ENABLED=true` - Capability gate: `multi_tenant` -See any multi-tenant operational docs under `docs/architecture/` if you plan to run this mode. diff --git a/frontend-modern/public/docs/AI.md b/frontend-modern/public/docs/AI.md index 33d644776..09d5dcb4a 100644 --- a/frontend-modern/public/docs/AI.md +++ b/frontend-modern/public/docs/AI.md @@ -29,7 +29,7 @@ Pulse Assistant is a **protocol-driven, safety-gated LLM tool surface** that: - **Grounds answers in real tool work** โ€” visible tool traces, read-after-write verification, and transcript hygiene prevent unsupported execution claims from being treated as facts - **Returns structured tool errors** โ€” the model can recover from clear, machine-readable failures -๐Ÿ“– **For a deep technical dive into the Assistant architecture, see [architecture/pulse-assistant-deep-dive.md](architecture/pulse-assistant-deep-dive.md).** +๐Ÿ“– **For a deep technical dive into the Assistant architecture, see [Assistant architecture](ASSISTANT_ARCHITECTURE.md).** ### Not Just Another Alerting System @@ -43,12 +43,12 @@ Pulse Patrol is a **scheduled and event-triggered governed operator** that: All while running entirely on your infrastructure with BYOK for complete privacy. -๐Ÿ“– **For a deep technical dive into the Patrol runtime, see [architecture/pulse-patrol-deep-dive.md](architecture/pulse-patrol-deep-dive.md).** +๐Ÿ“– **For a deep technical dive into the Patrol runtime, see [Patrol architecture](PATROL_ARCHITECTURE.md).** ๐Ÿงช **For independent live-fault qualification, safety gates, model comparison, and release-claim rules, see [AI_PATROL_QUALIFICATION.md](AI_PATROL_QUALIFICATION.md).** -See [architecture/pulse-assistant.md](architecture/pulse-assistant.md) for the original safety architecture documentation. +See [Assistant safety architecture](ASSISTANT_SAFETY.md) for the original safety architecture documentation. ### Assistant And MCP @@ -696,11 +696,11 @@ Pulse tracks token usage and costs: ### Deep Dives (Recommended for Technical Audiences) -- **[Pulse Assistant Deep Dive](architecture/pulse-assistant-deep-dive.md)** โ€” Complete technical breakdown of the model-owned tool surface: explicit context, session fact caching, FSM enforcement, parallel execution, grounded execution guardrails, structured errors -- **[Pulse Patrol Deep Dive](architecture/pulse-patrol-deep-dive.md)** โ€” Patrol runtime documentation: evidence assembly, deterministic signal extraction, model evaluation, investigation context, investigation orchestration +- **[Pulse Assistant Deep Dive](ASSISTANT_ARCHITECTURE.md)** โ€” Complete technical breakdown of the model-owned tool surface: explicit context, session fact caching, FSM enforcement, parallel execution, grounded execution guardrails, structured errors +- **[Pulse Patrol Deep Dive](PATROL_ARCHITECTURE.md)** โ€” Patrol runtime documentation: evidence assembly, deterministic signal extraction, model evaluation, investigation context, investigation orchestration ### Reference Documentation -- [Architecture: Pulse Assistant (Safety Gates)](architecture/pulse-assistant.md) โ€” Detailed FSM states, tool protocol, and invariants +- [Architecture: Pulse Assistant (Safety Gates)](ASSISTANT_SAFETY.md) โ€” Detailed FSM states, tool protocol, and invariants - [API Reference](API.md) โ€” Complete API endpoint documentation - [Plans and entitlements](PULSE_PRO.md) โ€” Community/Relay/Pro/Cloud features and licensing diff --git a/frontend-modern/public/docs/AI_AUTONOMY.md b/frontend-modern/public/docs/AI_AUTONOMY.md index 0de1f8a06..b5610ffe6 100644 --- a/frontend-modern/public/docs/AI_AUTONOMY.md +++ b/frontend-modern/public/docs/AI_AUTONOMY.md @@ -182,4 +182,4 @@ Token usage and estimated costs are tracked per provider: - [Pulse Intelligence Overview](AI.md) โ€” Full Pulse Intelligence system documentation - [Plans and Entitlements](PULSE_PRO.md) โ€” Feature availability by plan - [API Reference](API.md) โ€” Complete API documentation -- [Pulse Patrol Deep Dive](architecture/pulse-patrol-deep-dive.md) โ€” Technical architecture details +- [Pulse Patrol Deep Dive](PATROL_ARCHITECTURE.md) โ€” Technical architecture details diff --git a/frontend-modern/public/docs/API.md b/frontend-modern/public/docs/API.md index 4f9701c28..7e8c669b8 100644 --- a/frontend-modern/public/docs/API.md +++ b/frontend-modern/public/docs/API.md @@ -492,8 +492,8 @@ Detailed storage usage per node and pool. ### Recovery (formerly Backups / Snapshots) Pulse v6 uses the recovery API to provide a platform-agnostic view of backup and snapshot artifacts. -See `docs/architecture/RECOVERY_CONTRACT.md` for the provider-neutral contract -(subjects, points, rollups, posture, and filter semantics). +The endpoints below carry the provider-neutral contract covering subjects, +points, rollups, posture, and filter semantics. - `GET /api/recovery/points` - Query params: diff --git a/frontend-modern/public/docs/ASSISTANT_ARCHITECTURE.md b/frontend-modern/public/docs/ASSISTANT_ARCHITECTURE.md new file mode 100644 index 000000000..aef8dac3f --- /dev/null +++ b/frontend-modern/public/docs/ASSISTANT_ARCHITECTURE.md @@ -0,0 +1,99 @@ +# Pulse Assistant deep dive + +How the Assistant's agentic loop executes tool calls, for readers who want +more than the overview in [AI features](AI.md). + +The safety state machine has its own page. See +[Pulse Assistant safety architecture](ASSISTANT_SAFETY.md) for the states, +transitions, and invariants. This page covers the loop that runs around it. + +## The three-phase pipeline + +Each provider turn can return several tool calls at once. The loop processes +them in three phases, and the split matters because only one of the three is +safe to parallelise. + +**Phase 1, pre-check, runs sequentially.** This is where the state machine +gate, loop detection, and budget checks happen. Every call is judged before +any call runs. + +**Phase 2, execute, runs in parallel.** Independent calls run concurrently +through goroutines, with concurrency capped at four. + +**Phase 3, post-process, runs sequentially.** Streaming output, state machine +transitions, and knowledge extraction happen in a deterministic order, so +concurrent execution cannot produce non-deterministic session state. + +## What is not allowed to run in parallel + +Parallelism is bounded by real ordering requirements rather than applied +uniformly. + +A same-turn read-before-write dependency forces sequential execution. If one +turn contains both a `patrol_get_findings` read and a finding lifecycle write, +the batch runs in order, because the write's deduplication and assessment +precondition is established by that read. Letting them race inside one turn +would mean writing against a precondition that had not been checked. +Independent reads and independent finding writes still run in parallel. The +provider's original call order stays authoritative. + +Interactive input is also excluded. `pulse_question` never runs in parallel +with other tools, since asking you something is not an independent operation. + +## The look-before-asking gate + +The Assistant is discouraged from asking you a question before it has tried to +find the answer. If the model attempts to ask without having attempted any +tool call, the attempt is blocked and it is pushed to look first. + +The gate is bounded rather than absolute. It allows at most two blocks per +turn, after which the question goes through. A model that genuinely cannot +proceed without input is not trapped in a loop, and any real tool attempt +satisfies the gate immediately. + +## Structured errors + +Failures reach the model as stable machine-readable codes rather than as +prose, so it can branch on the failure instead of parsing an English sentence. +The codes are declared in `internal/agentcapabilities/errors.go` and include +`resource_not_found`, `operator_state_not_set`, `operator_state_invalid`, +`invalid_finding_request`, `finding_not_found`, `finding_action_not_allowed`, +`patrol_unavailable`, `invalid_action_request`, `capability_not_found`, +`action_execution_unavailable`, `action_actor_unavailable`, and `missing_id`. + +A call blocked by the state machine uses the same mechanism, returning the +`fsm_blocked` code with the state and tool that were involved. + +The same codes are published in the capability manifest at +`/api/agent/capabilities`, and a contract test fails the build if a handler +can emit a code the manifest does not declare, or the manifest declares a code +no handler emits. See [agent integrations](AGENT_SUBSTRATE.md). + +## Grounded execution + +Several guardrails exist to keep the model's claims tied to evidence it +actually gathered. + +The state machine supplies the structural half. A write moves the session into +verification, and the Assistant cannot deliver a final answer about that write +until it has read something afterwards. + +The prompts supply the rest. Instructions repeated across the agentic prompts +tell the model to treat infrastructure names, labels, logs, and other +collected values as untrusted data rather than as instructions, and not to +invent evidence, root cause, verification, remediation, or a claim that an +action was taken. + +Prompt instructions are the weaker of the two, which is exactly why the +verification requirement lives in code instead. Where a guarantee needs to +hold, it is enforced structurally. + +## Related reading + +- [Pulse Assistant safety architecture](ASSISTANT_SAFETY.md) for the state + machine in detail. +- [Patrol deep dive](PATROL_ARCHITECTURE.md) for the scheduled analysis + runtime. +- [Agent integrations](AGENT_SUBSTRATE.md) for driving the same surface + from an external agent. +- [AI features](AI.md) for the overview and configuration. diff --git a/frontend-modern/public/docs/ASSISTANT_SAFETY.md b/frontend-modern/public/docs/ASSISTANT_SAFETY.md new file mode 100644 index 000000000..7c8c2fdce --- /dev/null +++ b/frontend-modern/public/docs/ASSISTANT_SAFETY.md @@ -0,0 +1,100 @@ +# Pulse Assistant safety architecture + +The state machine that governs what the Pulse Assistant is allowed to do +during a chat session, the tool classification it runs on, and the invariants +it holds. + +The point of this machine is structural. Prompt wording can be argued with by +a model, and drifts as prompts are edited. These rules are enforced in code, +in `internal/ai/chat/fsm.go`, so neither a model nor a future prompt change +can talk its way past them. + +## Tool kinds + +Every tool call is classified before it runs. + +| Kind | Meaning | +|---|---| +| `resolve` | Discovery and query tools that find resources | +| `read` | Read-only tools such as logs, metrics, status, and config | +| `write` | Mutating tools such as restart, stop, start, delete, and file write | +| `user_input` | Interactive tools that ask you something | + +Classification is `ClassifyToolCall`, which delegates to the shared +`agentcapabilities` classifier so the Assistant and the rest of the agent +surface agree on what counts as a write. + +## States + +A session starts in `RESOLVING` and moves between four states. + +| State | What it means | +|---|---| +| `RESOLVING` | No validated target yet, so resources must be discovered first | +| `READING` | A target is established and querying is allowed | +| `WRITING` | Transitional, entered around a mutation | +| `VERIFYING` | A write happened and evidence has not been gathered since | + +Transitions on a successful tool call are as follows. + +- A `resolve` or `read` in `RESOLVING` moves the session to `READING`. +- A `write` from any state moves the session to `VERIFYING`, records the tool + and timestamp, and clears the read-after-write flag. +- A `resolve` or `read` while in `VERIFYING` sets read-after-write, which is + what satisfies the verification requirement. +- A `user_input` call does not advance state at all, because asking you a + question is neither discovery nor verification. + +`CompleteVerification` returns a verified session from `VERIFYING` to +`READING` so further writes become possible. + +## The invariants + +**No writing without a validated target.** A `write` attempted in `RESOLVING` +is blocked. The model must establish what it is acting on before it acts. + +**No writing again until the last write is verified.** A `write` attempted in +`VERIFYING` is blocked until a read or resolve has run since the write. + +**No final answer about an unverified change.** `CanFinalAnswer` refuses while +the session is in `VERIFYING` with no read-after-write. The Assistant cannot +tell you it restarted something and then decline to look at whether the +restart worked. + +**Repeated attempts do not wear the gate down.** Consecutive blocked writes in +`VERIFYING` increment a counter, and that counter is telemetry only. There is +no attempt threshold after which the verification requirement is waived. + +**Reads are never blocked.** No state blocks a `read`, `resolve`, or +`user_input`. The machine constrains mutation and the claims made about +mutation, not information gathering. + +## Blocked calls and recovery + +A blocked call returns an `FSMBlockedError` carrying the state, the tool, the +tool kind, a reason, and a recoverable flag. It surfaces to the model with the +stable code `ErrCodeFSMBlocked` rather than as prose, so the model can branch +on the code. + +Blocks are recoverable rather than terminal. The session tracks a pending +recovery per blocked operation, and a later successful call of the same tool +clears it. Pending recoveries expire after ten minutes. + +## Resetting + +`Reset` returns the session to `RESOLVING` and clears all tracking, which is +what a full session clear does. + +`ResetKeepProgress` is the softer variant used when context is cleared but +pinned items are kept. It drops verification tracking and moves a `VERIFYING` +session back to `READING`, without discarding that a target was established. + +Note that `WroteThisEpisode` means "wrote at all during this session" rather +than "wrote during the current verification cycle", and `CompleteVerification` +deliberately leaves it set. + +## Related reading + +- [AI features](AI.md) for the overview and configuration. +- [Patrol deep dive](PATROL_ARCHITECTURE.md) for the scheduled analysis + runtime, which is a separate loop from the Assistant. diff --git a/frontend-modern/public/docs/PATROL_ARCHITECTURE.md b/frontend-modern/public/docs/PATROL_ARCHITECTURE.md new file mode 100644 index 000000000..1e9c135d8 --- /dev/null +++ b/frontend-modern/public/docs/PATROL_ARCHITECTURE.md @@ -0,0 +1,152 @@ +# Pulse Patrol deep dive + +How a Patrol run actually works, for readers who want more than the overview +in [AI features](AI.md). + +The short version is that Patrol does not trust the model to notice +everything. A run pairs model judgement with deterministic detection, and the +deterministic half acts as a safety net over the model's output rather than as +an input to it. + +## A run, end to end + +A run moves through the following stages. Each is described in more detail +below. + +1. A pre-flight budget check, which fails fast before any context is built. +2. Guest intelligence gathering, covering discovery and reachability probing. +3. Deterministic triage, which narrows the surface the model will look at. +4. Seed context construction from the triage result. +5. The agentic loop, where the model calls tools and files findings. +6. Deterministic signal detection over the tool calls the model actually made. +7. An evaluation pass for signals the model saw but did not file. +8. An assessment sweep for findings the model left unassessed. + +## Budget check first + +The budget check runs before context assembly and before a chat service is +acquired. A run that cannot afford to complete is stopped at the cheapest +possible point rather than after the expensive work. + +## Guest intelligence + +Discovery and reachability probing run before the seed context is built, so +the context the model receives already reflects which guests are reachable. +Reachability results are kept, because they feed the deterministic detection +stage later in the run. + +## Deterministic triage and seed context + +Triage narrows the surface first, and the seed context is then built from what +triage returned rather than from everything Pulse knows. This keeps the +model's starting context focused on what looks interesting. + +Quiet infrastructure still goes through the configured model. Patrol does not +skip the model when nothing looks wrong, because "nothing is wrong" is itself +a judgement worth making explicitly. + +The turn budget for the loop is computed from the size of the surface. See +`computePatrolMaxTurns` and `computeTriageMaxTurns` in +`internal/ai/patrol_ai.go`. + +## The agentic loop + +The model runs an agentic loop with tool access. Findings are created through +tool calls during the loop rather than parsed out of prose afterwards, so a +finding exists because the model deliberately filed it. + +## Deterministic signal detection + +After the loop completes, Patrol collects the tool calls that ran and scans +their results for known problem signals. This scan is deterministic and does +not depend on model judgement at all. The entry point is `DetectSignals` in +`internal/ai/patrol_signals.go`. + +The signal types are: + +| Signal | Meaning | +|---|---| +| `smart_failure` | A disk reporting SMART failure | +| `high_cpu` | Sustained CPU above the threshold | +| `high_memory` | Sustained memory above the threshold | +| `high_disk` | Disk or pool usage above the threshold | +| `backup_failed` | A backup job that failed | +| `backup_stale` | No backup inside the staleness window | +| `guest_unreachable` | A guest that failed reachability probing | + +Thresholds come from your own alert configuration rather than from a separate +set of numbers, so detection agrees with what Pulse would alert on. +`SignalThresholdsFromPatrol` maps your configured alert thresholds onto the +signal thresholds, falling back per field to these defaults when a value is +not configured. + +| Threshold | Default | +|---|---| +| Storage warning | 75% | +| Storage critical | 95% | +| High CPU | 70% | +| High memory | 80% | +| Backup staleness | 48 hours | + +Backup staleness has no user-facing setting and always uses the default. + +Reachability signals detected during the guest intelligence stage are merged +into the same set at this point. + +## The evaluation pass + +Detected signals are compared against the findings the model actually filed. +Any signal with no corresponding finding is an unmatched signal, meaning the +data showed a problem and the model did not report it. + +Unmatched signals go to a bounded evaluation pass scoped to exactly those +signals. This is the safety net. It is why a run can still surface a failing +disk when the model overlooked one in a large sweep. + +## The assessment sweep + +Smaller models sometimes finish the main pass without recording an assessment +for every finding they filed. Patrol detects the missing assessments and runs +one bounded follow-up pass scoped to exactly those findings, rather than +rerunning the whole analysis or leaving the findings unassessed. + +## Investigation + +A finding may be investigated after it is filed. Investigation is a separate +loop from the run that produced the finding, coordinated through the +`InvestigationOrchestrator` contract in `pkg/aicontracts/interfaces.go`. + +Whether a finding is investigated at all depends on your autonomy level and on +the finding's own state. `Finding.ShouldInvestigate` in +`internal/ai/findings.go` is the gate, and it refuses in every one of these +cases. + +- Autonomy is unset or `monitor`. Investigation only runs at `approval`, + `assisted`, or `full`. +- The finding is resolved, dismissed, suppressed, or snoozed. +- The severity is not warning or critical. Info and watch findings are not + investigated. +- A fix is already queued for approval, so the next move belongs to you. +- The finding reached a terminal outcome, such as fix verified, resolved, fix + rejected, cannot fix, or needs attention. +- The finding has already been investigated three times. +- The last investigation was inside the cooldown, which is one hour by + default and shorter for timeout failures, since those are transient. + +Patrol also recovers investigations that were left running and retries ones +that timed out. See `recoverStuckInvestigations` and +`retryTimedOutInvestigations` in `internal/ai/patrol_findings.go`. + +## Why it is built this way + +Model judgement is good at explaining and correlating, and unreliable at +exhaustive enumeration. Deterministic detection is the reverse. Patrol uses +each for what it is good at, and the unmatched-signal check is the seam where +the deterministic half can correct the model. + +## Related reading + +- [AI features](AI.md) for the overview and configuration. +- [Patrol autonomy](AI_AUTONOMY.md) for what each autonomy level permits. +- [Patrol qualification](AI_PATROL_QUALIFICATION.md) for the evidence and + release-claim rules. diff --git a/frontend-modern/public/docs/PULSE_PRO.md b/frontend-modern/public/docs/PULSE_PRO.md index b9f0a812e..e2f901bb0 100644 --- a/frontend-modern/public/docs/PULSE_PRO.md +++ b/frontend-modern/public/docs/PULSE_PRO.md @@ -2,8 +2,6 @@ This document explains Pulse's user-facing plan structure, the locked self-hosted commercial model, and how those plans map to runtime feature gates. -For the canonical, code-aligned entitlement table (including internal tier names), see: -- `docs/architecture/ENTITLEMENT_MATRIX.md` ## Plan Mapping (User-Facing -> Code Tiers) @@ -134,7 +132,7 @@ Legend: - Included: `Y` / `N` - `Y*`: Enterprise/custom only (`enterprise` tier or explicit entitlement) -This matrix is derived from the canonical table in `docs/architecture/ENTITLEMENT_MATRIX.md` plus runtime history/limit semantics exposed through entitlements. +This matrix reflects the entitlement keys enforced in code plus the runtime history and limit semantics exposed through entitlements. | Constant | Capability Key | Display Name | Community | Relay | Pro | Cloud | Primary Gating Mechanism / Notes | |---|---|---|:---:|:---:|:---:|:---:|---| @@ -221,6 +219,6 @@ This returns a feature map including keys like `relay`, `ai_alerts`, `ai_autofix ## Deep Dives -- [Pulse Patrol Deep Dive](architecture/pulse-patrol-deep-dive.md) -- [Pulse Assistant Deep Dive](architecture/pulse-assistant-deep-dive.md) +- [Pulse Patrol Deep Dive](PATROL_ARCHITECTURE.md) +- [Pulse Assistant Deep Dive](ASSISTANT_ARCHITECTURE.md) - [Pulse Intelligence overview](AI.md) diff --git a/frontend-modern/public/docs/UPGRADE_v6.md b/frontend-modern/public/docs/UPGRADE_v6.md index 87c362e02..147c87884 100644 --- a/frontend-modern/public/docs/UPGRADE_v6.md +++ b/frontend-modern/public/docs/UPGRADE_v6.md @@ -384,4 +384,3 @@ Multi-tenant mode is opt-in and additionally license-gated: - Enablement flag: `PULSE_MULTI_TENANT_ENABLED=true` - Capability gate: `multi_tenant` -See any multi-tenant operational docs under `docs/architecture/` if you plan to run this mode.