docs: add comprehensive deep-dive documentation for AI subsystems

Adds detailed architecture documentation for Pulse Patrol and Pulse Assistant. Updates AI.md and PULSE_PRO.md. Also includes additional tests.
This commit is contained in:
rcourtman 2026-02-02 10:29:07 +00:00
parent df43f08cf2
commit fa1b74792e
22 changed files with 4322 additions and 203 deletions

View file

@ -2,58 +2,75 @@
Pulse Patrol is available to everyone with BYOK (your own AI provider). Pulse Pro unlocks auto-fix and advanced analysis. Learn more at <https://pulserelay.pro> or see the technical overview in [PULSE_PRO.md](PULSE_PRO.md).
## What Patrol Actually Does (Technical)
---
## Overview
Pulse includes two AI-powered systems:
1. **Pulse Assistant** — An interactive chat interface for ad-hoc troubleshooting, investigations, and infrastructure control.
2. **Pulse Patrol** — A scheduled, context-aware analysis service that continuously monitors your infrastructure, learns what's normal, predicts issues, and generates actionable findings.
Both systems are built on the same tool-driven architecture where the LLM acts as a proposer and Go code enforces safety gates.
### Not Just Another Chatbot
Pulse Assistant is a **protocol-driven, safety-gated agentic system** that:
- **Proactively gathers context** — understands resources before you ask (context prefetcher)
- **Learns within sessions** — extracts and caches facts to avoid redundant queries (knowledge accumulator)
- **Enforces workflow invariants** — FSM prevents dangerous state transitions
- **Supports parallel tool execution** — efficient batch operations with concurrency control
- **Detects and prevents hallucinations** — phantom execution detection
- **Auto-recovers from errors** — structured error envelopes enable self-correction
📖 **For a deep technical dive into the Assistant architecture, see [architecture/pulse-assistant-deep-dive.md](architecture/pulse-assistant-deep-dive.md).**
### Not Just Another Alerting System
Pulse Patrol is a **multi-layered intelligence platform** that:
- **Learns** what's normal for your environment (baseline engine)
- **Predicts** issues before they become critical (pattern detection + forecasting)
- **Correlates** events across your entire infrastructure (root cause analysis)
- **Remembers** past incidents and successful remediations (incident memory)
- **Investigates** issues autonomously when configured (investigation orchestrator)
- **Verifies** fixes and tracks remediation effectiveness (verification loops)
All while running entirely on your infrastructure with BYOK for complete privacy.
📖 **For a deep technical dive into the intelligence subsystems, see [architecture/pulse-patrol-deep-dive.md](architecture/pulse-patrol-deep-dive.md).**
See [architecture/pulse-assistant.md](architecture/pulse-assistant.md) for the original safety architecture documentation.
---
## Pulse Patrol
Patrol is a scheduled analysis pipeline that builds a rich, system-wide snapshot and produces actionable findings.
**Inputs (real data, not guesses):**
- Live state: nodes, VMs/CTs, storages, backups, docker, kubernetes.
- Recent metrics history (trends + rate of change).
- Alert state and recent failures.
- Diagnostics (connectivity, permissions, agent status).
### How Patrol Works
**Enrichment:**
- Normalizes metrics across resource types.
- Applies context rules (e.g., ignore idle test nodes, de-noise transient blips).
- Correlates related issues to avoid duplicate noise.
```
Scheduled/Event Trigger
│
▼
buildSeedContext() ── infrastructure snapshot
│
▼
LLM analysis (with tools) ← pulse_storage, pulse_metrics, pulse_alerts, etc.
│
▼
DetectSignals() ── deterministic signal detection from tool outputs
│
▼
createFinding() ── validated, deduplicated findings stored
│
▼ (if configured)
MaybeInvestigateFinding() ── automatic investigation + remediation
```
**Outputs:**
- Findings with severity, category, and remediation hints.
- Summary stats for coverage and risk level.
- Optional auto-fix and remediation workflows (Pro).
## Why Patrol Is Different From Traditional Alerts
Alerts are threshold-based and narrow. Patrol is context-based and cross-system.
- **Alerts**: "Disk > 90%."
- **Patrol**: "ZFS pool is 86% but trending +4%/day; projected to hit 95% within a week. Largest consumer is datastore X. Recommend prune or expand."
## Examples of Patrol Findings (Realistic)
- Backup jobs succeeded but datastore usage jumped 20% since last run.
- Node health is OK but cluster clock drift is growing.
- Multiple VMs in the same storage pool are all retrying snapshots.
- Docker host is healthy but containers in one stack are flapping.
## Controls and Limits
- **Schedule**: from 10 minutes to 7 days (default 6 hours).
- **Scope**: only configured resources and connected agents.
- **Safety**: command execution remains disabled by default.
- **Cost control**: BYOK only — use provider limits and budgets to control spend.
## Privacy
Patrol runs on your server and only sends the minimal context needed for analysis to the configured provider (when AI is enabled). No telemetry is sent to Pulse by default.
Pulse AI adds an optional assistant for troubleshooting and proactive monitoring. It is **off by default** and can be enabled per instance.
## What Makes AI Patrol Different
Unlike chatting with a generic AI where you manually describe your infrastructure, Patrol runs automatically and sees **your entire infrastructure at once** - every node, VM, container, storage pool, backup job, and Kubernetes cluster. It's not just a static checklist; it's an LLM analyzing real-time data enriched with historical context.
### Context Patrol Receives (That Generic LLMs Can't See)
### What Patrol Sees
Every patrol run passes the LLM comprehensive context about your environment:
@ -68,23 +85,37 @@ Every patrol run passes the LLM comprehensive context about your environment:
| **Ceph** | Cluster health, OSD states, PG status |
| **Agent Hosts** | Load averages, memory, disk, RAID status, temperatures |
### Enriched Context (The Real Differentiator)
### Enriched Context
Beyond raw metrics, Patrol enriches the context with intelligence that transforms raw data into actionable insights:
Beyond raw metrics, Patrol enriches the context with intelligence:
- **Trend analysis** - 24h and 7d patterns showing `growing`, `stable`, `declining`, or `volatile` behavior
- **Learned baselines** - Z-score anomaly detection based on what's *normal for your environment*
- **Capacity predictions** - "Storage pool will be full in 12 days at current growth rate"
- **Infrastructure changes** - Detected config changes, VM migrations, new deployments
- **Resource correlations** - Pattern detection across related resources (e.g., containers on same host)
- **User notes** - Your annotations explaining expected behavior ("runs hot for transcoding")
- **Dismissed findings** - Respects your feedback and suppressed alerts
- **Incident memory** - Learns from past investigations and successful remediations
- **Trend analysis** — 24h and 7d patterns showing `growing`, `stable`, `declining`, or `volatile` behavior
- **Learned baselines** — Z-score anomaly detection based on what's *normal for your environment*
- **Capacity predictions** — "Storage pool will be full in 12 days at current growth rate"
- **Infrastructure changes** — Detected config changes, VM migrations, new deployments
- **Resource correlations** — Pattern detection across related resources
- **User notes** — Your annotations explaining expected behavior
- **Dismissed findings** — Respects your feedback and suppressed alerts
- **Incident memory** — Learns from past investigations and successful remediations
### Deterministic Signal Detection
Patrol doesn't rely solely on LLM judgment. It parses tool call outputs and fires deterministic signals for known problems:
| Signal Type | Trigger | Default Threshold |
|------------|---------|-------------------|
| `smart_failure` | SMART health status not OK/PASSED | N/A |
| `high_cpu` | Average CPU usage | 70% |
| `high_memory` | Average memory usage | 80% |
| `high_disk` | Storage pool usage | 75% (warning), 95% (critical) |
| `backup_failed` | Recent backup task with error status | Within 48h |
| `backup_stale` | No backup completed for VM/CT | 48+ hours |
| `active_alert` | Critical/warning alert in list | N/A |
Thresholds can be configured via alert settings to match user-defined values.
### Examples of What Patrol Catches
Because it's an LLM with full context, Patrol catches issues that static threshold-based alerting misses:
| Issue | Severity | Example |
|-------|----------|---------|
| **Node offline** | Critical | Proxmox node not responding |
@ -95,9 +126,7 @@ Because it's an LLM with full context, Patrol catches issues that static thresho
| **Storage issues** | Critical | PBS datastore errors, ZFS pool degraded |
| **Ceph problems** | Warning/Critical | Degraded OSDs, unhealthy PGs |
| **Kubernetes issues** | Warning | Pods stuck in Pending/CrashLoopBackOff |
| **Restart loops** | Warning | VMs that keep restarting without errors |
| **Clock drift** | Warning | Node time drift affecting Ceph/HA |
| **Unusual patterns** | Varies | Any anomaly the LLM identifies as unusual for your setup |
| **SMART failures** | Critical | Disk health check failed |
### What Patrol Ignores (by design)
@ -111,55 +140,156 @@ Patrol is **intentionally conservative** to avoid noise:
> **Philosophy**: If a finding wouldn't be worth waking someone up at 3am, Patrol won't create it.
## Features
### Finding Severity
- **Interactive chat**: Ask questions about current cluster state and get AI-assisted troubleshooting.
- **Patrol**: Background checks periodically (default: 6 hours) that generate findings. Interval is configurable from 10 minutes to 7 days, or set to 0 to disable.
- **Alert-triggered analysis (Pro)**: Optional token-efficient analysis when alerts fire.
- **Kubernetes AI analysis (Pro)**: Deep cluster analysis beyond basic monitoring.
- **Command execution**: When enabled, AI can run commands via connected agents.
- **Finding management**: Dismiss, resolve, or suppress findings to prevent recurrence.
- **Cost tracking**: Tracks token usage and supports monthly budget limits.
- **Critical**: Immediate attention required (service down, data at risk)
- **Warning**: Should be addressed soon (disk filling, backup stale)
Alert-triggered analysis runs attach a timeline event to the alert, so investigations remain auditable alongside acknowledgements and remediation steps.
Note: `info` and `watch` level findings are filtered out to reduce noise.
> **License note**: Kubernetes AI analysis is gated by the `kubernetes_ai` Pulse Pro feature.
### Managing Findings
## Pulse Assistant (Chat): How It Works
Findings can be managed via the UI or API:
Pulse Assistant is **tool-driven**. It does not "guess" system state — it calls live tools and reports their outputs.
- **Get help**: Chat with AI to troubleshoot the issue
- **Resolve**: Mark as fixed (finding will reappear if the issue resurfaces)
- **Dismiss**: Mark as expected behavior (creates suppression rule)
Dismissed and resolved findings persist across Pulse restarts.
---
## Autonomy Levels
Patrol supports three autonomy modes that control how much action it can take:
| Mode | Behavior | License |
|------|----------|---------|
| **Monitor** | Detect issues only. No investigation or fixes. | Free (BYOK) |
| **Investigate** | Investigates findings and proposes fixes. All fixes require approval before execution. | Free (BYOK) |
| **Auto-fix** | Automatically fixes issues and verifies results. Critical findings still require approval by default. | Pro |
### Investigation Flow
When a finding is created in Investigate or Auto-fix mode:
```
Finding created
│
▼
MaybeInvestigateFinding()
│
├─ Has orch + chatService?
│ │
│ ▼
│ InvestigateFinding()
│ │
│ ▼
│ Create chat session
│ │
│ ▼
│ AI analysis (with tools)
│ │
│ ▼
│ [Fix proposed?] ──Yes──► Queue approval (or auto-execute in full mode)
│ │
│ No
│ ▼
│ Update finding with outcome
│
└─ Skip investigation
```
### Investigation Configuration
| Setting | Default | Description |
|---------|---------|-------------|
| `MaxTurns` | 15 | Maximum agentic turns per investigation |
| `Timeout` | 10 min | Maximum duration per investigation |
| `MaxConcurrent` | 3 | Maximum concurrent investigations |
| `MaxAttemptsPerFinding` | 3 | Maximum investigation attempts per finding |
| `CooldownDuration` | 1 hour | Cooldown before re-investigating |
| `TimeoutCooldownDuration` | 10 min | Shorter cooldown for timeout failures |
| `VerificationDelay` | 30 sec | Wait before verifying fix |
### Investigation Outcomes
| Outcome | Meaning |
|---------|---------|
| `resolved` | Issue resolved during investigation |
| `fix_queued` | Fix proposed, awaiting approval |
| `fix_executed` | Fix auto-executed successfully |
| `fix_failed` | Fix attempted but failed |
| `fix_verified` | Fix worked, issue confirmed resolved |
| `fix_verification_failed` | Fix ran but issue persists |
| `needs_attention` | Requires human intervention |
| `cannot_fix` | Issue cannot be automatically fixed |
| `timed_out` | Investigation timed out (will retry sooner) |
---
## Pulse Assistant (Chat)
Pulse Assistant is a **tool-driven** chat interface. It does not "guess" system state — it calls live tools and reports their outputs.
### The Model's Workflow (Discover → Investigate → Act)
- **Discover**: Uses `pulse_query` (or `pulse_discovery`) to find real resources and IDs.
- **Investigate**: Uses `pulse_read` to run bounded, read-only commands and check status/logs.
- **Act** (optional): Uses `pulse_control` for changes, then verifies with a read.
### Safety Gates That Make It Trustworthy
- **Strict Resolution (optional)**: When enabled, the assistant must discover a resource before it can act on it. This prevents fabricated IDs.
- **Read/Write separation**: Read-only commands go through `pulse_read`; write actions go through `pulse_control`. This keeps the workflow state machine honest.
- **Verification after writes**: After any write, the assistant must perform a read check before it can finish the response.
- **Non‑interactive guardrails**: Commands that could hang (e.g., `tail -f`) are rewritten into bounded, safe forms.
- **Approval mode**: In Controlled mode, every write requires explicit user approval. Autonomous mode is available only with Pro.
1. **Discover**: Uses `pulse_query` or `pulse_discovery` to find real resources and IDs
2. **Investigate**: Uses `pulse_read` to run bounded, read-only commands and check status/logs
3. **Act** (optional): Uses `pulse_control` for changes, then verifies with a read
### What You See As a User
- **Clear tool usage**: Each step shows which tool ran and what it returned.
- **Structured recovery**: If a tool is blocked, the assistant adapts (e.g., runs discovery, switches tools, or asks for approval).
- **Verified outcomes**: Changes are followed by a read check before the assistant claims success.
### Available Tools
## Why It's Impressive (and Reliable)
| Tool | Classification | Purpose |
|------|---------------|---------|
| `pulse_query`, `pulse_discovery` | Resolve | Resource discovery and query |
| `pulse_read` | Read | Read-only operations: exec, file, find, tail, logs |
| `pulse_metrics` | Read | Performance metrics |
| `pulse_storage` | Read | Storage information |
| `pulse_kubernetes` | Read | Kubernetes cluster info |
| `pulse_pmg` | Read | Proxmox Mail Gateway stats |
| `pulse_alerts` | Read/Write | Alert management (resolve/dismiss are writes) |
| `pulse_docker` | Read/Write | Docker operations (control/update are writes) |
| `pulse_knowledge` | Read/Write | Knowledge persistence (remember/note/save are writes) |
| `pulse_file_edit` | Read/Write | File operations (write/append are writes) |
| `pulse_control` | Write | Guest control, service management |
| `pulse_patrol` | Read | Patrol findings and status |
Pulse Assistant behaves like a careful operator:
- It **grounds answers in live data** instead of assumptions.
- It **adapts** when guardrails block an action.
- It **verifies** changes before reporting success.
- It **keeps you in control** with explicit approval gates.
### Safety Gates
The assistant enforces multiple safety gates:
1. **Discovery Before Action** — Action tools cannot operate on resources that weren't first discovered
2. **Verification After Write** — After any write, the model must perform a read/status check before providing a final answer
3. **Read/Write Separation** — Read operations route through `pulse_read` (stays in READING state); write operations route through `pulse_control` (enters VERIFYING state)
4. **Phantom Detection** — Detects when the model claims execution without tool calls
5. **Approval Mode** — In Controlled mode, every write requires explicit user approval
6. **Execution Context Binding** — Commands execute within the resolved resource's context, not on parent hosts
### Control Levels
| Level | Behavior | License |
|-------|----------|---------|
| **Read-only** | AI can observe and query data only | Free |
| **Controlled** | AI asks for approval before executing commands | Free |
| **Autonomous** | AI executes actions without prompting | Pro |
### Using Approvals (Controlled Mode)
When control level is **Controlled**, write actions pause for approval:
1. Tool returns `APPROVAL_REQUIRED: { approval_id, command, ... }`
2. Agentic loop emits `approval_needed` SSE event
3. UI shows approval card with the proposed command
4. **Approve** to execute and verify, or **Deny** to cancel
5. Only users with admin privileges can approve/deny
---
## Configuration
Configure in the UI: **Settings → System → AI Assistant**
AI settings are stored encrypted at rest in `ai.enc` under the Pulse config directory. Patrol findings and history are stored in `ai_findings.json`, `ai_patrol_runs.json`, and usage data in `ai_usage_history.json`. These files are located in `/etc/pulse` for systemd installs, or `/data` for Docker/Kubernetes.
### Supported Providers
- **Anthropic** (API key or OAuth)
@ -178,11 +308,74 @@ You can set separate models for:
- Patrol (`patrol_model`)
- Auto-fix remediation (`auto_fix_model`)
### Storage
AI settings are stored encrypted at rest in `ai.enc` under the Pulse config directory. Related files:
| File | Purpose |
|------|---------|
| `ai.enc` | Encrypted AI configuration and credentials |
| `ai_findings.json` | Patrol findings |
| `ai_patrol_runs.json` | Patrol run history |
| `ai_usage_history.json` | Token usage data |
| `ai_chat_sessions.json` | Legacy chat sessions (UI sync) |
| `baselines.json` | Learned resource baselines |
| `ai_correlations.json` | Resource correlation data |
| `ai_patterns.json` | Detected patterns |
Config directory: `/etc/pulse` (systemd) or `/data` (Docker/Kubernetes)
### Testing
- Test provider connectivity: `POST /api/ai/test` and `POST /api/ai/test/{provider}`
- List available models: `GET /api/ai/models`
---
## Schedule and Triggers
Patrol runs on a configurable schedule:
| Interval | Description |
|----------|-------------|
| Disabled | Patrol runs only when manually triggered |
| 10 min – 7 days | Configurable interval (default: 6 hours) |
Patrol can also be triggered by:
- **Manual run**: Click "Run Patrol" in the UI
- **Alert-triggered analysis (Pro)**: Runs when an alert fires
- **API call**: `POST /api/ai/patrol/run`
---
## AI Intelligence Layer
Pulse includes a unified intelligence system that aggregates data from all AI subsystems:
### Components
| Component | Purpose |
|-----------|---------|
| **Baseline Engine** | Learns normal behavior, detects anomalies via z-score |
| **Pattern Detector** | Identifies recurring issues and trends |
| **Correlation Engine** | Links related issues across resources |
| **Incident Memory** | Tracks past incidents and successful remediations |
| **Knowledge Store** | Persists user annotations and learned preferences |
| **Forecast Engine** | Predicts capacity issues and resource exhaustion |
### Health Scoring
Each resource receives a health score (A–F) based on:
- Current metrics vs baseline
- Active findings and alerts
- Recent incidents
- Trend direction (improving/stable/declining)
---
## Model Matrix (Pulse Assistant)
This table summarizes the most recent **Pulse Assistant** eval runs per model. Patrol is still in development and is not scored yet.
Time/tokens reflect the combined **Smoke + Read-only** matrix run.
Transient provider errors (rate limits, unavailable chat endpoints) are skipped when rendering the table.
This table summarizes the most recent **Pulse Assistant** eval runs per model.
Update the table from eval reports:
```
@ -206,75 +399,54 @@ scripts/eval/run_model_matrix.sh
| openai:gpt-5.2-chat-latest | ✅ | ✅ | 8s | 12,595 | 2026-01-29 |
<!-- MODEL_MATRIX_END -->
### Testing
- Test provider connectivity: `POST /api/ai/test` and `POST /api/ai/test/{provider}`
- List available models: `GET /api/ai/models`
## Patrol Service (BYOK)
Patrol runs automated health checks on a configurable schedule (default: every 6 hours). It passes comprehensive infrastructure context to the LLM (see "Context Patrol Receives" above) and generates findings when issues are detected.
Patrol runs with your configured provider (BYOK). Auto-fix and assisted/full autonomy require Pulse Pro.
### Finding Severity
- **Critical**: Immediate attention required (service down, data at risk)
- **Warning**: Should be addressed soon (disk filling, backup stale)
Note: `info` and `watch` level findings are filtered out to reduce noise.
### Managing Findings
Findings can be managed via the UI or API:
- **Get help**: Chat with AI to troubleshoot the issue
- **Resolve**: Mark as fixed (finding will reappear if the issue resurfaces)
- **Dismiss**: Mark as expected behavior (creates suppression rule)
Dismissed and resolved findings persist across Pulse restarts.
### AI-Assisted Remediation
When chatting with AI about a patrol finding, the AI can:
- Run diagnostic commands on connected agents
- Propose fixes with explanations
- Automatically resolve findings after successful remediation (Pro auto-fix)
---
## Safety Controls
Pulse includes settings that control how "active" AI features are:
- **Autonomous mode (Pro)**: When enabled, AI may execute safe commands without approval.
- **Patrol auto-fix (Pro)**: Allows patrol to attempt automatic remediation.
- **Alert-triggered analysis (Pro)**: Limits AI to analyzing specific events when alerts occur.
- **Autonomous mode (Pro)**: When enabled, AI may execute safe commands without approval
- **Patrol auto-fix (Pro)**: Allows patrol to attempt automatic remediation
- **Alert-triggered analysis (Pro)**: Limits AI to analyzing specific events when alerts occur
- **Full autonomy unlock (Pro)**: Enables auto-fix for critical findings without approval (requires explicit toggle)
If you enable execution features, ensure agent tokens and scopes are appropriately restricted.
### Control Levels
Pulse uses three AI permission levels for infrastructure control:
- **Read-only**: AI can observe and query data only.
- **Controlled**: AI asks for approval before executing commands or control actions.
- **Autonomous (Pro)**: AI executes actions without prompting.
### Using Approvals (Controlled Mode)
When control level is **Controlled**, write actions pause for approval:
- In chat, you’ll see an approval card with the proposed command.
- **Approve** to execute and verify the change, or **Deny** to cancel it.
- Only users with admin privileges can approve/deny.
### Advanced Network Restrictions
Pulse blocks AI tool HTTP fetches to loopback and link-local addresses by default. For local development, you can allow loopback targets:
Pulse blocks AI tool HTTP fetches to loopback and link-local addresses by default. For local development:
- `PULSE_AI_ALLOW_LOOPBACK=true`
Use this only in trusted environments.
---
## Privacy
Patrol runs on your server and only sends the minimal context needed for analysis to the configured provider (when AI is enabled). No telemetry is sent to Pulse by default.
---
## Why Patrol Is Different From Traditional Alerts
Alerts are threshold-based and narrow. Patrol is context-based and cross-system.
- **Alerts**: "Disk > 90%"
- **Patrol**: "ZFS pool is 86% but trending +4%/day; projected to hit 95% within a week. Largest consumer is datastore X. Recommend prune or expand."
---
## Cost Tracking
Pulse tracks token usage and costs:
- View usage summary: `GET /api/ai/cost/summary`
- Reset counters: `POST /api/ai/cost/reset` (admin)
- Set monthly budget limits in AI settings
---
## Troubleshooting
| Issue | Solution |
@ -282,4 +454,19 @@ Use this only in trusted environments.
| AI not responding | Verify provider credentials in **Settings → System → AI Assistant** |
| No execution capability | Confirm at least one agent is connected |
| Findings not persisting | Check Pulse has write access to `ai_findings.json` in the config directory |
| Too many findings | This shouldn't happen - please report if it does |
| Too many findings | This shouldn't happen — please report if it does |
| Investigation stuck | Check circuit breaker status at `/api/ai/circuit/status`; may auto-reset after cooldown |
| Model not available | Ensure provider API key is valid and model ID matches provider format |
## Related Documentation
### Deep Dives (Recommended for Technical Audiences)
- **[Pulse Assistant Deep Dive](architecture/pulse-assistant-deep-dive.md)** — Complete technical breakdown of the agentic architecture: context prefetching, knowledge accumulation, FSM enforcement, parallel execution, phantom detection, auto-recovery
- **[Pulse Patrol Deep Dive](architecture/pulse-patrol-deep-dive.md)** — Full intelligence layer documentation: baseline learning, pattern detection, forecasting, correlation analysis, incident memory, investigation orchestration
### Reference Documentation
- [Architecture: Pulse Assistant (Safety Gates)](architecture/pulse-assistant.md) — Detailed FSM states, tool protocol, and invariants
- [API Reference](API.md) — Complete API endpoint documentation
- [Pulse Pro](PULSE_PRO.md) — Pro features and licensing