Pulse/docs/PATROL_ARCHITECTURE.md
rcourtman 63fc89f701 Write the deep-dive docs the AI pages promised
AI.md, AI_AUTONOMY.md and PULSE_PRO.md linked nine times into
docs/architecture/, which .gitignore marks as not for public release. The
targets were never missing, they were deliberately unpublished, so every one
of those links was dead for anybody but the maintainer.

Write the three promised documents against the code and publish them under
docs/ where the rest of the shipped set lives.

PATROL_ARCHITECTURE.md covers a run end to end. The interesting part is that
deterministic signal detection runs after the model, not before, so unmatched
signals catch what the model failed to file. Signal types, the thresholds
those signals derive from your own alert settings, and every condition in
Finding.ShouldInvestigate are documented from internal/ai/patrol_signals.go
and internal/ai/findings.go.

ASSISTANT_SAFETY.md documents the session state machine in
internal/ai/chat/fsm.go, its four states, the tool classification it runs on,
and its invariants. No write without a validated target, no second write
before the first is verified, no final answer about an unverified change, and
no attempt count that wears the gate down.

ASSISTANT_ARCHITECTURE.md covers the agentic loop around that machine, the
three-phase pipeline, why only execution parallelises and at what cap, the
read-before-write batch that must stay ordered, the look-before-asking gate
and its bound, and the stable error codes.

Also removed two older pointers into the same private directory, in API.md
and UPGRADE_v6.md, and the two references to ENTITLEMENT_MATRIX.md in
PULSE_PRO.md. That file exists locally and is a monetization document, so
publishing it is not a documentation decision.

Unresolvable intra-doc links are now 2 of 218, both internal release-control
documents deliberately withheld from the shipped set.

Contract-Neutral: documentation only
2026-08-03 22:20:57 +01:00

6.2 KiB

Pulse Patrol deep dive

How a Patrol run actually works, for readers who want more than the overview in AI features.

The short version is that Patrol does not trust the model to notice everything. A run pairs model judgement with deterministic detection, and the deterministic half acts as a safety net over the model's output rather than as an input to it.

A run, end to end

A run moves through the following stages. Each is described in more detail below.

  1. A pre-flight budget check, which fails fast before any context is built.
  2. Guest intelligence gathering, covering discovery and reachability probing.
  3. Deterministic triage, which narrows the surface the model will look at.
  4. Seed context construction from the triage result.
  5. The agentic loop, where the model calls tools and files findings.
  6. Deterministic signal detection over the tool calls the model actually made.
  7. An evaluation pass for signals the model saw but did not file.
  8. An assessment sweep for findings the model left unassessed.

Budget check first

The budget check runs before context assembly and before a chat service is acquired. A run that cannot afford to complete is stopped at the cheapest possible point rather than after the expensive work.

Guest intelligence

Discovery and reachability probing run before the seed context is built, so the context the model receives already reflects which guests are reachable. Reachability results are kept, because they feed the deterministic detection stage later in the run.

Deterministic triage and seed context

Triage narrows the surface first, and the seed context is then built from what triage returned rather than from everything Pulse knows. This keeps the model's starting context focused on what looks interesting.

Quiet infrastructure still goes through the configured model. Patrol does not skip the model when nothing looks wrong, because "nothing is wrong" is itself a judgement worth making explicitly.

The turn budget for the loop is computed from the size of the surface. See computePatrolMaxTurns and computeTriageMaxTurns in internal/ai/patrol_ai.go.

The agentic loop

The model runs an agentic loop with tool access. Findings are created through tool calls during the loop rather than parsed out of prose afterwards, so a finding exists because the model deliberately filed it.

Deterministic signal detection

After the loop completes, Patrol collects the tool calls that ran and scans their results for known problem signals. This scan is deterministic and does not depend on model judgement at all. The entry point is DetectSignals in internal/ai/patrol_signals.go.

The signal types are:

Signal Meaning
smart_failure A disk reporting SMART failure
high_cpu Sustained CPU above the threshold
high_memory Sustained memory above the threshold
high_disk Disk or pool usage above the threshold
backup_failed A backup job that failed
backup_stale No backup inside the staleness window
guest_unreachable A guest that failed reachability probing

Thresholds come from your own alert configuration rather than from a separate set of numbers, so detection agrees with what Pulse would alert on. SignalThresholdsFromPatrol maps your configured alert thresholds onto the signal thresholds, falling back per field to these defaults when a value is not configured.

Threshold Default
Storage warning 75%
Storage critical 95%
High CPU 70%
High memory 80%
Backup staleness 48 hours

Backup staleness has no user-facing setting and always uses the default.

Reachability signals detected during the guest intelligence stage are merged into the same set at this point.

The evaluation pass

Detected signals are compared against the findings the model actually filed. Any signal with no corresponding finding is an unmatched signal, meaning the data showed a problem and the model did not report it.

Unmatched signals go to a bounded evaluation pass scoped to exactly those signals. This is the safety net. It is why a run can still surface a failing disk when the model overlooked one in a large sweep.

The assessment sweep

Smaller models sometimes finish the main pass without recording an assessment for every finding they filed. Patrol detects the missing assessments and runs one bounded follow-up pass scoped to exactly those findings, rather than rerunning the whole analysis or leaving the findings unassessed.

Investigation

A finding may be investigated after it is filed. Investigation is a separate loop from the run that produced the finding, coordinated through the InvestigationOrchestrator contract in pkg/aicontracts/interfaces.go.

Whether a finding is investigated at all depends on your autonomy level and on the finding's own state. Finding.ShouldInvestigate in internal/ai/findings.go is the gate, and it refuses in every one of these cases.

  • Autonomy is unset or monitor. Investigation only runs at approval, assisted, or full.
  • The finding is resolved, dismissed, suppressed, or snoozed.
  • The severity is not warning or critical. Info and watch findings are not investigated.
  • A fix is already queued for approval, so the next move belongs to you.
  • The finding reached a terminal outcome, such as fix verified, resolved, fix rejected, cannot fix, or needs attention.
  • The finding has already been investigated three times.
  • The last investigation was inside the cooldown, which is one hour by default and shorter for timeout failures, since those are transient.

Patrol also recovers investigations that were left running and retries ones that timed out. See recoverStuckInvestigations and retryTimedOutInvestigations in internal/ai/patrol_findings.go.

Why it is built this way

Model judgement is good at explaining and correlating, and unreliable at exhaustive enumeration. Deterministic detection is the reverse. Patrol uses each for what it is good at, and the unmatched-signal check is the seam where the deterministic half can correct the model.