Pulse/docs/ALERT_ENGINE_EVOLUTION.md

# Alert Engine Evolution

Status: Active
Date: 2026-08-26
Scope: `pulse` only
Predecessor: `docs/CANONICAL_ALERT_ENGINE_MIGRATION_2026-03-10.md`

## Purpose

The March 2026 canonical migration correctly rejected a full alert-engine
rewrite and began moving alert *inputs* onto canonical identity
(`internal/unifiedresources`) and specs (`internal/alerts/specs`). It
deliberately froze two layers: transition state stays in the manager's
tracking maps, and notification-suppression rules stay where they are.

The post-March issue record shows the recurring regression classes live in
exactly those frozen layers: config saves emitting resolved notifications
(#1682), grouping silently dropping N−1 of N alerts (#1683), resolves sent
for suppressed firings (#1553), self-resolving fire/resolve churn deleting
history (#1693), chronic re-fires on terminal container state (#1724).
Cleaner inputs feeding the same fragile core does not retire these classes.

This document extends the migration's end-state one layer deeper. The
mechanism stays the same — strangler migration inside one engine, no
big-bang rewrite — applied to the transition core and the suppression path.

## Target model (delta over the March doc)

The March doc's layers 1 (unified resource identity), 2 (canonical specs),
and 5 (notification fan-out boundary in `monitor_alerts.go`) stand
unchanged. Two layers are added or replaced:

### Event log (new, Phase 0)

An append-only alert event log, SQLite-backed, owned by `internal/alerts`.
Every lifecycle transition (pending, firing, resolved, acknowledged,
escalated) and every notification decision — including suppressions, with
the mechanism and reason that held them — is an event. History, the
delivery log, frequency analytics, and `AlertDeliveryDiagnosis` become
projections of one log instead of separately maintained structures.
Transparency stops being a feature and becomes a property of the
architecture.

Phase 0 is strictly additive: the existing manager writes events alongside
its current behavior. No lifecycle semantics change.

### Incident reducer (replaces the frozen transition core, Phases 1–2)

A pure transition function — `next(state, signal, rules, clock) →
(state', events)` — replaces the manager's string-keyed tracking maps.
Identity is computed in exactly one place (canonical resource ID + spec
ID); lifecycle is an explicit state machine; acknowledgement, snooze, and
flapping are typed fields on one incident record. Because the reducer is
pure, the lifecycle contract suite (config save ⇒ zero events; suppressed
firing ⇒ no resolve; N grouped ⇒ N rendered) runs exhaustively and gates
releases.

The March doc's "stable behaviors that must not change" list is the seed
of the reducer's characterization suite: ack preservation, cooldown
re-notify, escalation carry-over, and confirmation semantics are pinned by
parity tests against the live manager before any cutover, not rewritten
from memory.

## Phases

- **Phase 0 — event log, additive.** Event store + manager wiring for
  transition and suppression events + API + UI surfacing of the existing
  delivery diagnosis and held-notification rows. Zero lifecycle risk.
  Registered as coverage gap `alert-engine-suppression-observability`.
- **Phase 1 — reducer in shadow mode.** Built beside the manager, fed the
  same inputs, outputs diffed continuously (the
  `unified_eval_parity_test.go` pattern). No user-visible change.
- **Phase 2 — family-by-family cutover.** Worst identity offenders first
  (Docker, PBS/storage). Each family cutover deletes its tracking maps and
  the imperative bulk of its `Check*` path. Alert IDs and override keys
  stay stable until an explicit config migration.
  **Status: complete (2026-08-27).** All four per-observation transition
  families run on the reducer core as the authoritative state:
  match-spec lifecycle (`evaluateCanonicalLifecycleAlert`), legacy
  `checkMetric`, canonical metric threshold, and the stateful family.
  The legacy tracking maps (`offlineConfirmations`,
  `offlineRecoveryConfirmations`, `nodeOfflineCount`,
  `connectionDegradedCount`, `dockerOfflineCount`, `dockerStateConfirm`,
  `pendingAlerts`) are deleted; their cleanup-loop hygiene moved into the
  core (`PruneStalePending`, pending reaping on container removal,
  healthy observations from the online handlers).
  *Deliberately scoped out:* the unified-incidents reconciler
  (`unifiedIncident*` in the manager). It is a batch reconciliation
  family, not a per-observation confirmation run — its provider hands it
  complete incident sets and its `firstSeen` bookkeeping is already
  identity-correct — so forcing it through the per-signal reducer would
  add translation without retiring a defect class. It stays a follow-on:
  if the event log becomes the sole history authority (Phase 3+), it
  should emit reducer-shaped events at that boundary instead.
- **Phase 3 — declarative rule model.** Scope selector + condition +
  policy replaces the per-platform config blocks and `DisableAll*`
  booleans, with a translator from the existing `AlertConfig` so persisted
  user configs keep working. UI migrates tab by tab.
  **Status: engine side complete (2026-08-27).** The effective alert
  policy for a resource is answered by one ordered fold
  (`internal/alerts/alert_policy.go`): type default block → the type's
  `DisableAll` switches → custom rules (guest-scoped, priority order) →
  the per-resource override through the identity-aware lookup for the
  kind. The persisted `AlertConfig` is the translator's input and keeps
  its shape; the engine no longer reads it piecemeal — every scattered
  `DisableAll*` read and threshold lookup routes through the fold, pinned
  by characterization tests against the legacy resolution before any call
  site moved.
  *UI migration: deliberately deferred, demand-gated.* The per-platform
  thresholds presentation is what users ask in — "VM CPU threshold", not
  "scope selector" — and the demand ledger holds no signal for a
  rules-first editing surface (the nearest entry warns against bolting
  schedules onto alert rules). The tabs now sit on a single resolution
  surface, so a rules-first UI, an effective-policy inspector ("why is
  this alert off?"), or both can be built without further engine work
  when a signal lands. Re-scope through
  `repos/pulse-pro/FEATURE_REQUESTS.md` at that point.

## What is kept

The delivery pipeline (`internal/notifications` queue, DLQ, receipts,
templates, delivery health), the operator-intent fabric, unified
resources, the specs/evaluator layer, the UI shell, and the
`monitor_alerts.go` fan-out boundary — which becomes an event consumer.

## What is retired (by the end of Phase 2–3)

The manager's per-family tracking maps, the imperative per-platform
check bodies, the scattered suppression checks, and — once the event log
is authoritative — the JSON snapshot history file.

Status (2026-08-27): the tracking maps are deleted; the check bodies'
imperative transition bulk is replaced by the reducer core (what remains
of them is evidence assembly for the spec evaluator); suppression
decisions emit typed, reasoned events through the Phase 0 log; policy
reads go through the Phase 3 fold.

Post-plan hardening (2026-08-27, same-day follow-through):

- **The lifecycle contract suite is a release gate.** Engine contracts
  (config-save silence, resolve-exactly-once, ack halts escalation,
  acknowledged recovery suppressed) in
  `internal/alerts/lifecycle_contract_test.go`; grouped-rendering
  contracts (N grouped ⇒ N rendered on email, apprise, and every
  webhook service template; cancel-before-delivery holds the recovery)
  in `internal/notifications/grouped_rendering_contract_test.go`.
- **The event log is the alert history authority.** Lifecycle events
  carry full alert snapshots; history is projected from the log
  (`history_projection.go`), the legacy JSON entries migrate in once
  and the files retire to `*.imported`, clears are append-only
  tombstones, and every alert family — reconciler and system alerts
  included — records its fired event. The JSON snapshot history file is
  retired. Parity with the JSON model is pinned by
  `history_projection_parity_test.go`.
- **The identity/config migration is complete.** Alert configuration now
  carries an additive identity schema version and the monitor runs a
  non-mutating, fail-closed migration plan against the live resource registry
  before persisting it. Proven guest, guest-disk, storage, Docker, and
  provider-declared succession keys move to their single write identity;
  ambiguous, conflicting, unknown, and temporarily absent rows are retained.
  Active-alert snapshots (including acknowledgement fields) are rewritten with
  canonical state IDs after restore. The planner is idempotent across rollback
  and re-upgrade, rejects future schema versions, and the frontend round-trips
  the marker on ordinary configuration saves.