Pulse/docs/CANONICAL_ALERT_ENGINE_MIGRATION_2026-03-10.md

## v6 Canonical Alert-Engine Migration

Status: Superseded in scope by `docs/ALERT_ENGINE_EVOLUTION.md` (2026-08-26)
Date: 2026-03-10
Scope: `pulse` only

> **Supersession note (2026-08-26).** This document's input-layer direction
> (canonical identity, specs, the `monitor_alerts.go` fan-out boundary)
> stands. Its two freezes do **not**: the transition core and the
> suppression path are being replaced by the deterministic reducer
> (`internal/alerts/reducer`) per `docs/ALERT_ENGINE_EVOLUTION.md`, and
> "reuse `checkMetric`/`dispatchAlert` mechanics, don't replace them" is no
> longer current direction. The "Stable Behaviors" list below is not a
> freeze: it is a characterization inventory, now pinned as executable
> parity scenarios (`internal/alerts/reducer_parity_*_test.go`). Listed
> behaviors may change deliberately — update the pinned scenario first,
> then the implementation, and ship a migration when persisted user state
> (alert IDs, override keys, acks) is affected. Do not treat this file as
> authority for new work.

## Purpose

This document defines the canonical v6 alert-engine target for this repo and the migration path inside the current codebase.

It is intentionally not a rewrite plan. The current system already has the pieces that matter:

- `internal/unifiedresources` already owns canonical resource identity, parent-child relationships, metrics targets, and provider-native incidents.
- `internal/alerts/unified_eval.go` already centralizes threshold-based metric evaluation for unified resources.
- `internal/alerts/unified_incidents.go` already mirrors canonical resource incidents into the live alert manager.
- `internal/monitoring/monitor_alert_sync.go` already syncs unified incidents into shared alert state.
- `internal/monitoring/monitor_alerts.go` already owns the downstream fan-out to tenant websocket broadcast, notifications, incident recording, and AI callbacks.

What is still legacy is responsibility placement: most `Check*` methods in `internal/alerts/alerts.go` still mix collection, threshold resolution, transition rules, alert construction, and notification entry.

## Canonical Target Model

The target model keeps one alert engine and splits it into explicit layers.

### 1. Unified Resource Identity

Source of truth: `internal/unifiedresources`

Canonical resource identity comes from `unifiedresources.Resource`:

- `Resource.ID` is the long-term alert `resource_id`.
- `Resource.Type` is the long-term alert `resource_type`.
- `Resource.Canonical`, `ParentID`, `ParentName`, `MetricsTarget`, and `DiscoveryTarget` are the only allowed inputs for display naming, grouping, and source lookup.
- `Resource.Incidents` is the only canonical provider-native incident feed.

Important migration bridge:

- Today, alert `resource_id` and threshold override key are not always the same key.
- Example: host alerts emit `agent:<hostID>` resource IDs, while override lookup still resolves on raw `host.ID`.
- The migration must therefore carry both a `ResourceID` and a `ThresholdKey` until config storage is migrated safely.

### 2. Canonical Alert Specs

Add a spec layer inside `internal/alerts` and make it the only input to transition logic.

Minimum spec families:

- `MetricSpec`
- `StateSpec`
- `IncidentSpec`

Minimum fields every spec must carry:

- stable `AlertID`
- canonical `ResourceID`
- explicit `ThresholdKey`
- canonical `ResourceType`
- `Node`, `Instance`, `ResourceName`
- `AlertType`
- current value or incident payload
- resolved message and metadata
- transition policy fields such as `RequiredConfirmations`, `MonitorOnly`, and `DisableConnectivity`

This is the contract that replaces ad hoc alert assembly inside each `Check*` method.

### 3. Evaluator Engine

Source of truth: `internal/alerts`

The evaluator owns only:

- threshold resolution
- time-threshold delay handling
- hysteresis trigger/clear behavior
- active/resolved transition rules
- acknowledgement and escalation preservation
- rate limiting
- quiet-hours suppression
- flapping suppression

`checkMetric`, `preserveAlertState`, `dispatchAlert`, and the resolved-alert flow already do most of this. The migration should reuse those mechanics, not replace them.

### 4. Transition Persistence

Transition state remains in the existing manager until the migration is complete:

- `activeAlerts`
- `recentlyResolved`
- `recentAlerts`
- `pendingAlerts`
- `offlineConfirmations`
- `ackState`
- flapping and rate-limit maps
- `historyManager`

Do not create a second persistence path. The migration succeeds by moving callers onto the same state machine.

### 5. Notification Fan-Out

`internal/monitoring/monitor_alerts.go` remains the notification boundary:

- tenant websocket broadcasts
- notification manager sends/cancels
- incident timeline recording
- AI alert callbacks

The evaluator emits alert transitions. Monitoring owns delivery.

## Why a Full Rewrite Is the Wrong Move

A full rewrite is the wrong move here for four concrete reasons.

1. The hard part already exists.
The current manager already preserves ack state, start times, quiet-hours behavior, cooldown re-notify logic, rate limiting, flapping suppression, and resolved notification behavior. Rebuilding that from scratch would duplicate the highest-risk code path.

2. The codebase already has a working canonical seam.
`unified_eval.go` and `unified_incidents.go` prove the repo can migrate incrementally by moving input builders first and leaving transition state alone.

3. Pollers are still source-shaped.
The monitor layer still fans out typed models from Proxmox, agents, Docker, PBS, PMG, storage, backups, and snapshots. Replacing the engine outright would force every poller and every alert family to move at once.

4. Downstream contracts already depend on the current manager.
`monitor_alerts.go` assumes one live alert manager feeds websocket, notifications, incident recording, and AI. Swapping the engine wholesale would multiply migration surfaces instead of reducing them.

The correct move is to keep one manager and migrate input assembly until legacy `Check*` methods are thin builders only.

## Stable Behaviors That Must Not Change Mid-Migration

These behaviors are already relied on by tests, persisted state, or user workflow and must remain stable until a dedicated migration phase says otherwise.

- Existing alert ID formats stay stable for migrated families. Do not change IDs while `preserveAlertState`, ack carry-over, and history continuity still depend on them.
- Existing threshold override keys stay stable until config migration is implemented explicitly.
- `checkMetric` hysteresis, per-type time thresholds, per-metric time thresholds, cooldown re-notify, and critical escalation-on-level-change stay unchanged.
- Ack, start time, escalation metadata, and last notification timestamps continue to survive alert rebuilds through `preserveAlertState`.
- Notification suppression rules stay unchanged: quiet hours, rate limiting, flapping, monitor-only, activation state, and resolved-notification rules.
- Current confirmation semantics stay unchanged:
  - node offline: 3 polls
  - PBS offline: 3 polls
  - PMG offline: 3 polls
  - storage offline: 2 polls
  - guest powered-off: 2 polls
- Unified incident suppression semantics stay unchanged, including parent-child de-dup in `internal/alerts/unified_incidents.go`.
- Tenant broadcast routing stays in monitoring and remains tenant-scoped.

## Migration Phases

### Phase 1: Introduce Canonical Builder Contracts

Goal:
Create explicit canonical alert-spec builders without changing downstream alert behavior.

Code moves:

- Add internal spec types in `internal/alerts` for metric, state, and incident evaluation.
- Give every spec both `ResourceID` and `ThresholdKey`.
- Add builders that project `unifiedresources.Resource` into evaluator input.
- Add builder helpers for typed source models where unified resources are not yet the direct caller.
- Keep `CheckUnifiedResource` and `SyncUnifiedResourceIncidents` as the first-class entry points for canonical metric and incident inputs.

What stays:

- `checkMetric`
- `preserveAlertState`
- existing notification callbacks
- existing manager maps and history persistence

Delete at end of phase:

- Nothing. This phase is structure-only.

Exit criteria:

- New evaluator helpers accept specs rather than raw source models.
- New alert logic is forbidden from being added directly inside typed `Check*` methods.

### Phase 2: Convert Metric Families First

Goal:
Make typed metric checks collectors/builders only.

Code moves:

- Convert `CheckGuest`, `CheckNode`, `CheckHost`, `CheckPBS`, and `CheckStorage` into:
  - source-specific collection and suppression
  - canonical spec building
  - shared evaluator calls
- Reuse `evaluateUnifiedMetrics` and `checkMetric` for the final transition decision.
- Keep source-specific metadata assembly in the builders.
- For host-agent metrics, explicitly bridge the current mismatch between raw host override keys and emitted `agent:<id>` alert resource IDs.

Required parity gates:

- extend `internal/alerts/unified_eval_parity_test.go`
- keep `internal/alerts/threshold_resolution_shared_test.go` green
- keep `internal/alerts/override_normalization_test.go` green

Delete at end of phase:

- Inline metric trigger/clear code inside the migrated `Check*` methods.
- Duplicated threshold-resolution branches that now live in shared builder/evaluator helpers.

Exit criteria:

- Those five `Check*` methods no longer create metric alerts directly.
- Metric alert construction happens only through shared spec-to-evaluator code.

### Phase 3: Move State and Event Families Onto the Same Engine

Goal:
Stop building `Alert` structs directly inside typed methods for non-metric resource alerts.

Code moves:

- Convert offline and power-state handling to `StateSpec` with per-family confirmation policy.
- Convert provider-specific event families to builders:
  - host SMART risk and wearout
  - RAID degradation/rebuild state
  - ZFS pool and device health
  - Docker container state, health, restart-loop, OOM, memory-limit, and image-update events
  - PMG queue depth, oldest message, backlog, anomaly, and node queue events
- Keep the current confirmation counts, alert IDs, messages, and metadata fields stable.
- Keep `SyncUnifiedResourceIncidents` as the canonical incident path, but make it emit `IncidentSpec` into shared transition helpers instead of assembling `Alert` objects directly.

Delete at end of phase:

- Direct `Alert{...}` construction in migrated event branches of `internal/alerts/alerts.go`.
- Resource-family-specific state transition duplication that now exists only to manage confirmations and resolved handling.

Exit criteria:

- Typed `Check*` methods act as collectors/builders for both metric and event families.
- Shared transition helpers own all create/update/resolve behavior for migrated families.

### Phase 4: Make Monitoring Call the Canonical Path First

Goal:
Make unified resources the primary alert input surface without breaking domains that still need typed collection.

Code moves:

- After unified resource refresh, evaluate canonical resources first.
- Expand `internal/monitoring/monitor_alert_sync.go` from incident-only sync into the canonical unified alert-sync entry point.
- Keep typed poller calls only for domains not yet represented well enough in unified resources.
- Current expected holdouts:
  - backups from `internal/monitoring/monitor_backups.go`
  - snapshots from `internal/monitoring/monitor_backups.go`
- For those holdouts, move them to spec builders even if they are not yet unified-resource-backed.

Delete at end of phase:

- Direct typed metric fan-out from monitor loops for resource families already covered by unified resources.
- Compatibility wrappers in monitoring that only exist to feed old evaluator entry points.

Exit criteria:

- The primary evaluation path for guest, node, host, PBS, storage, and unified incidents starts from unified resource state.
- Typed monitor calls remain only for documented exceptions.

### Phase 5: Collapse the Identity Bridge and Remove Legacy Entrypoints

Goal:
Finish the migration by making unified identity the only alert identity contract.

Code moves:

- Migrate threshold override/config storage from legacy keys to canonical unified resource IDs.
- Remove explicit `ThresholdKey` bridging once config and alert state agree on the same ID contract.
- Convert backups and snapshots into canonical builder specs with stable resource references.
- Retire compatibility wrappers once no monitor path depends on them.

Delete at end of phase:

- legacy key-translation helpers that only exist for override compatibility
- typed `Check*` evaluator logic that survived as wrappers
- any duplicated incident or metric entrypoints made obsolete by the canonical path

Exit criteria:

- `unifiedresources.Resource.ID` is the single alert identity contract for evaluation and config.
- Remaining `Check*` methods are either deleted or are thin adapters used only at external boundaries.

## How Legacy `Check*` Methods Should Look After Migration

After Phase 3, a legacy `Check*` method should do only this:

1. Read source-specific models.
2. Apply source-specific suppression or shaping that cannot live in the generic evaluator.
3. Build canonical specs with explicit resource identity, threshold key, metadata, and transition policy.
4. Hand the specs to shared evaluator helpers.

It should not:

- resolve trigger/clear transitions inline
- mutate `activeAlerts` directly
- construct resolved-alert flows directly
- send notifications directly
- own history or acknowledgement persistence

## Deletion Rules By Area

Delete code only when the replacement path is already live and parity-tested.

- Metric-family code can be deleted once the shared metric builder/evaluator path produces identical IDs, thresholds, and metadata.
- Event-family code can be deleted once the shared state/incident helpers preserve confirmation counts, ack carry-over, and resolution behavior.
- Monitor fan-out code can be deleted only after unified resource evaluation is the first caller in the poll loop.
- Key-translation helpers can be deleted only after config override storage is migrated to canonical unified IDs.

## Highest-Risk Migration Edges

These are the places most likely to create regressions if the migration is done sloppily.

1. Identity drift between `ResourceID` and override key.
This is already visible in host-agent alerts. Do not collapse those keys until config migration is real.

2. Alert ID churn.
Changing IDs too early will break ack continuity, history continuity, re-notify cooldown state, and flapping history.

3. Partial transition rewrites.
If a builder starts creating `Alert` structs directly again, transition policy will drift across families.

4. Backup and snapshot exceptions.
Those flows still come from recovery/backups rather than unified resources. They should join the canonical spec layer before any attempt to delete their typed paths.

5. Incident duplication.
`SyncUnifiedResourceIncidents` already suppresses redundant parent/child incidents. Any replacement path must preserve that exact behavior.

## Execution Order

Do the work in this order:

1. builder/spec layer
2. metric-family migration
3. state/event-family migration
4. monitor unified-first routing
5. config identity migration and final deletions

Any attempt to start with Phase 4 or Phase 5 first will create a temporary second alert engine, which this repo does not need.