Add privacy-safe alert quality telemetry

This commit is contained in:
Pulse Test 2026-08-29 13:59:13 +01:00
parent 88da3e05c9
commit 0df9747bd7
19 changed files with 1364 additions and 126 deletions

View file

@ -93,6 +93,21 @@ Every field is listed below with the reason it exists. Nothing else is included
| Alerts fired 30d | `18` | Count unique locally retained alert occurrences in the current 30-day window without sending alert text, resource IDs, or timestamps |
| Alerts acknowledged 30d | `7` | Count acknowledgements in the current 30-day window without sending actors, reasons, alert IDs, or timestamps |
| Alerts resolved 30d | `12` | Count resolved alert records in the current 30-day window without sending resolution details, alert IDs, or resource IDs |
| Active alerts by severity | `1` info, `2` warning, `1` critical | Three aggregate counts that reconcile to active alerts |
| Active alert age buckets | `1` under 1h, `2` 1h-24h, `1` 1d-7d, `0` 7d+ | Four closed age buckets with no alert timestamps |
| Alerts fired by severity 30d | `2` info, `10` warning, `6` critical | Three aggregate occurrence counts that reconcile to alerts fired 30d |
| Alerts resolved by severity 30d | `1` info, `7` warning, `4` critical | Three aggregate occurrence counts that reconcile to alerts resolved 30d |
| Alert resolution duration buckets 30d | `3` under 15m, `2` 15m-1h, `5` 1h-24h, `2` 1d-7d, `0` 7d+ | Five closed time-to-resolution buckets with no occurrence timestamps |
| Repeat alert occurrences 30d | `4` | Count occurrences after the first local occurrence of the same canonical alert identity, without sending that identity |
| Snoozed alert occurrences 30d | `3` | Count occurrences with a durable suppression lifecycle transition |
| Alerts resolved while snoozed 30d | `2` | Count snoozed occurrences whose recovery transition arrived before an unsnooze transition |
| Alert manager tenants | `2` | Denominator for tenant-level alert configuration and persistence fields |
| Alert delivery active tenants | `2` | Count tenant alert managers with delivery enabled and activation state active |
| Alert flapping detection enabled tenants | `2` | Count tenant alert managers with flapping detection configured, not detected flapping episodes |
| Alert intent policy configured tenants | `1` | Count tenant alert managers with an explicitly revised default, resource-type, or resource intent policy |
| Alert event history authoritative tenants | `2` | Count tenant alert managers whose durable event history is authoritative |
| Alert active state authoritative tenants | `2` | Count tenant alert managers whose durable active-state projection is authoritative |
| Alert active state persistence degraded tenants | `0` | Count tenant alert managers carrying the fixed local degraded-state marker |
| Notification attempts 7d | `14` | Count delivery attempts, including retry attempts, in the locally retained seven-day queue window without sending recipients, endpoints, titles, or message content |
| Notification deliveries 7d | `11` | Count successfully delivered queue records in the local seven-day window without sending channel, recipient, endpoint, or content |
| Notification failures 7d (terminal in schema v3) | `3` | Count terminal failed or dead-lettered delivery outcomes in the local seven-day window without sending retry-attempt failures, error text, endpoint, recipient, or message content |
@ -235,6 +250,16 @@ or usage counters to the new release. Listener, startup, runtime, API, UI, and
asset failures collapse into fixed categories. Addresses, URLs, IP addresses,
asset names, response bodies, and raw errors are neither sent nor stored.
Telemetry schema v14 adds the alert-quality aggregates listed above. The
sender folds identities and lifecycle transitions locally, then sends counts
only. Automatic versus operator resolution is deliberately absent because the
canonical alert lifecycle records recovery evidence but has no operator-resolve
action or resolution-actor provenance. Maintenance suppression effectiveness
is also absent because intent policies can suppress a candidate before an alert
occurrence exists, leaving no trustworthy fired-alert denominator. Detected
flapping episodes are not reported because their diagnostic event path may be
dropped under pressure. Configuration adoption is reported instead.
#### Server-side handling and retention
- Telemetry pings are stored on the Pulse license server only for aggregate install/use analysis.
@ -243,6 +268,7 @@ asset names, response bodies, and raw errors are neither sent nor stored.
- Aggregate reports preserve the closed deployment-method buckets, but treat them as best-effort current-runtime evidence. In particular, `container_other` and `binary_other` are unknown fallbacks for many upgraded installs, not precise original installation provenance.
- Aggregate reports describe the known-age bucket as time since Pulse first created the schema-v2 lifecycle record. It is only a lower bound for an upgraded install and must not be presented as original installation age.
- Aggregate release-health reports use the direct schema-v13 current and immediately previous observations. They do not infer new-release health from rolling update, feature, alert, or action counters.
- Aggregate alert-quality release comparisons require schema v14 on both cohorts, compare rolling counters only between consecutive same-version heartbeats, and stratify cohort exposure by known-age and heartbeat-count buckets. A first heartbeat is baseline-only, and cohorts with materially different exposure are not presented as like-for-like outcomes.
- External-agent/MCP activity is stored only as a coarse adapter-origin flag plus capability-class counters: context, event stream, provisioning, operator state, findings, and action requests.
- The receiver stores only fields in its versioned telemetry allowlist. A cross-repository parity check prevents client fields from being silently dropped and prevents the storage contract from growing beyond the disclosed payload.
- Telemetry rows older than **90 days** are purged automatically.

View file

@ -556,6 +556,25 @@ inspectability, or convert missing/stale evidence into health.
## Current State
### Alert-quality telemetry folds only canonical durable lifecycle truth
`internal/alerts/telemetry_quality.go` folds active alerts and the canonical
30-day history projection into identity-free aggregate counts. Severity totals,
active-age buckets, resolution-duration buckets, repeated occurrences, and
snooze outcomes are computed locally. Canonical state and occurrence time may
be used transiently for folding, but neither leaves the process. The snapshot
also exposes tenant-level configuration and event/active-state authority counts
so downstream rates have denominators and persistence failures remain visible.
The contract deliberately does not classify automatic versus operator
resolution because alert resolution currently records recovery evidence, not a
resolution actor or operator-resolve action. It does not claim maintenance
suppression effectiveness because an intent policy can suppress a candidate
before an alert occurrence exists. It also excludes detected flapping episode
counts because that diagnostic event path may drop under pressure. Flapping
configuration adoption remains measurable. `telemetry_quality_test.go` pins
occurrence folding, snooze sequencing, and exact bucket boundaries.
The alert webhook editor exposes the delivery contract's language-neutral
`MessageKey`, event, resource type and node display name alongside the existing
type, severity and metric fields. Presentation must advertise the backend-owned

View file

@ -4209,6 +4209,16 @@ auto-register mutation boundary.
## Current State
### Telemetry preview is the canonical schema-v14 alert-quality payload
The system settings telemetry preview continues to return the exact `Ping`
built by the outbound sender. Schema v14 adds aggregate alert lifecycle,
active-age, resolution-duration, repeat, snooze, configuration-adoption, and
persistence-health fields. `TelemetryPingPreview` declares the same fields, and
cross-repository schema parity covers the public Go payload, the frontend type,
and the private receiver allowlist. Older payload rows remain valid with zero
defaults for additive v14 columns.
Production package boundaries are active under `internal/api/alerting/` and
`internal/api/configapi/`, backed by shared tenant-context, scope-enforcement,
install-token, and agent-binding packages. Alert/notification and

View file

@ -1178,6 +1178,16 @@ cleanup so readers cannot retain orphaned runtime or alert projections.
## Current State
### Install snapshots aggregate alert quality across isolated tenant managers
`ReloadableMonitor.AggregateInstallSnapshotCounts` loads each provisioned
organization through its own monitor, reads one privacy-bounded alert-quality
snapshot from that tenant's alert manager, and adds only aggregate integers to
the install snapshot. Organization IDs and alert identities are never copied
into telemetry state. Existing fired, acknowledged, and resolved totals now
come from the same canonical fold as the v14 quality fields, keeping legacy
consumers compatible while preventing two calculations from drifting.
### Proxmox and agent disk observations share full hardware identity
PVE may publish a RAID array volume's full NAA value as a bare serial while

View file

@ -2585,6 +2585,7 @@
"internal/alerts/reducer_parity_refire_test.go",
"internal/alerts/resolved_lock_discipline_test.go",
"internal/alerts/synology_test.go",
"internal/alerts/telemetry_quality_test.go",
"internal/alerts/threshold_resolution_shared_test.go",
"internal/alerts/unified_incident_confirmation_test.go",
"internal/alerts/update_alerts_test.go",
@ -6101,6 +6102,7 @@
"internal/monitoring/proxmox_large_cluster_poll_budget_test.go",
"internal/monitoring/pve_protection_observation_test.go",
"internal/monitoring/ratetracker_test.go",
"internal/monitoring/reload_test.go",
"internal/monitoring/truenas_poller_test.go",
"internal/unifiedresources/code_standards_test.go"
]

View file

@ -740,6 +740,22 @@ tokens, and path-normalization variants.
## Current State
### Alert-quality telemetry remains aggregate and comparison-safe
Telemetry schema v14 exports only counts and closed buckets. It exports no
resource or destination identity, hostname, alert or rule content, lifecycle
timestamp, actor, reason, error text, or clickstream. Public and frontend-served
privacy disclosures enumerate the fields and explain the deliberately excluded
resolution, maintenance-suppression, and detected-flapping classifications.
The adoption report treats a version's first heartbeat as baseline-only and
uses only consecutive same-version changes for rolling counters. Optional
release comparisons require schema v14 in both cohorts, then match observations
by the latest known-install-age bucket and version heartbeat-count bucket.
Unmatched exposure is reported rather than pooled, and the specific 6.4.0
versus 6.3.2 comparison is rejected as schema-incompatible instead of presenting
legacy alert volume as alert quality.
### Docker update preflight remains read-only across the unified-agent bridge
The `pulse-agent` late-bound Docker updater delegates the typed, read-only