mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-19 02:28:59 +00:00
Add privacy-safe alert quality telemetry
This commit is contained in:
parent
88da3e05c9
commit
0df9747bd7
19 changed files with 1364 additions and 126 deletions
|
|
@ -93,6 +93,21 @@ Every field is listed below with the reason it exists. Nothing else is included
|
|||
| Alerts fired 30d | `18` | Count unique locally retained alert occurrences in the current 30-day window without sending alert text, resource IDs, or timestamps |
|
||||
| Alerts acknowledged 30d | `7` | Count acknowledgements in the current 30-day window without sending actors, reasons, alert IDs, or timestamps |
|
||||
| Alerts resolved 30d | `12` | Count resolved alert records in the current 30-day window without sending resolution details, alert IDs, or resource IDs |
|
||||
| Active alerts by severity | `1` info, `2` warning, `1` critical | Three aggregate counts that reconcile to active alerts |
|
||||
| Active alert age buckets | `1` under 1h, `2` 1h-24h, `1` 1d-7d, `0` 7d+ | Four closed age buckets with no alert timestamps |
|
||||
| Alerts fired by severity 30d | `2` info, `10` warning, `6` critical | Three aggregate occurrence counts that reconcile to alerts fired 30d |
|
||||
| Alerts resolved by severity 30d | `1` info, `7` warning, `4` critical | Three aggregate occurrence counts that reconcile to alerts resolved 30d |
|
||||
| Alert resolution duration buckets 30d | `3` under 15m, `2` 15m-1h, `5` 1h-24h, `2` 1d-7d, `0` 7d+ | Five closed time-to-resolution buckets with no occurrence timestamps |
|
||||
| Repeat alert occurrences 30d | `4` | Count occurrences after the first local occurrence of the same canonical alert identity, without sending that identity |
|
||||
| Snoozed alert occurrences 30d | `3` | Count occurrences with a durable suppression lifecycle transition |
|
||||
| Alerts resolved while snoozed 30d | `2` | Count snoozed occurrences whose recovery transition arrived before an unsnooze transition |
|
||||
| Alert manager tenants | `2` | Denominator for tenant-level alert configuration and persistence fields |
|
||||
| Alert delivery active tenants | `2` | Count tenant alert managers with delivery enabled and activation state active |
|
||||
| Alert flapping detection enabled tenants | `2` | Count tenant alert managers with flapping detection configured, not detected flapping episodes |
|
||||
| Alert intent policy configured tenants | `1` | Count tenant alert managers with an explicitly revised default, resource-type, or resource intent policy |
|
||||
| Alert event history authoritative tenants | `2` | Count tenant alert managers whose durable event history is authoritative |
|
||||
| Alert active state authoritative tenants | `2` | Count tenant alert managers whose durable active-state projection is authoritative |
|
||||
| Alert active state persistence degraded tenants | `0` | Count tenant alert managers carrying the fixed local degraded-state marker |
|
||||
| Notification attempts 7d | `14` | Count delivery attempts, including retry attempts, in the locally retained seven-day queue window without sending recipients, endpoints, titles, or message content |
|
||||
| Notification deliveries 7d | `11` | Count successfully delivered queue records in the local seven-day window without sending channel, recipient, endpoint, or content |
|
||||
| Notification failures 7d (terminal in schema v3) | `3` | Count terminal failed or dead-lettered delivery outcomes in the local seven-day window without sending retry-attempt failures, error text, endpoint, recipient, or message content |
|
||||
|
|
@ -235,6 +250,16 @@ or usage counters to the new release. Listener, startup, runtime, API, UI, and
|
|||
asset failures collapse into fixed categories. Addresses, URLs, IP addresses,
|
||||
asset names, response bodies, and raw errors are neither sent nor stored.
|
||||
|
||||
Telemetry schema v14 adds the alert-quality aggregates listed above. The
|
||||
sender folds identities and lifecycle transitions locally, then sends counts
|
||||
only. Automatic versus operator resolution is deliberately absent because the
|
||||
canonical alert lifecycle records recovery evidence but has no operator-resolve
|
||||
action or resolution-actor provenance. Maintenance suppression effectiveness
|
||||
is also absent because intent policies can suppress a candidate before an alert
|
||||
occurrence exists, leaving no trustworthy fired-alert denominator. Detected
|
||||
flapping episodes are not reported because their diagnostic event path may be
|
||||
dropped under pressure. Configuration adoption is reported instead.
|
||||
|
||||
#### Server-side handling and retention
|
||||
|
||||
- Telemetry pings are stored on the Pulse license server only for aggregate install/use analysis.
|
||||
|
|
@ -243,6 +268,7 @@ asset names, response bodies, and raw errors are neither sent nor stored.
|
|||
- Aggregate reports preserve the closed deployment-method buckets, but treat them as best-effort current-runtime evidence. In particular, `container_other` and `binary_other` are unknown fallbacks for many upgraded installs, not precise original installation provenance.
|
||||
- Aggregate reports describe the known-age bucket as time since Pulse first created the schema-v2 lifecycle record. It is only a lower bound for an upgraded install and must not be presented as original installation age.
|
||||
- Aggregate release-health reports use the direct schema-v13 current and immediately previous observations. They do not infer new-release health from rolling update, feature, alert, or action counters.
|
||||
- Aggregate alert-quality release comparisons require schema v14 on both cohorts, compare rolling counters only between consecutive same-version heartbeats, and stratify cohort exposure by known-age and heartbeat-count buckets. A first heartbeat is baseline-only, and cohorts with materially different exposure are not presented as like-for-like outcomes.
|
||||
- External-agent/MCP activity is stored only as a coarse adapter-origin flag plus capability-class counters: context, event stream, provisioning, operator state, findings, and action requests.
|
||||
- The receiver stores only fields in its versioned telemetry allowlist. A cross-repository parity check prevents client fields from being silently dropped and prevents the storage contract from growing beyond the disclosed payload.
|
||||
- Telemetry rows older than **90 days** are purged automatically.
|
||||
|
|
|
|||
|
|
@ -556,6 +556,25 @@ inspectability, or convert missing/stale evidence into health.
|
|||
|
||||
## Current State
|
||||
|
||||
### Alert-quality telemetry folds only canonical durable lifecycle truth
|
||||
|
||||
`internal/alerts/telemetry_quality.go` folds active alerts and the canonical
|
||||
30-day history projection into identity-free aggregate counts. Severity totals,
|
||||
active-age buckets, resolution-duration buckets, repeated occurrences, and
|
||||
snooze outcomes are computed locally. Canonical state and occurrence time may
|
||||
be used transiently for folding, but neither leaves the process. The snapshot
|
||||
also exposes tenant-level configuration and event/active-state authority counts
|
||||
so downstream rates have denominators and persistence failures remain visible.
|
||||
|
||||
The contract deliberately does not classify automatic versus operator
|
||||
resolution because alert resolution currently records recovery evidence, not a
|
||||
resolution actor or operator-resolve action. It does not claim maintenance
|
||||
suppression effectiveness because an intent policy can suppress a candidate
|
||||
before an alert occurrence exists. It also excludes detected flapping episode
|
||||
counts because that diagnostic event path may drop under pressure. Flapping
|
||||
configuration adoption remains measurable. `telemetry_quality_test.go` pins
|
||||
occurrence folding, snooze sequencing, and exact bucket boundaries.
|
||||
|
||||
The alert webhook editor exposes the delivery contract's language-neutral
|
||||
`MessageKey`, event, resource type and node display name alongside the existing
|
||||
type, severity and metric fields. Presentation must advertise the backend-owned
|
||||
|
|
|
|||
|
|
@ -4209,6 +4209,16 @@ auto-register mutation boundary.
|
|||
|
||||
## Current State
|
||||
|
||||
### Telemetry preview is the canonical schema-v14 alert-quality payload
|
||||
|
||||
The system settings telemetry preview continues to return the exact `Ping`
|
||||
built by the outbound sender. Schema v14 adds aggregate alert lifecycle,
|
||||
active-age, resolution-duration, repeat, snooze, configuration-adoption, and
|
||||
persistence-health fields. `TelemetryPingPreview` declares the same fields, and
|
||||
cross-repository schema parity covers the public Go payload, the frontend type,
|
||||
and the private receiver allowlist. Older payload rows remain valid with zero
|
||||
defaults for additive v14 columns.
|
||||
|
||||
Production package boundaries are active under `internal/api/alerting/` and
|
||||
`internal/api/configapi/`, backed by shared tenant-context, scope-enforcement,
|
||||
install-token, and agent-binding packages. Alert/notification and
|
||||
|
|
|
|||
|
|
@ -1178,6 +1178,16 @@ cleanup so readers cannot retain orphaned runtime or alert projections.
|
|||
|
||||
## Current State
|
||||
|
||||
### Install snapshots aggregate alert quality across isolated tenant managers
|
||||
|
||||
`ReloadableMonitor.AggregateInstallSnapshotCounts` loads each provisioned
|
||||
organization through its own monitor, reads one privacy-bounded alert-quality
|
||||
snapshot from that tenant's alert manager, and adds only aggregate integers to
|
||||
the install snapshot. Organization IDs and alert identities are never copied
|
||||
into telemetry state. Existing fired, acknowledged, and resolved totals now
|
||||
come from the same canonical fold as the v14 quality fields, keeping legacy
|
||||
consumers compatible while preventing two calculations from drifting.
|
||||
|
||||
### Proxmox and agent disk observations share full hardware identity
|
||||
|
||||
PVE may publish a RAID array volume's full NAA value as a bare serial while
|
||||
|
|
|
|||
|
|
@ -2585,6 +2585,7 @@
|
|||
"internal/alerts/reducer_parity_refire_test.go",
|
||||
"internal/alerts/resolved_lock_discipline_test.go",
|
||||
"internal/alerts/synology_test.go",
|
||||
"internal/alerts/telemetry_quality_test.go",
|
||||
"internal/alerts/threshold_resolution_shared_test.go",
|
||||
"internal/alerts/unified_incident_confirmation_test.go",
|
||||
"internal/alerts/update_alerts_test.go",
|
||||
|
|
@ -6101,6 +6102,7 @@
|
|||
"internal/monitoring/proxmox_large_cluster_poll_budget_test.go",
|
||||
"internal/monitoring/pve_protection_observation_test.go",
|
||||
"internal/monitoring/ratetracker_test.go",
|
||||
"internal/monitoring/reload_test.go",
|
||||
"internal/monitoring/truenas_poller_test.go",
|
||||
"internal/unifiedresources/code_standards_test.go"
|
||||
]
|
||||
|
|
|
|||
|
|
@ -740,6 +740,22 @@ tokens, and path-normalization variants.
|
|||
|
||||
## Current State
|
||||
|
||||
### Alert-quality telemetry remains aggregate and comparison-safe
|
||||
|
||||
Telemetry schema v14 exports only counts and closed buckets. It exports no
|
||||
resource or destination identity, hostname, alert or rule content, lifecycle
|
||||
timestamp, actor, reason, error text, or clickstream. Public and frontend-served
|
||||
privacy disclosures enumerate the fields and explain the deliberately excluded
|
||||
resolution, maintenance-suppression, and detected-flapping classifications.
|
||||
|
||||
The adoption report treats a version's first heartbeat as baseline-only and
|
||||
uses only consecutive same-version changes for rolling counters. Optional
|
||||
release comparisons require schema v14 in both cohorts, then match observations
|
||||
by the latest known-install-age bucket and version heartbeat-count bucket.
|
||||
Unmatched exposure is reported rather than pooled, and the specific 6.4.0
|
||||
versus 6.3.2 comparison is rejected as schema-incompatible instead of presenting
|
||||
legacy alert volume as alert quality.
|
||||
|
||||
### Docker update preflight remains read-only across the unified-agent bridge
|
||||
|
||||
The `pulse-agent` late-bound Docker updater delegates the typed, read-only
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue