Resolve the process memory-controller hierarchy before selecting the tightest v1 hard limit, so systemd and nested appliance limits actually inform the existing Go runtime headroom.
Contract-Neutral: runtime memory-limit detection only; no API or persistence contract change
Treat node-scoped HTTP 595 responses as debug-level resource unavailability instead of repeated authentication warnings. Preserve warnings and returned errors for real credential failures.
Refs #1794.
Contract-Neutral: corrects internal log severity without changing API, resource, or extension contracts
Change-source: pulse-maintainer
The PVE disks/list poll and the agent SMART report both observe a RAID
controller's exported volume, but PVE surfaces its NAA identifier as a
bare-hex serial while smartctl reports the same value as a naa.-prefixed
WWN with no serial. The linked-disk join compared serial to serial and
WWN to WWN verbatim, so the volume rendered twice (/dev/sda and sda)
with independent metric histories.
Hardware identity comparison now normalizes reporter framing (naa., eui.,
wwn-, 0x prefixes, case) and folds serial and WWN together before
comparing. Values are never truncated: sibling volumes on one controller
share their leading WWN bytes, and a truncated udev ID_WWN must stay
unequal to a full identifier. Placeholder serials no longer count as an
identity match.
Refs #1720
Prevent the idempotent poll test from cancelling the first request after the handler starts but before the service records a successful synchronization.
Change-source: pulse-maintainer
Sort backup-server and datastore rows deterministically so provider refresh order cannot reset expanded detail state. Exclude optical devices from PVE, legacy agent, and canonical physical-disk projections.
Fixes#1785Fixes#1786
Contract-Neutral: Corrects refresh ordering and non-disk filtering without changing public APIs or subsystem contracts.
Change-source: pulse-maintainer
Telemetry could see only saved connections, so an install that tried to
reach a node and could not was indistinguishable from one that never
opened the add-node dialog. Both report zero configured connections and
stall at the same activation stage. Fleet data shows that population is
real and concentrated three to one in container deployments, and nothing
recorded whether those installs attempted a connection at all.
Record node connection test attempts and failures in a bounded,
day-bucketed tally in the config directory, pruned to a 31-day retention
window, and report both over the install-ID rotation window as
node_test_attempts_30d and node_test_failures_30d.
Recording starts only once a request carries a target and credentials, so
an incomplete form is never counted as a node that could not be reached.
A host string that turns out to be unusable does count, because the
attempt was made and it failed. Only the add-node dialog endpoint is
instrumented: instrumenting the unused test-config endpoint as well would
double-count a single operator action.
The tally holds counts alone. Hosts, credentials, and error text never
enter it, which is why it is plain JSON rather than encrypted history.
An enabled Patrol that can never run and one that runs and finds nothing
were indistinguishable in the fleet: both presented as high run counts
with zero AI calls and zero findings. The install that motivated this
sat blocked for over a month because provider initialisation failed once
at boot and was never retried; the self-heal landed separately, but
telemetry still cannot see which blocked cause dominates in the field.
Schema v10 exports the fixed machine cause code (for example
provider_not_configured) only while Patrol is in the blocked runtime
state. The cause rides the router-owned Pulse Intelligence snapshot into
the outbound ping. Blocked-reason text, provider endpoints, model names,
and configuration stay on the install; an untyped blocked reason exports
nothing rather than free text, and a disabled, active, or mid-run Patrol
exports an empty value even when a stale cause is still recorded.
The unified agent's Linux installer only offered the root profile, and the
docs called non-root unsupported. That default is the most-cited reason
security-conscious evaluators reject Pulse without trying it. install.sh
gains --least-privilege: the service runs as a dedicated nologin
pulse-agent system user with every existing hardening directive, no
LXC-attach ambient capabilities, docker-group membership for socket reads,
and refusal (not silent root fallback) on appliance platforms, non-systemd
init systems, and --enable-commands. Optional --grant-smart and --grant-pct
restore the only two collectors that need elevation through
visudo-validated exact-command sudoers rules and root-owned wrappers the
agent reaches via new absolute-path-only PULSE_SMARTCTL_PATH /
PULSE_PCT_PATH overrides; the pct grant covers pct list and pct df only
and can never widen into pct exec. --update preserves the profile and its
grants by reading the installed unit.
The agent now authors a privilege block in its report (effective root,
service user, active helpers), carried through models into the fleet
doctor as a descriptive field: Agent Doctor shows the profile and its
helpers instead of presenting intentionally absent collectors as a fault,
and a least-privilege agent can never be marked unhealthy on that evidence
alone.
Two counters could not answer the question they exist for.
pulse_intelligence_patrol_runs_30d was counted from the operator-facing
patrol run history, which is capped at MaxPatrolRunHistory (100). On any
install patrolling on a normal schedule that cap is reached within hours
of a thirty-day window opening, so the field reported the cap rather than
the run count, and the calls-per-run ratio derived from it was inflated by
a censored denominator. Patrol run history now carries an uncapped daily
tally beside the capped list, advanced from a persisted high-water mark so
repeated full-list saves cannot double count, pruned to 31 days, and read
in preference to the list. An install with no tally yet falls back to the
list, so the counter never regresses on upgrade.
Schema v8 split agent-side pre-mutation refusals into target-change,
prerequisite and contract categories so they would stop hiding in "other".
In production all three are zero fleet-wide and "other" still absorbs every
refusal, because the dominant reason code is the legacy preflight_refused
aggregate recorded when an agent sends no machine reason code at all.
Agents older than the typed refusal contract report every refusal that way,
so the split is starved rather than broken, and folding the two together
made those two states indistinguishable. Schema v9 counts uncoded refusals
separately.
A handler that flushes before its first write commits the response
headers with an implicit 200. The gzip wrapper deferred its
compress-or-not decision to the first write, so such a handler would
send headers without Content-Encoding and then a compressed body,
which clients cannot decode. Every current flusher is an SSE handler
whose content type never compresses, so this was latent, but any
future streaming JSON endpoint would have corrupted silently. Flush
now runs the decision first, matching net/http header-commit
semantics.
Large estates that exceed the WebSocket frame ceiling recover over an
uncompressed /api/state poll loop, which made the biggest deployments
pay the heaviest transfer cost. Wrap the main HTTP handler in a gzip
middleware that compresses JSON and other text responses roughly an
order of magnitude, while leaving WebSocket upgrades, SSE streams,
Range requests, small declared bodies, and binary content untouched.
Contract-Neutral: transport-only gzip response middleware, no RBAC or persistence semantics change
An explicit connection address override on a cluster member was silently
discarded when VerifySSL was enabled and the member had no per-endpoint
fingerprint: the hostname-for-TLS preference displaced the operator's
address, so overriding an undialable discovered hostname changed nothing.
The override now wins in every TLS mode, and the hostname preference
applies only to auto-discovered addresses.
Failed endpoint recovery attempts also logged their cause at debug level
only, leaving the recurring 'No endpoints recovered' warning without a
reason. The warning now carries per-endpoint failure reasons and the
sanitized error is stored so the UI health status shows it too.
Refs #1665
Contract-Neutral: behavioral bug fix refs #1665: cluster member IPOverride honored in all TLS modes, recovery failure reasons surfaced; no public contract delta
6b79aa997 bounded WriteBatchSync itself, which broke its read-your-writes
contract on slow disks: CI's metrics write-amplification and 500-node
load tests count committed rows after writing, and mock seeding reads
store coverage straight back, so the 2-second early return failed both
(runs 31475700902, 31494553977). Fast local disks masked it.
WriteBatchSync returns to a full commit wait. The monitoring pipeline's
four sync sites move to WriteBatchBounded, which carries the bounded
enqueue-plus-wait semantics, so the #1437 slow-disk stall fix stays
exactly where the hazard is. Both paths share prepareWriteBatch
validation, and a new regression test pins WriteBatchSync waiting past
the bounded budget.
Refs #1437
Contract-Neutral: behavioral fix: split bounded pipeline writes from synchronous batch writes, restores read-your-writes (#1437 follow-up), no public contract delta
WriteBatchSync waited unboundedly for the ingestion worker's commit. The
monitoring pipeline calls it inline from state broadcast, agent ingest,
and poll publish, so a metrics disk slow enough to back up the write
queue froze the monitor after its first cycle: polls stopped being
scheduled, PollStatus.LastSuccess never advanced, and healthy API
sources degraded to stale/agent-only while SQLite ground through
retention maintenance (107s cleanup, multi-second commits on the
reporter's instance).
enqueueAndWait now shares a single 2s budget across enqueue and commit.
A queue that cannot accept the batch within the budget drops it with a
warning, matching enqueueWrite's saturation behavior. A batch that
enqueues but has not committed stays queued and is not lost; the caller
moves on and a rate-limited warning records the backlog. Healthy disks
keep read-your-writes semantics.
Refs #1437
Contract-Neutral: behavioral fix: bound metrics store sync write wait (#1437), no public contract delta
The PBS poller called GET /nodes once per cycle just to relearn the
node hostname, which is stable for the life of the connection. On
tokens that cannot read /nodes, PBS logged a 403 every ten seconds
forever. Cache the name on the client after the first success and
defer retries for 30 minutes after a permission denial so a widened
ACL still heals without a restart. Transient failures keep retrying
each cycle.
Refs #1691
Stripe checkouts could not be told apart by origin: every session was
stamped checkout_origin pulserelay_landing, so an upgrade started from an
RBAC gate looked identical to one started from the website. Gate CTAs now
stamp a closed-vocabulary source token (gate-<feature>, estate-card, or the
plans-page default) onto the owned billing plan route; the plan page threads
it into the purchase-start handoff, and the license server persists it on the
checkout intent and stamps Stripe metadata checkout_source.
Attribution is authenticated-session-only by construction. The public
/pricing route and the public pricing URL never carry it, and
getSelfHostedPurchaseStartUrl scrubs the parameter from forwarded query
strings so a crafted website link cannot claim in-app origin. Server-side,
source is validated against the same kebab vocabulary, skip-listed from the
Pulse Account portal redirect exactly as feature already is, and travels
only inside the handoff body. It is request-scoped and persists nothing on
the install; the cancel return echoes it so a retry keeps its origin.
The handoff field is omitempty because the license server decodes strictly,
so source-less installs stay compatible with a server either side of the
field being added. That ordering is recorded in pulse-pro OPERATIONS.md.
The 2026-08-07 telemetry read showed installs at or above 5 PVE nodes, 10
Docker hosts, or 3 VMware hosts convert to paid at ~8x the rate of smaller
estates. The ping now carries that classification as a server-derived
boolean so receiver-side cohort queries keep a stable column even if the
thresholds move later; it is derived in the pkg/server snapshot closure
from the same AggregateInstallSnapshotCounts values the payload already
sends, so no new information leaves the install.
The thresholds move to internal/monitoring/business_estate.go as the
single definition; the session-capability surface behind the in-product
business-estate card delegates to it, and dropping the now-unneeded direct
pkg/licensing import there restores TestPkgLicensingImportBoundary, which
f0e2243b4 had left red. All three payload surfaces (Ping struct, private
receiver, TelemetryPingPreview) move together per
check_telemetry_schema_parity.py, and both PRIVACY.md copies document the
field.
Verified live on an isolated worktree backend with 6 mock PVE nodes: the
Settings telemetry preview renders schema_version 8 with business_estate
true, and /api/security/status still reports
sessionCapabilities.businessEstate true through the delegated thresholds.
TestSummarySection_PaginatesCardGridForManyMetrics failed once on CI
(run 31058903048) with metric card "Disk Read" missing and all of page
2 absent from the extracted text. The card-grid pagination is fine; the
bug is in the test helper. Its stream regexp
(?s)stream\r?\n(.*?)\r?\nendstream let the optional \r consume the final
byte of the compressed payload whenever that byte (the last Adler-32
checksum byte) happened to be 0x0D. The truncated stream fails to
inflate and the helper silently skipped the entire page, roughly 1 in
256 streams. fpdf's compressed bytes are a pure function of the report
text, which embeds the report period, so specific date windows fail
deterministically. The CI window (period Jul 30 00:23 to Aug 6 00:23
UTC) reproduces on the first render while neighbouring windows pass,
which is why the test passed locally and on the previous run.
Slice each stream by the /Length declared in its object dictionary the
way a real PDF reader does. fpdf always writes /Length as a direct
integer, and compressed bytes can contain any sequence, so keyword
scanning can never be exact. Verified against the pinned CI window plus
400 shifted windows, and the full package.
Schema v6 shipped audit_logging_persistent and audit_events_30d as Pro adoption
signals. Neither discriminated. pkg/server installs the SQLite audit logger on
every install for defense in depth and gates only the read/export endpoints, so
the boolean was true on all 8 installs that had taken rc.8 and 0 rows in the
retained table have ever had it false. The event count measured that background
write volume: three of those eight unlicensed community installs were pegged at
the receiver's 100000 clamp ceiling, with the rest between 4863 and 67509.
Schema v7 replaces both with audit_reads_30d, a count of requests that cleared
the license gate on an audit read or export surface. A read requires a human
action, so unlike store presence or write volume it cannot settle into a
constant. The recorder is wrapped INSIDE RequireLicenseFeature so unentitled
requests never count, and the persisted marker carries a timestamp and a coarse
activity class from a fixed allowlist. Query filters, actors, ranges, and every
audit row read stay on the install.
The retired columns are left in the live database. They hold real rc.8 rows and
migrations only add, so dropping them would be a pointless risk; nothing writes
them once the receiver struct loses the fields.
Adds the guard this class needed. LicensedFeatureAdoptionFields registers every
field that exists to measure licensed-feature adoption, and
TestLicensedFeatureAdoptionFieldsDiscriminate builds an unused install through
the real production snapshot paths, installs a real SQLite audit logger exactly
as pkg/server does, records a baseline audit event, and fails if any registered
field is non-zero. Pinning a console logger there would have made the guard pass
while the payload lied, so it deliberately does not. The guard was verified by
reintroducing the v6 sourcing and confirming it fails with the field named.
A companion test pins the three retired fields so they cannot return under
their old names.
This is the third instance of one bug class. v6 removed
pulse_intelligence_patrol_autofixes_30d, hardcoded to zero with no increment
site, and then introduced two fields that were constant in the other direction.
Three occurrences is a guard, not a habit.
Verified end to end on a running unlicensed install: the payload that reported
audit_logging_persistent true under v6 now reports audit_reads_30d 0, and
seeding two in-window reads, one outside the window, and one with an invalid
activity class yields 2.
Six of the eight Pro-exclusive features had no telemetry field at all, so
there was no way to answer whether RBAC, audit logging, scheduled reporting,
agent profiles, alert-triggered AI, or Kubernetes AI were being used by the
installs paying for them. Schema v6 adds nine content-free adoption signals:
alert_ai_enabled AIConfig.IsAlertTriggeredAnalysisEnabled()
rbac_custom_roles non-built-in roles, per org
rbac_user_assignments user-to-role assignments, per org
audit_logging_persistent a persistent audit store is active, not console
audit_events_30d audit events retained inside the window
report_schedules configured scheduled reports
report_schedules_enabled scheduled reports switched on
report_schedules_run_30d schedules whose last run falls inside the window
agent_profiles configured agent profiles
Counts only. Role names, permissions, usernames, schedule names, delivery
recipients, report scope, profile names, and every audit event field stay on
the install. kubernetes_ai needs no field of its own: it is derivable at read
time from alert_ai_enabled combined with the existing kubernetes_clusters
count, and a dedicated field would be redundant.
Config-sourced signals are read through applyLicensedFeatureConfigSnapshot;
RBAC and audit live behind the router and are read through
Router.ApplyLicensedFeatureTelemetrySnapshot. The RBAC read goes through a new
TenantRBACProvider.PeekManager so a background telemetry read can never
provision an RBAC store for an org that has never used RBAC.
Also removes pulse_intelligence_patrol_autofixes_30d and the AutoFixCount
field behind it. patrol_run.go hardcoded AutoFixCount to 0 and no increment
site existed anywhere in the tree, so the counter was zero in all 233,364
retained production pings. That was a wiring bug, not evidence that nobody
uses Patrol fixes; governed fixes are delivered through the approved-action
pipeline, which is already instrumented. The field was plumbed through run
records, history persistence, the Assistant handoff, and telemetry while being
structurally incapable of holding a non-zero value.
Verified end to end against a running install rather than only in unit tests,
which is precisely the check the autofix counter never had: seeding three
report schedules (two enabled, one last run inside the window) and two agent
profiles produced report_schedules 3, report_schedules_enabled 2,
report_schedules_run_30d 1, agent_profiles 2 in the Settings telemetry
preview, and signing in moved audit_events_30d to 1.
The private receiver landed first in pulse-pro 78ff7dd so the new fields are
accepted on arrival.
Pulse under a systemd MemoryMax, docker --memory, or Kubernetes limit ran
with a GC that was blind to the cap: GOGC=100 lets total heap float to
roughly twice the live set, so a capped process grows until the kernel
OOM-kills it instead of collecting harder near the boundary. The demo
droplet's kill history (three OOM kills in July under an 800M cap) is
this mechanism, and any containerized install with a memory limit is
exposed the same way.
At startup, when GOMEMLIMIT is not set by the operator, resolve the
process cgroup (v2 ancestor walk taking the smallest memory.max on the
path, v1 limit_in_bytes fallback) and set the runtime soft limit to 90%
of it, leaving headroom for stacks, mmap, and CGO. Best effort: no
detectable limit leaves GC defaults untouched.
Contract-Neutral: runtime GC limit alignment: no payload or contract delta
