6b79aa997 bounded WriteBatchSync itself, which broke its read-your-writes
contract on slow disks: CI's metrics write-amplification and 500-node
load tests count committed rows after writing, and mock seeding reads
store coverage straight back, so the 2-second early return failed both
(runs 31475700902, 31494553977). Fast local disks masked it.
WriteBatchSync returns to a full commit wait. The monitoring pipeline's
four sync sites move to WriteBatchBounded, which carries the bounded
enqueue-plus-wait semantics, so the #1437 slow-disk stall fix stays
exactly where the hazard is. Both paths share prepareWriteBatch
validation, and a new regression test pins WriteBatchSync waiting past
the bounded budget.
Refs #1437
Contract-Neutral: behavioral fix: split bounded pipeline writes from synchronous batch writes, restores read-your-writes (#1437 follow-up), no public contract delta
WriteBatchSync waited unboundedly for the ingestion worker's commit. The
monitoring pipeline calls it inline from state broadcast, agent ingest,
and poll publish, so a metrics disk slow enough to back up the write
queue froze the monitor after its first cycle: polls stopped being
scheduled, PollStatus.LastSuccess never advanced, and healthy API
sources degraded to stale/agent-only while SQLite ground through
retention maintenance (107s cleanup, multi-second commits on the
reporter's instance).
enqueueAndWait now shares a single 2s budget across enqueue and commit.
A queue that cannot accept the batch within the budget drops it with a
warning, matching enqueueWrite's saturation behavior. A batch that
enqueues but has not committed stays queued and is not lost; the caller
moves on and a rate-limited warning records the backlog. Healthy disks
keep read-your-writes semantics.
Refs #1437
Contract-Neutral: behavioral fix: bound metrics store sync write wait (#1437), no public contract delta
Use a native absolute path in the SMART override test so Windows verifies the same contract as Unix. Move the metrics concurrency proof into the canonical verification artifact and document that live ingestion remains independent from lifecycle maintenance.
Run startup maintenance, rollups, and retention on a lifecycle-owned worker separate from buffered metric ingestion. SQLite still serializes write transactions, while maintenance CPU/read work can no longer stop writeCh from draining until it drops live batches. Join both workers before closing the shared database.\n\nFixes #1601
The metrics store capped its SQLite pool at one connection, so every UI
history read queued behind every buffered-write commit, and behind the
WAL checkpoints those commits pick up at the 4000-page threshold. On
write-heavy installs (many Docker agents with churning containers) that
serialization presented as sustained 120-260ms COMMIT warnings and an
unresponsive UI even with idle CPU and fast disks. Writes were never the
risk: flush, rollup, retention, and maintenance already funnel through
the single background worker goroutine, and the WriteBatchSync poller
path serializes on the WAL write lock via busy_timeout.
Raising the pool exposed a second bug: auto_vacuum(INCREMENTAL) in the
per-connection DSN pragmas replays as a database-header write whenever
the pool opens a new connection, which blocks connection creation behind
the active writer for up to the full 30s busy_timeout. auto_vacuum is a
persistent database property that migrateAutoVacuum already establishes
once at startup, so the per-connection copy is dropped.
Refs #1601
Contract-Neutral: behavioral fix: metrics store read concurrency and per-connection auto_vacuum pragma removal, no public contract delta (#1601)
A full local go test run on 2026-06-11 failed pkg/metrics while vite
builds were saturating the machine. Reproduced under synthetic 8-core
load: only the TestSLO_* wall-clock p95 assertions failed, never the
functional tests. A latency budget measured on a shared dev machine
cannot distinguish host contention from a code regression; load storms
inflated medians up to ~4x with no code change, so no absolute
threshold separates the two.
Route the latency SLO assertions (pkg/metrics, internal/api,
internal/monitoring) and the load/stress perf assertions (internal/api)
through helpers that keep strict enforcement on GitHub Actions runners
(controlled environment, existing hosted-runner envelopes unchanged)
and skip locally on overrun, printing the full p50/p95/p99
distribution in the skip message. A local pass still means the budget
was genuinely met. Error-response and correctness checks remain hard
failures. CI -race runs are unaffected: these tests already skip under
the race detector.
Verified: pkg/metrics, internal/api, and internal/monitoring all pass
with 8 CPU burners saturating the host, the scenario that previously
turned TestSLO_RollupTierBatchedFleet and four other budget tests red.
Mock mode seeded rich in-memory chart history but passed a nil store to the
seeder and the live tick, so the sqlite metrics store held almost nothing:
vmware/truenas/docker resources got one row per poll from the platform
ingestion paths and mock Proxmox guests got nothing at all (the unified sync
skips every resource in mock mode). Performance reports rendered
"Data Points: 0" for PVE guests and "Samples: 1" elsewhere, with no charts.
Re-enable store seeding behind an explicit PULSE_MOCK_SEED_METRICS_STORE
opt-in that scripts/hot-dev.sh and scripts/toggle-mock.sh export exactly
where they point PULSE_DATA_DIR at the isolated tmp/mock-data dir. Without
the opt-in (a production install flipping PULSE_MOCK_MODE on its real data
dir) the store stays untouched, which is what the old nil guard protected.
Replace the dormant dense seeding policy (every in-memory timestamp written
to both hourly and daily tiers, ~11M rows at current fixture scale) with
tier-correct backfill driven by the deterministic mock.SampleMetric runtime:
daily 30d at 4h spacing, hourly 7d at 2h, minute 24h at 15m, raw left to the
live tick (now also store-connected under the opt-in). Timestamps sit on the
spacing grid and a new Store.MaxTimestampsForTier coverage query fills only
the gap since the previous boot, so restarts neither duplicate rows nor
re-pay the seed: a fresh seed wrote 264,960 rows in 8.4s (71MB) and a
restart wrote only the gap plus fixture series whose mock IDs are not
boot-stable (k8s pod names, ceph FSID, a pre-existing generator defect).
Verified live in mock mode: 30-day PDF reports for a mock PVE VM
(checkout-web-01), a vSphere VM, the TrueNAS host, and a Docker container
now show 1260/1260/1260/540 data points with rendered time-series charts.
golangci-lint run ./... failed on ~190 pre-existing errcheck violations and
5 unformatted files, burying any new regression in noise. Fix all of them:
- Test files that hand-rolled mock-mode set/restore (vmware, truenas, and
friends) now use the canonical setMockModeForTest/testutil.SetMockMode
helper instead of drift copies that ignored SetEnabled errors.
- internal/mock and internal/monitoring tests get package-local
mustSetEnabled/mustSetMockEnabled/mustSetMonitorMockMode helpers that
fail the test on toggle errors.
- pkg/auth/sqlite_manager.go, pkg/metrics/store.go, pkg/server/server.go:
rollbacks in defers use the explicit-discard idiom, migration renames and
rollup commits log failures, the hosted reaper goroutine logs an error
exit, shutdown mock-disable logs failures.
- Remaining test sites check errors with t.Fatalf/t.Errorf or explicitly
discard best-effort calls (restore-chmods, handler-closure unmarshals)
per existing repo style.
- gofmt: internal/api/maintenance_verification.go, internal/ai/demo.go and
three findings test files.
Only dupl findings remain (44 pre-existing production-code duplication
pairs) — those need real refactors, not mechanical fixes.
Full test suites pass for every touched package.
Follow-up to 2b8ce06c0: that commit's contract update states reclaim skips
the WAL checkpoint entirely when the freelist is empty, but the prior commit
captured the pre-refinement store.go (the re-stage was lost to a .git/index.lock
race). Make the code match the contract: return early on a freelist_count read
error and when freelist == 0, so a steady-state no-op cycle does not run an
extra hourly wal_checkpoint(TRUNCATE).
runRetention reclaimed only PRAGMA incremental_vacuum(5000) (~20MB) and
only when that hourly cycle deleted rows. On instances where an hourly
retention pass frees more than 5000 pages, the freelist grows net-positive
every cycle and the sqlite file bloats unboundedly (5GB+ of free pages over
~60MB of live data) even though row retention works. This is the documented
'50+ resources -> 5GB+' symptom the 5000-page batch was meant to fix.
Reclaim every cycle (so a pre-existing backlog drains even in a no-delete
hour) and size each reclaim to the current freelist, capped at 50000 pages
(~200MB) so a large backlog drains over several cycles without holding the
write lock for minutes. Skip the checkpoint when the freelist is empty so
steady-state WAL cadence is unchanged.
Surfaced by the demo server (pulse-relay): metrics.db had grown to 5.9GB
(5GB freelist), filling its 25GB disk and breaking the nightly backup.
Adds TestStoreRetentionReclaimsFreePages to the perf-and-scalability proof
set and documents the reclaim invariant in the subsystem contract.
Tighten v5-to-v6 upgrade safety, release installability, provider MSP mode handling, AI cost accounting, metrics flushing, and frontend guardrails for the v6.0.0 GA candidate.
Extract alert config types, normalization, and identity helpers into internal/alerts/config while preserving the existing alerts package API through aliases and wrappers.
Move Manager callback lifecycle state into a same-package callbackBus, keeping public Set/Subscribe methods unchanged.
Harden metrics SQLite artifacts to owner-only permissions and cover permissive umask behavior.
Proof: go test -json ./internal/api -count=1; go test ./internal/alerts/... ./internal/monitoring ./internal/ai/... ./internal/websocket ./internal/config ./pkg/metrics; go test ./internal/alerts/... ./pkg/metrics
Docker container URL preserved on update (#1054): container updates
recreate the container with a new runtime ID. The agent now includes
{oldContainerId, newContainerId} in the completion ACK payload; the
server uses this to copy persisted metadata (custom URLs, descriptions,
tags) to the new ID so nothing is lost. Migration is a copy, not a move,
so rollback scenarios still find metadata under the original ID.
Reduce metrics.db write amplification (#1124): add a UNIQUE index on
(resource_type, resource_id, metric_type, timestamp, tier) so rollup
reprocessing after a failed checkpoint uses INSERT OR IGNORE instead of
creating duplicate rows. Existing duplicates are deduplicated once on
startup if the index creation would otherwise fail. Also sets
wal_autocheckpoint(500) to checkpoint the WAL more frequently, preventing
unbounded WAL growth.
Fixes#1054Fixes#1124
VACUUM creates a full copy of the database. Running retention first
deletes stale data (5GB → ~60MB live), so the VACUUM copies far less
data — faster startup and much less temporary disk space needed.
The auto_vacuum(INCREMENTAL) pragma from the previous commit only takes
effect on new databases. SQLite requires a full VACUUM to restructure
existing files when switching from NONE to INCREMENTAL. Without this,
users upgrading from bloated 5GB+ databases would never reclaim space.
Adds a one-time migration on startup that detects the current auto_vacuum
mode and runs VACUUM to convert if needed. Subsequent startups skip the
migration since the mode is already INCREMENTAL.
The metrics database could grow to 5GB+ for modest setups because:
1. Retention deletes rows hourly but SQLite never reclaims the space
2. WAL file grows unbounded without explicit checkpointing
3. No cleanup runs on startup, so restarts accumulate stale data
Fixes:
- Enable auto_vacuum=INCREMENTAL so deleted pages can be reclaimed
- Run incremental_vacuum after each retention cleanup
- Force WAL checkpoint(TRUNCATE) after deletes to prevent WAL bloat
- Run retention on startup to clean stale data immediately
Expected DB size for a 50-resource setup drops from 5GB+ to ~60-70MB.
Ref: GitHub Discussion #1231
When querying short time ranges (1h, 6h), the metrics store only looked
in TierRaw and TierMinute which were empty in mock mode. The seeded data
was stored in TierHourly and TierDaily.
Updated tierFallbacks to include coarser tiers as fallbacks:
- TierRaw now falls back to TierMinute, then TierHourly
- TierMinute now falls back to TierRaw, then TierHourly
This ensures sparkline data is available in mock/demo mode where
historical data is seeded into coarser tiers.
