The AI action broker treated an unreadable operator lock as unlocked:
isResourceRemediationLocked returned (false, nil) with no audit store
wired, and the caller logged store errors then dispatched anyway. An
operator's NeverAutoRemediate=true could be silently ignored whenever
the policy store was missing or erroring, which is unacceptable while
Patrol and Assistant run at assisted or full autonomy.
Posture change at the dispatch decision point:
- isResourceRemediationLocked now reports unknown state (nil store or
lookup failure) as an ErrRemediationLockStateUnknown-wrapped error
instead of silently defaulting to unlocked.
- New checkRemediationLockForDispatch gate: dispatches without an
approved human decision fail CLOSED on unknown lock state and
surface "remediation lock state unknown; operator approval
required". Human-approved dispatches keep the historical fail-open
behavior with a warning log. A confirmed lock still refuses even
approved dispatches, as before.
- executeNativeActionWithAudit (TrueNAS app start/stop/restart) now
enforces the lock too; it previously skipped the check entirely.
- Refusals persist Failed audit records with stable
remediation_lock_state_unknown: / resource_remediation_locked:
ErrorMessage prefixes.
- ai-runtime subsystem contract updated to pin the new posture.
Tests cover store-error and nil-store at both autonomy postures on
both dispatch paths; routing/control tests now wire an in-memory
audit store since autonomous dispatch without one is refused.
The public demo wrote ~110K resource_changes rows/day (restart 60K/day,
state_transition 45K/day), making the Changes timeline unreadable and
keeping unified_resources.db churning. Four generator-level engines,
all verified with before/after soaks against scratch mock backends:
- Per-tick flap probabilities ran 43,200x/day on the 2s update loop
(docker restart p=0.01/tick alone is ~430 restarts/day/container).
Churn rates are now expressed as events per day per entity and
converted per tick against the configured interval, tuned to a few
fleet-wide events per day with dwell times long enough to see.
- The pod scheduling reconciler fought the per-tick scenario re-pin
and fabricated a fresh random StartTime on every recovery; derived
uptime moved backwards, which change emission records as a restart.
The reconciler is now idempotent: stable per-pod park/reschedule
choices, StartTime never regenerated, and only pods it parked
itself (NodeNotReady/ClusterOffline/NodeLost) get recovered, so
curated Pending and ImagePullBackOff stories stay put.
- Swarm cluster objects were fabricated per host under one shared
cluster key, so registry dedupe alternated between the divergent
candidates every poll (service renames, status flips, node
re-parenting). One leader manager now reports services, tasks,
secrets, configs and the node inventory, like a real control plane.
- Demo docker host profiles cycled 2 hostnames across 4 online hosts,
collapsing canonical identities, and scripted-offline hosts had
their sighting refreshed right at the 2 minute staleness threshold,
sawtoothing them online/offline. Four distinct host profiles now
exist and offline hosts keep a stably stale sighting.
Before/after soak with a live client: pre-fix ~110-160 rows/min
sustained (restart ~45/min, matching the droplet's 60K/day); post-fix
zero rows/min at steady state with the curated degraded stories
(CrashLoop payments-worker, ImagePullBackOff, offline hosts) intact.
f62f35e24 restored the v5 used | cache | free memory split for Proxmox
nodes and guests, but standalone host agents still reported a flat
used/free pair, so the Machines page memory bar could not show the
reclaimable segment. Flagged by the Machines page v5 parity audit.
- Host agent reports cacheBytes (gopsutil Available minus Free); the
ZFS ARC adjustment recomputes free so used + cache + free still
covers the total.
- ApplyHostReport maps the field into models.Memory.Cache and clamps
inconsistent or older-agent reports so used + cache never exceeds
total.
- AgentMemoryMeta carries cache onto unified resources so the frontend
agent payload exposes it.
- Mock generic hosts split a third of non-used pages as cache, and the
node-linked host conversion now holds the invariant instead of
stacking the node's cache on top of a recomputed free.
- Contracts: monitoring, unified-resources, and storage-recovery now
document the split (also covering the f62f35e24 node/guest surface,
which landed without contract deltas).
The alert engine stamps metadata.resourceType on every alert, but the
websocket state path converts alerts.Alert to models.Alert, which had no
Metadata field, so every active alert reached the frontend stripped. The
history Type badge then fell back to unified-store lookups that miss
nodes (alert.resourceId is the platform-native node ID while unified
resources mint canonical ids, and alert.resourceName is the raw node
name while unified resources prefer the display name), rendering
Unknown. In mock mode the generated history rows had the same gap.
models.Alert gained the Metadata field in f62f35e24 (it rode along with
the memory-cache commit); this completes the transport:
- copy Metadata in activeAlertsSnapshot (websocket active alerts),
GetRecentlyResolved (resolved alerts to state), and the mock
UpdateAlertSnapshots conversion; sources are deep clones already
- deep-copy Metadata in models cloneAlert to keep the snapshot
clone contract honest
- stamp resourceType in the mock history generator using the real
engine vocabulary (node, vm, system-container)
- recognize system-container in the history Type badge map; that is
what the v6 engine stamps for LXC guests
Verified live in mock mode: history previously resolved 277 of 780
rows to Unknown (all node alerts); now 718/718 rows and 19/19 active
alerts carry resourceType and zero badges render Unknown.
The mock random-metrics updater recomputes Used/Free from the sampled
percentage but left Cache at its generation-time value, so a drifting
node could show used+cache > total and a 102% 'Shown in Proxmox' row.
Clamp the cache into the non-used pages in applyMemoryUsage, and teach
the memory-bar presentation to clamp defensively so a momentarily
inconsistent snapshot can never render segments past 100%.
v5 modeled memory as used | cache | free (Memory.Cache, 'reclaimable
buff/cache') and the memory bar's tooltip carried a 'Shown in Proxmox'
row explaining why Pulse's percentage reads lower than the Proxmox UI's
cache-inclusive number — a recurring support question. The v6 rebuild
deleted the field from the backend model, so the split and the
reconciliation vanished product-wide. Flagged by the Proxmox overview
parity audit.
Backend: re-add Memory.Cache; split it out via a shared
splitReclaimableMemory helper at the node resolver (node status reports
truly-free directly) and the VM builder (when guest meminfo reported
free pages); transport as proxmox.memoryCache on unified resources
alongside swap/balloon; mock mode populates plausible cache for nodes
and VMs.
Frontend: cache prop on StackedMemoryBar with the v5 muted-amber
segment between active and balloon, tooltip rows for reclaimable cache,
truly-free Free (balloon-capped), and the 'Shown in Proxmox'
reconciliation; guest and node memory adapters normalize free to
truly-free at the boundary; guest and node drawers grow a Reclaimable
cache row; the Proxmox nodes table passes cache and node swap through.
Mock fixtures never set LastSeen on PVE nodes, VMs, LXC containers, or
storage. The registry used to paper over that by replacing zero sightings
with ingest time; since 53faa4e46 preserves zero ("never seen") and stamps
those sources "unknown", mock mode rendered its whole PVE estate with dash
last-seen and unknown source freshness.
Stamp sightings in updateFixtureStateMetricsAt, which runs at generation
and on every refresh tick, before the RandomMetrics gate so static-metrics
fixtures stay fresh too. Online nodes and everything they host get the
refresh time (a poll delivers its full inventory, stopped guests
included); anything on an offline node keeps its old stamp, with zero
backdated ten minutes so the UI shows a stale sighting rather than
"never".
resourceFromStorage and resourceFromDockerContainer stamped LastSeen with
time.Now() at conversion because their source models carried no poll
timestamp. The registry rebuilds from the retained state snapshot every
cycle, so those resources re-reported a fresh sighting each rebuild even
after their upstream source (PVE instance, docker host agent) stopped
delivering, and their per-source SourceStatus could never go stale via
markStaleLocked. 53faa4e46 fixed this fabrication at the ingest layer but
left these two adapter-level stamps.
- models.Storage gains LastSeen (omitzero), stamped where entries are
built: the PVE storage poll (poll start time, including synthesized
cluster-shared entries; preserved entries for unpolled nodes keep their
old stamp), the PBS datastore conversion (PBS instance sighting), Ceph
pool projection (cluster LastUpdated), and the mock generator (offline
mock nodes get a backdated stamp so the stale path renders).
- resourceFromStorage passes storage.LastSeen through; zero stays zero
("never seen") instead of becoming conversion time.
- resourceFromDockerContainer uses host.LastSeen: containers are delivered
wholesale with each host report, so the host report timestamp is the
container sighting. This matches every other docker sub-resource adapter
(services, tasks, volumes, networks, images already use host.LastSeen).
- ingestStorage routes PBS-poller datastore entries (instance "pbs-<name>",
type pbs) to SourcePBS, parented to the PBS instance. Keying them
SourceProxmox would judge their freshness against the 60s Proxmox stale
threshold while PBS polls every 60s by default, flapping healthy
datastores stale between polls; SourcePBS carries the cadence-matched
120s threshold. PVE-reported pbs-typed storage.cfg backends stay
SourceProxmox. Side effect: syncUnifiedStorageMetrics no longer skips
PBS datastore storage, so those entries gain usage history.
- storageFromReadStateView round-trips LastSeen so the legacy storage API
reports the honest sighting; mock refresh re-stamps available storage on
each simulated poll.
The parent host/node staleness was already honest, so platform pages
reflected outages at the parent level; this makes the per-resource source
freshness honest too.
Mock fixture IDs churned across backend restarts, breaking resource
identity continuity in mock mode: k8s pod and deployment names picked
their namespace and prefix with rand.Intn per boot, the ceph FSID was
pure rand.Int63n, and generic agent hosts randomized both platform
profile (which feeds the host ID) and hostname. Every restart re-keyed
those unified resources and re-seeded ~31k orphan metric rows that
lingered for the full 90d mock retention.
Derive all of them from mockStableChoice/mockStableDecimalString over
stable inputs (cluster ID, item ordinal, instance name) instead,
matching the generator's existing stable-ID idiom and its documented
contract. Add a regression test that builds the fixture graph twice
and requires every identity set to match.
Verified live: two mock-mode boots now share all 185 distinct metric
resource IDs and the second boot's backfill seeds 0 rows (was ~31k).
golangci-lint run ./... failed on ~190 pre-existing errcheck violations and
5 unformatted files, burying any new regression in noise. Fix all of them:
- Test files that hand-rolled mock-mode set/restore (vmware, truenas, and
friends) now use the canonical setMockModeForTest/testutil.SetMockMode
helper instead of drift copies that ignored SetEnabled errors.
- internal/mock and internal/monitoring tests get package-local
mustSetEnabled/mustSetMockEnabled/mustSetMonitorMockMode helpers that
fail the test on toggle errors.
- pkg/auth/sqlite_manager.go, pkg/metrics/store.go, pkg/server/server.go:
rollbacks in defers use the explicit-discard idiom, migration renames and
rollup commits log failures, the hosted reaper goroutine logs an error
exit, shutdown mock-disable logs failures.
- Remaining test sites check errors with t.Fatalf/t.Errorf or explicitly
discard best-effort calls (restore-chmods, handler-closure unmarshals)
per existing repo style.
- gofmt: internal/api/maintenance_verification.go, internal/ai/demo.go and
three findings test files.
Only dupl findings remain (44 pre-existing production-code duplication
pairs) — those need real refactors, not mechanical fixes.
Full test suites pass for every touched package.
Dead-code sweep. Functions flagged unreachable by golang.org/x/tools/cmd/deadcode
and confirmed unused across pulse, pulse-enterprise, pulse-pro and pulse-mobile by
adversarial cross-repo verification. Cross-module reachability was checked
explicitly (only pkg/ exported symbols are importable by other modules; internal/
packages and _test.go files are not). go build, go vet and test-compile all pass.
Two real bugs that surfaced once the 20m test timeout let the
internal/api and internal/monitoring packages run to completion.
cloneVMwareInventoryMetrics omitted four fields:
Commit 23ea4e487 (Surface vSphere VM uptime and guest disk usage)
added UptimeSeconds, DiskUsedBytes, DiskTotalBytes, and DiskPercent
to vmware.InventoryMetrics but did not extend
cloneVMwareInventoryMetrics. The clone left those pointer fields
aliased to the source struct, so the mock fixture refresh path
(refreshVMwareInventoryMetrics writing through metrics.UptimeSeconds
via ensureInt64Ptr) and the snapshot read path
(inventoryUptimeSeconds dereferencing metrics.UptimeSeconds) raced
on the same heap-allocated int64.
TestMonitorBuildBroadcastFrontendStateUsesCanonicalMockUnifiedResources
exemption:
The test asserts broadcast state does not publish the lowercase-
hyphenated legacy docker host label so canonical docker hosts
surface their human-readable DisplayName. Commit 89abed099
(2026-05-24) added the docker-swarm-node resource type whose Name
is the swarm node hostname (matching how Docker Swarm identifies
node members), which collided with the legacy-label rejection.
Refine the assertion to apply only to host-type resources
(docker-host, agent, node).
Closes the only API-coverage gap from the Docker / Kubernetes IA
maturity review: Roles, ClusterRoles, RoleBindings, and
ClusterRoleBindings now flow from the Kubernetes agent through the
canonical resource registry into the Kubernetes platform-page
Configuration tab.
Agent: pkg/agents/kubernetes/report.go gains four new report struct
types that carry summary counts plus subject-kind sets; individual
subject names and full PolicyRule contents are deliberately omitted
so Pulse stays a "what permissions exist where" surface, not an RBAC
enumeration tool. internal/kubernetesagent/agent.go gains four
collectors that call rbacv1.RoleList/ClusterRoleList/etc. through the
existing runKubernetesCallWithRetry wrapper, matching the
ServiceAccount collector's RBAC-forbidden retry pattern.
Canonical: internal/models mirrors with NormalizeCollections coverage;
convert* funcs in internal/monitoring/kubernetes_agents.go translate
agent report -> model; ResourceTypeK8sRole / K8sClusterRole /
K8sRoleBinding / K8sClusterRoleBinding join the canonical type set;
registry ingest* + adapter resourceFrom* functions emit one Resource
per RBAC object with ruleCount / roleKind / roleName / subjectCount /
subjectKinds / aggregationLabels on the K8s meta; search mapping in
internal/api/resources.go and the privacy allow-list in
internal/api/org_handlers.go pick up the four new type tokens; the
K8s privacy category in unifiedresources/policy_metadata.go classifies
them like the rest of K8s.
Frontend: ResourceType union + ResourceKubernetesMeta carry the new
kinds and RBAC summary fields; KubernetesPageSurface query asks for
them; the page model buckets them into the Configuration group;
KubernetesConfigTable renders Role / ClusterRole rule counts and the
aggregated flag, plus RoleBinding / ClusterRoleBinding role refs and
"N subjects · Kind1, Kind2 +overflow" subject summaries.
Curated demo seeds per-namespace Roles + RoleBindings plus an
aggregated ClusterRole + ClusterRoleBinding for pulse-demo-monitoring
in each cluster so the Configuration tab renders 18 RBAC rows across
the three demo clusters.
Contracts updated for the canonical-shape guard: monitoring,
api-contracts, unified-resources, frontend-primitives,
organization-settings (canonical) plus agent-lifecycle and
storage-recovery (dependent via Extension Points). Verification
proofs extended: kubernetes_registry_test.go, kubernetes_agents_test.go,
agent_inventory_test.go (new TestCollectRBACInventoryReportsSummaryCountsOnly
that pins the subject-name-omission contract), demo_scenarios_test.go,
adapter_coverage_test.go, contract_test.go, org_handlers_test.go,
resourceIdentity.test.ts, reportingResourceTypes.test.ts,
KubernetesConfigTable.test.tsx, and the
subsystem_lookup_test.py line-anchor bumps that the contract edits
shifted (api-contracts 246 -> 253, organization-settings 92 -> 93).
Verified:
- go build ./internal/... ./cmd/... clean
- go test ./internal/unifiedresources/..., ./internal/mock/...,
./internal/kubernetesagent/..., ./internal/api/...,
the K8s subset of ./internal/monitoring/... all clean (three
pre-existing unrelated monitoring failures noted earlier remain
unchanged by this commit)
- npm run type-check, lint:eslint, lint:theme,
lint:canonical-platforms clean
- vitest: 70 K8s frontend tests pass including the new RBAC render
coverage in KubernetesConfigTable.test.tsx
- browser proof on /kubernetes/configuration: 36 config rows
including 18 RBAC rows across three clusters; ClusterRole
"pulse-demo-monitoring" shows "12 rules · Aggregated";
ClusterRoleBinding shows "3 subjects · Group, ServiceAccount +1"
Per-cluster node profiles, kubelet versions, and degraded scenarios
replace the global rotation that made every demo cluster look like a
copy of the same one. Production EU keeps its prod-euw1-k8s-{01..05}
nodes and the NotReady worker on prod-euw1-k8s-03 (preserving the
existing host-posture test contract). Staging EU runs
stage-euw1-k8s-{01..05} and carries the payments-worker
CrashLoopBackOff. Development EU runs dev-euw1-{01..05} and carries
an ImagePullBackOff on cron-nightly-backfill (re-labelled from the
previous "Pending / PodInitializing" rotation so the curated
reconciler doesn't recover it). The unused Edge profile gets distinct
edge-pop-{lax,nrt,fra,iad,sin}-01 names + k3s version for when the
cluster count is bumped above three.
A new TestKubernetesDemoClustersTellDistinctStories test guards the
slice goal: each cluster's nodes use its own prefix, exactly one
cluster carries each degraded scenario, and every cluster has a
unique kubelet version. The monitoring subsystem contract is updated
to reflect the new three-cluster cast (Production EU + Staging EU +
Development EU) plus the per-cluster scenario distribution.
Side effect: with distinct node names per cluster, the K8s page
model's cluster-to-node matching now resolves all five nodes for
each cluster (previously two clusters showed "0 nodes" because every
cluster's nodes shared the same prod-euw1-* names, breaking
buildKubernetesClusterChildCounts' clusterId lookup).
Verified:
- go vet ./internal/mock/..., go test ./internal/mock/... clean
- browser proof on /kubernetes/overview: three clusters render with
distinct versions (v1.30.4 / v1.31.2 / v1.32.0-rc.1) and 5 nodes
each (vs the previous 5/0/0 split)
- /kubernetes/nodes: 15 rows across the three clusters with three
distinct name prefixes; one red NotReady dot on Production EU's
prod-euw1-k8s-03; fourteen green Ready dots elsewhere
Collect native Kubernetes config, policy, and autoscaling objects.
Project the new resource types through API filters, unified resources, mock fixtures, and Kubernetes tabs.
Keep Secret inventory metadata-only and route k8s-secret policy as restricted local-only.
The vSphere adapter's InventoryMetrics struct only carried
throughput / utilisation metrics. Uptime and guest filesystem
usage weren't piped through at all, so the workloads table
rendered "0s" and empty cells for every vSphere VM.
Backend (internal/vmware):
- InventoryMetrics gains UptimeSeconds plus DiskUsedBytes /
DiskTotalBytes / DiskPercent. Documented in the struct comment
with the API sources they come from.
- PerformanceManager counter catalog adds sys.uptime.latest for
hosts and VMs and sys.osUptime.latest for VMs. The mapping
prefers guest OS uptime when present (Tools-reported) and falls
back to VMX-process uptime. Counters verified against vSphere 8
developer documentation.
- New per-VM REST collector calls
GET /api/vcenter/vm/{vm}/guest/local-filesystem and aggregates
per-mount capacity / free_space into DiskTotal / DiskUsed /
DiskPercent. A 503 from vCenter (Tools not reporting) is
classified as a non-fatal enrichment issue and the row stays
blank rather than failing the collection.
- enrichInventorySnapshot now takes automationSessionID so the
signals path can hit the REST endpoint alongside the VI/JSON
PerformanceManager queries.
- Resource projection layer wires UptimeSeconds onto
Resource.Uptime for hosts and VMs and the disk fields onto
metrics.disk; cloneInventoryMetrics tracks the new pointers.
Mock (internal/mock):
- refreshVMwareInventoryMetrics synthesizes plausible per-resource
uptime (1h - 30d base, climbing forward with snapshot time) and,
for VMs only, a stable guest filesystem total (32-256 GiB) with
naturally-oscillating used bytes via SampleMetric. Powered-off
VMs drop the new pointers so the frontend renders "-" rather
than zero, matching how the canonical "no data" signal already
works for offline guests.
Frontend (useWorkloads.ts):
- The WorkloadGuest uptime fallback chain now lands on the
canonical resource.uptime field. vSphere doesn't populate a
platform-specific carve-out (only the canonical field), so the
earlier proxmox/agent/docker/kubernetes-only chain was silently
dropping vSphere uptime.
Contracts:
- monitoring.md documents the new InventoryMetrics fields, their
vSphere collection sources, and the mock-fixture expectation.
- performance-and-scalability.md adds the canonical
resource.uptime fallback rule to the workload mapping section.
Proofs:
- internal/mock/platform_fixtures_test.go asserts that powered-on
vSphere VMs surface uptime + guest disk fields and powered-off
VMs drop them.
- frontend-modern/src/hooks/__tests__/useWorkloads.test.ts adds a
vSphere uptime fallback case.
- Existing vmware client test
(TestClientCollectInventoryPreservesBaseInventoryWhenOptionalEnrichmentDegrades)
teaches the mock vCenter to serve the new endpoint and updates
the assertions to match the additional non-fatal issue surfaced
when the unavailableVMGuestInfo knob also degrades the
filesystem read.
Project vCenter network inventory through canonical resources and add the vSphere Networks table backed by vCenter network topology. Align resource presentation coalescing so state and resource APIs share the same host contract.
Move vSphere back to the admitted first-lab-ready stage in the support manifest, regenerate shared projections, and keep onboarding/navigation copy from presenting VMware as fully supported before live vCenter proof.
Add a global resource timeline endpoint for provider activity and wire vSphere Activity to VMware timeline changes. Seed mock VMware activity through the same supplemental-change path and keep the relevant resource contract tests current.
Scope TrueNAS child source IDs under the system source key so duplicate pool, dataset, app, VM, share, and disk names from different appliances do not merge.
Use the resulting hierarchy for per-system TrueNAS overview counts and scoped mock metric targets.
Round out the curated demo estate so every platform page surfaces a
non-healthy entry rather than rendering uniformly green:
- generator.go: always seed a secondary PBS and PMG instance (drop the
random spawn gates) so the demo PBS and PMG pages reliably show two
instances.
- demo_scenarios.go (applyDemoDockerScenario): force docker host index 2
to "field-office-edge-01" with status=offline and exited containers,
giving the Docker page a disconnected host with stopped workloads.
- demo_scenarios.go (applyDemoKubernetesScenario): mark
prod-euw1-k8s-03 NotReady in the last cluster so the k8s nodes
table shows a NotReady worker.
- demo_scenarios.go (applyDemoHostScenario): also force the
prod-euw1-k8s-03 host agent offline so the Hosts page shows an
outage that matches the k8s nodes view.
- demo_scenarios.go (applyDemoStorageScenario): mark the dr-vault PBS
and mail-gateway-us PMG instances as degraded with degraded
connection health.
- demo_scenarios_test.go: update the curated-posture helpers to expect
the new offline/degraded entries while still asserting the rest of
the estate stays healthy after metric refreshes.
Configure pve5 as offline, cascading to stopped status on all its guests and zeroing metrics to simulate a node outage. Also force standalone offline states on postgres-replica-01 and dev-portal-01 workloads. Update test helpers and platforms coverage checklist in platform_support_contract_test.go.
Flip vmware-vsphere from governance_state=admitted, readiness_stage=
first-lab-ready to supported on both axes. The phase-1 floor (vCenter
client, canonical agent/vm/storage projection, alerts integration,
Assistant read paths, mock fixtures, page surface, automated proof for
the read-only boundary) has been in place; the only remaining gate was
a live vCenter run, and we are taking the platform claim on the strength
of the implementation rather than blocking on that proof.
Add vmware-vsphere to default_infrastructure_source_order, to the
First-class platforms list and Current Support Matrix in
PLATFORM_SUPPORT_MODEL, and to the Pulse primary navigation
(automatically via SUPPORTED_PLATFORM_IDS in the regenerated frontend
manifest). Rename and invert the Go contract test that asserted vSphere
stays admitted, and let the admitted-platform helpers tolerate an empty
set now that nothing currently lives there. Drop the "in first-lab-
ready readiness" line from the vSphere empty state.
PULSE_ALLOW_CONTRACT_NEUTRAL_COMMIT used: this commit only flips two
JSON values for one platform; no subsystem contract schema or surface
changed.
The /proxmox/ceph tab was mounting <StorageSurface forcedView="pools"
forcedSourceFilter="proxmox-pve">. That surface ships its own internal
Pools/Physical-disks tab switcher (StoragePageControls always renders
StorageViewSwitcher), and the forcedView createEffect immediately
reverted any click on those tabs — so the buttons looked interactive
but did nothing, and neither view was Ceph-aware in the first place.
Build a real ProxmoxCephTable backed by the canonical CephCluster
resource (type='ceph') that already flows through /api/resources. Add
ResourceCephMeta/ResourceCephPoolMeta/ResourceCephServiceMeta types on
the frontend Resource (the backend already projects these on
unifiedresources.Resource.Ceph; the frontend type was missing). Render
one row per cluster with health, FSID, MON/MGR quorum, OSD up/in
counts, PGs, pool count + stored bytes, capacity utilisation, daemon
service summary, and the Ceph health message.
Also fix the mock generator: generateCephClusters returns nil unless
some Storage entry has type cephfs/rbd/ceph, and no mock storage was
ever generated with those types — so the Ceph tab has always been
empty under mock mode. Seed a cluster-wide RBD pool (48TB) and CephFS
data pool (24TB) on the mock-cluster instance so the canonical
adapter has real cluster topology to synthesize.
The Proxmox Backups → Recent tasks sub-tab landed empty under mock
mode because PVEBackups.BackupTasks was hard-coded to an empty slice.
Generate 2-6 recent runs per ~70% of guests over the past 7 days,
mostly OK with occasional failures and an in-progress run, so the new
ProxmoxBackupsTable shows realistic data instead of an empty state.
Two specific platform-page quality issues from the audit:
1. **/kubernetes/overview only had 1 cluster.** Bumping the K8s cluster
count past 1 had been deferred because the prior
monitor-broadcast equivalence test compared the raw snapshot count
to the broadcast count exactly, and broadcast's
`coalesceBroadcastResources` + second-pass coalesce inside
`convertResourcesForBroadcast` legitimately drops merge candidates
that the raw snapshot keeps. Switch the test to compare against
the canonical snapshot count within a ±5% tolerance so future
fixture bumps stay green without loosening any of the test's
exact-name and exact-identity assertions. With that in place, bump
`K8sClusterCount` 1 → 3 in `internal/mock/generator.go`,
`scripts/toggle-mock.sh`, and the matching
`scripts/tests/test-toggle-mock.sh` so the canonical mock estate
ships with production + staging + edge clusters end-to-end.
Live mock survey: k8s-cluster: 3, k8s-deployment: 42, pod: 120,
plus 15 K8s nodes merged onto their agent hosts.
2. **/vmware/storage looked empty under platform-page chrome.**
`resolveStorageSourceKey` was reading only `storage.type` (the
on-disk technology like `vsan`, `vmfs`, `nfs41`, `zfs-pool`) and
never consulted `storage.platform` (the canonical platform key
like `vmware-vsphere` or `truenas`). Source filter chip options
were therefore generated as `vsan`, `vmfs`, `nfs41`, etc., and
`forcedSourceFilter='vmware-vsphere'` had nothing to match.
Prefer the canonical `storage.platform` tag when set, so VMware
datastores group under `vmware-vsphere`, TrueNAS pools group under
`truenas`, PBS datastores under `proxmox-pbs`, etc., for both the
chip options and the embedded platform-page filter.
Browser verification (Playwright, chromium, live mock-mode dev runtime):
- 9 tests pass.
Targeted vitest:
- `src/features/storageBackups` + `src/utils/__tests__/sourcePlatforms.test.ts` +
`src/components/Storage/__tests__/storageSourceOptions.test.ts` (31
files / 141 tests) green.
Go tests:
- `go test ./internal/mock/... ./internal/monitoring/... ./internal/vmware/...`
all green.
Contracts updated:
- `monitoring.md` Shared Boundaries: new K8s multi-cluster default,
±5% tolerance for the broadcast equivalence assertion.
- `deployment-installability.md` Shared Boundaries: toggle-mock.sh /
DefaultConfig parity updated for the K8sClusterCount=3 baseline.
Four documented platform-page gaps from the prior round are closed:
1. **Docker Swarm services canonical projection.** The unified resource
adapter requires `host.Swarm.ClusterID`/`ClusterName` for
`dockerSwarmClusterKey` to produce a stable service source ID; the
mock generator was leaving those fields empty so all generated
services were dropped. Anchor every mock Swarm host to a single named
cluster (`mock-swarm-cluster-1` / `edge-swarm`) so manager and worker
hosts share Swarm identity and their services deduplicate correctly
across managers. Live mock survey now exposes 15 docker-service rows
(was 0).
2. **Docker Swarm services UI restored.** The `/docker/services`
sub-tab is back. `DockerPageSurface` mounts a `PlatformResourceTable`
with the canonical operator toolbar (search + status chips +
counter); `dockerPageModel.ts` re-introduces the services bucket;
the model test asserts the three-tab shape and the services bucket.
3. **TrueNAS Systems / Overview sub-tab restored.** Re-survey of the
canonical adapter confirms `truenas.FixtureRecords` already emits
the top-level TrueNAS appliance as a unified `agent` row tagged
with the `truenas` platform (see `internal/truenas/provider.go::
truenasRecordsFromSnapshot`). TrueNAS now defaults to
`/truenas/overview` and the page model exposes a `systems` bucket.
4. **VMware fixture inventory scaled to a mature SMB lab.**
`internal/vmware/fixtures.go::appendEdgeClusterFixtures`
programmatically appends an Edge DC with 3 more ESXi hosts
(esxi-05..07), 12 more VMs across Tier 1 / Stateful / Workstations /
Observability / Archive tiers (mixed healthy/warning/powered-off,
mixed Linux/Windows guest OS), and 4 more datastores (VMFS / NFS41 /
vSAN / cold-iSCSI). Live mock survey now shows 43 VMs (was 31), 18
agents (was 15), and 60 storage rows (was 55) across two datacenters.
5. **TrueNAS / vSphere Storage source filter chip suppression.**
`StoragePageControls` gains a `suppressSourceFilter` prop and
`Storage.tsx` automatically applies it whenever `forcedSourceFilter`
is set, so platform-page embeds no longer render the now-locked
Source filter chip alongside the operator toolbar.
Resource survey under the new mock baseline (live `/api/resources`):
- TOTAL 342 unique resources (was 307)
- app-container: 75, storage: 60, system-container: 44, vm: 43,
pod: 40, physical_disk: 19, agent: 18, docker-service: 15,
k8s-deployment: 14, docker-host: 5, network-endpoint: 5,
pbs: 2, pmg: 1, k8s-cluster: 1
Browser verification (Playwright, chromium, live mock-mode dev runtime):
- 9 tests pass. Every populated sub-tab — Docker Hosts / Containers /
Swarm services, Kubernetes Clusters / Nodes / Pods / Deployments,
TrueNAS Systems / Storage / Apps, vSphere Hosts / VMs / Storage —
asserts both populated canonical rows AND a visible operator search
input.
Targeted vitest (77 files / 358 tests) + Go tests (./internal/vmware,
./internal/mock, ./internal/monitoring) all green.
Contracts updated:
- `storage-recovery.md` Shared Boundaries: TrueNAS defaults to the
Systems overview now that the canonical adapter emits a TrueNAS-
platform agent row; `suppressSourceFilter` auto-applies under
`forcedSourceFilter`.
- `unified-resources.md` Extension Points: same; the canonical TrueNAS
adapter emits the appliance as a unified resource so the builder
default lands on a populated Systems sub-tab.
- `Storage.test.tsx` extended with the source-filter suppression
contract assertion.
Mock pages were sparse: 3 Proxmox nodes × 3 VMs × 3 LXCs, 2 Docker
hosts × 5 containers, 1 K8s cluster × 3 nodes × 10 pods × 4
deployments. That populated platform pages with handfuls of rows
rather than table density that exercises sorting, grouping, drawers,
and responsive layout.
Bump `internal/mock/generator.go::DefaultConfig` to target a mature
small-to-mid homelab / SMB environment:
- NodeCount: 3 → 5 (matches the curated demo scenario's pve1..pve5
regional naming)
- VMsPerNode: 3 → 6
- LXCsPerNode: 3 → 8
- DockerHostCount: 2 → 5
- DockerContainersPerHost: 5 → 14
- GenericHostCount: 2 → 4
- K8sClusterCount: 1 (unchanged; the curated demo and broadcast
coalesce tests assume a single cluster identity)
- K8sNodesPerCluster: 3 → 5
- K8sPodsPerCluster: 10 → 40
- K8sDeploymentsPerCluster: 4 → 14
Resource survey under the new defaults (live mock backend):
- TOTAL 307 unique resources (was ~50-100)
- app-container: 75, storage: 55, system-container: 44, pod: 40,
vm: 31, physical_disk: 19, agent: 15, k8s-deployment: 14,
docker-host: 5, network-endpoint: 5, pmg: 2, pbs: 1, k8s-cluster: 1
Platform pages now feel populated under mock mode:
- /docker/overview: 5 hosts (was 2)
- /docker/containers: 75 containers (was 13)
- /kubernetes/nodes: 5 (was 3)
- /kubernetes/pods: 40 (was 10)
- /kubernetes/deployments: 14 (was 4)
`internal/mock/demo_scenarios.go` extended to season `local`,
`local-zfs`, and per-node iso/service-pool storage names for pve6 and
beyond, so future NodeCount bumps don't regress the curated demo into
generic "service-pool" labels (a test guard explicitly forbids that
alias). A new `TestDemoScenarioStorageNamingHandlesScaledNodeCount`
covers the scaled-NodeCount path.
`internal/monitoring/monitor_unified_state_test.go` updated to compare
the broadcast count against the coalesced snapshot count rather than
the raw snapshot count — the broadcast path merges resources that
share a canonical host key (K8s nodes onto linked agent hosts), so
larger fixture sizes legitimately produce more merge candidates, and
the prior raw-equality assertion would have broken on any future
fixture growth too. The test still asserts every canonical name and
mock identity it checked before.
`scripts/toggle-mock.sh` (`mock_default_entries`) and the matching
`scripts/tests/test-toggle-mock.sh` assertions are aligned with the
new defaults so `npm run mock:edit` and per-dev `.env` seeding match
the canonical baseline.
Contracts updated:
- `monitoring.md` Shared Boundaries: records the new DefaultConfig
target sizes and the requirement that demo-scenario seasoning stay
aligned with NodeCount changes.
- `deployment-installability.md` Shared Boundaries: records that
`mock_default_entries()` in toggle-mock.sh must stay aligned with
`internal/mock.DefaultConfig` so CLI/toggle/runtime mock densities
never drift apart.
Targeted Go tests:
- `go test ./internal/mock/...` green
- `go test ./internal/monitoring/...` green
Playwright (chromium, live mock-mode dev runtime):
- 9 tests, all pass; populated assertions now hit dense tables (5
hosts, 14+ containers, 40 pods, etc.).
Known remaining fixture gaps (canonical adapter, not config):
- VMware fixture inventory in `internal/vmware/fixtures.go` is
hardcoded at 4 hosts / 6 VMs / 4 datastores; not scaled in this
commit.
- TrueNAS fixture inventory in `internal/truenas/fixtures.go` is
similarly hardcoded; not scaled in this commit.
