Second increment of the resource-payload-static-metadata gap: the full
activeAlerts array (~37-75KB at mock-estate churn) re-shipped whenever
any single alert changed. Alerts now ride the generalized keyed delta
transport: the per-field keying and transition handling in the state
delta engine is a declarative keyedDeltaFields list shared with the
connected-infrastructure projection, and activeAlertsDelta carries
id-keyed merge patches, removed ids, and order. Measured on the mock
estate the per-tick alert payload drops to 184-1981 bytes.
One boundary held deliberately fixed: alert application timing does not
change. The client applies alert deltas to an isolated baseline and
feeds the exact same applyActiveAlerts path immediately on arrival,
input-active or not — alert lifecycle truth never waits for the
operator-input idle gate (alerts subsystem boundary, documented in
6ecdc81d1), and the pending-acknowledgment safeguards see byte-
equivalent records. Alerts handed to the store are cloned so it never
adopts baseline-owned objects. Deltas without a baseline are ignored
and heal on the next full payload; unkeyable entries fall back to
whole-field diffing.
Remaining gap scope unchanged: snapshot-side static metadata
(canonicalIdentity alias history ~609KB per snapshot, and the partially
regressed aiSafeSummary/policy omissions) stays open under
resource-payload-static-metadata.
First increment of the resource-payload-static-metadata gap: the
connected-infrastructure reporting projection was re-shipped whole
(~55-60KB at 50-node scale) on every realtime broadcast even though
measurement showed the only field that ever changes tick-to-tick is
lastSeen, and the client deep-reconciled all 69 rich items per tick —
the dominant reactive cost inside every websocket message after the
resource merge itself.
The projection now rides the same per-client keyed delta engine as
resources. buildClientStateSnapshot keys it by id beside the resource
map, and buildClientStateDelta emits connectedInfrastructureDelta with
id-keyed JSON merge patches, removed ids, and order through one shared
keyed-array builder. Measured on the mock estate the per-tick payload
drops from ~55-60KB to 0.7-4KB of timestamp patches. Entries that
cannot key by id fall the whole field back to plain whole-payload
diffing, and keyed/unkeyed baseline transitions re-ship the full array
once, so the keyed path can never corrupt the projection.
The client mirrors the resource baseline pattern: full payloads
establish an isolated raw baseline, deltas apply as merge patches
through the now-generic keyed apply helper, and the store sync hands
reconcile reference-stable objects for untouched items so a tick
reconciles only what changed instead of deep-walking the projection.
Deltas without a baseline are ignored and heal on the next full
payload; the operator-input deferral gate carries the pending changed
ids across a gesture and flushes them in the same coalesced batch.
The remaining gap scope (snapshot-side static metadata such as
canonicalIdentity alias history, ~609KB per snapshot) stays open under
resource-payload-static-metadata.
The read deadline was refreshed only by protocol pong frames, so a client
whose ping/pong control frames were delayed more than 6 seconds or eaten
by a middlebox (Cloudflare edge, AV proxies) was killed every 60 seconds
even while its 30-second JSON heartbeats were arriving. Demo logs show
the signature clearly: sessions dying at exactly 60s/114s/168s, i.e.
zero, one, or two pongs before execution, while the user watched the
badge cycle through Reconnecting on an otherwise healthy connection.
Refresh the read deadline on every successful read, widen it to 90s so a
background tab throttled to one heartbeat per minute still survives, and
ping every 30s so pong-only clients tolerate two lost round trips.
Contract-Neutral: websocket keepalive hardening: read-deadline/ping cadence only, no payload or contract delta
Dead-code sweep. Functions flagged unreachable by golang.org/x/tools/cmd/deadcode
and confirmed unused across pulse, pulse-enterprise, pulse-pro and pulse-mobile by
adversarial cross-repo verification. Cross-module reachability was checked
explicitly (only pkg/ exported symbols are importable by other modules; internal/
packages and _test.go files are not). go build, go vet and test-compile all pass.
Under load, 5x concurrent /api/state degraded from 276ms (single) to ~4s
each (linear), because every caller serialized on the monitor lock to
rebuild and JSON-encode the full 1.6MB state. /api/diagnostics had the
same dogpile shape on cache miss, even though its 45s TTL cache was
working as designed.
Wrap both handlers in a per-tenant singleflight.Group so concurrent
callers share the work: 20x concurrent /api/state now completes in 358ms
wall (~14-50x improvement). Diagnostics warm-cache responses are now
sub-3ms; cold compute coalesces.
Also drop websocket Upgrader buffers from 4MB read/write to 64KB. gorilla
streams larger payloads across the buffer transparently, so the 4MB
allocation per connection was overhead that scaled badly with concurrent
clients (100 clients * 8MB = 800MB just in buffers).
Contract-neutral: no endpoint, response body, header, or wire-format
change.
- Use X-Forwarded-Proto/X-Forwarded-Scheme for scheme detection
- Use X-Forwarded-Host for host matching behind reverse proxies
- Update tests with remoteAddr for CSWSH protection validation
- AI Intelligence endpoints (/api/ai/intelligence/*, /api/ai/forecast/*,
/api/ai/unified/findings, etc.) now require ai:execute scope to prevent
low-privilege tokens from reading sensitive intelligence data
- AI Knowledge endpoints (/api/ai/knowledge/*) now require ai:chat scope
to prevent arbitrary guest data access across the fleet
- AI Debug Context (/api/ai/debug/context) now requires settings:read scope
to prevent system prompt and infrastructure details leakage
- WebSocket origin check now validates peer IP is private when allowing
private network origins, mitigating CSWSH attacks where a malicious page
on the same LAN tries to hijack connections using victim's session cookie
- Initialize Alert and Notification managers with tenant-specific data directories
- Add panic recovery to WebSocket safeSend for stability
- Record host metrics to history for sparkline support
Implements multi-tenant infrastructure for organization-based data isolation.
Feature is gated behind PULSE_MULTI_TENANT_ENABLED env var and requires
Enterprise license - no impact on existing users.
Core components:
- TenantMiddleware: extracts org ID, validates access, 501/402 responses
- AuthorizationChecker: token/user access validation for organizations
- MultiTenantChecker: WebSocket upgrade gating with license check
- Per-tenant audit logging via LogAuditEventForTenant
- Organization model with membership support
Gating behavior:
- Feature flag disabled: 501 Not Implemented for non-default orgs
- Flag enabled, no license: 402 Payment Required
- Default org always works regardless of flag/license
Documentation added: docs/MULTI_TENANT.md
Add atomic `closed` flag to Client struct and `safeSend()` helper method
to prevent race condition when sending to client channels. The race
occurred when a client disconnected while a goroutine was trying to send
initial state - the channel could be closed between the registration
check and the actual send.
All sends to client.send now go through safeSend() which checks the
closed flag first. The flag is set atomically before closing the channel
in all code paths (unregister, dispatchToClients, broadcast, shutdown).
Related to #1048
- checkOrigin: Remove redundant same-origin check at line 155 that was
already handled at line 116 (origin == requestOrigin)
Function improved from 95.1% to 97.4% coverage.
Test coverage for pure utility functions:
- isValidPrivateOrigin: validates private network origins (security)
- normalizeForwardedProto: normalizes ws/wss to http/https for proxies
- sanitizeValue: handles NaN/Inf values in JSON data
- cloneMetadata/cloneMetadataValue: deep copies metadata maps/slices
- cloneAlert/cloneAlertData: deep copies alert structures
Coverage increased from 20.9% to 37.3% (80 test cases).
Fixed goroutine leaks in WebSocket hub from missing shutdown mechanism:
Problem:
1. Hub.Run() has infinite loop with no exit condition
2. runBroadcastSequencer() reads from channel forever
3. No way to cleanly shutdown hub during restarts or tests
Solution:
- Added stopChan chan struct{} field to Hub
- Initialize stopChan in NewHub()
- Added Stop() method that closes stopChan
- Modified Run() main loop to select on stopChan
- On shutdown: close all client connections and return
- Modified runBroadcastSequencer() from 'for range' to select
- Changed from: for msg := range h.broadcastSeq
- Changed to: for { select { case msg := <-h.broadcastSeq: ... case <-h.stopChan: ... }}
- On shutdown: stop coalesce timer and return
Shutdown sequence:
1. Call hub.Stop() to close stopChan
2. Both Run() and runBroadcastSequencer() exit their loops
3. All client send channels are closed
4. Clients map is cleared
5. Pending coalesce timer is stopped
Impact:
- Enables graceful shutdown during service restarts
- Prevents goroutine leaks in tests
- Allows proper cleanup of WebSocket connections
- No more orphaned broadcast sequencer goroutines
This commit addresses 5 critical P0 bugs that cause security vulnerabilities, crashes, and data corruption:
**P0-1: Recovery Tokens Replay Attack Vulnerability** (recovery_tokens.go:153-159)
- **SECURITY CRITICAL**: Single-use recovery tokens could be replayed
- **Problem**: Lock upgrade race - two concurrent requests both pass initial Used check
1. Both acquire RLock, see token.Used = false
2. Both release RLock
3. Both acquire Lock and mark token.Used = true
4. Both return true - TOKEN REUSED
- **Impact**: Attacker with intercepted token can use it multiple times
- **Fix**: Re-check token.Used after acquiring write lock (TOCTOU prevention)
**P0-2: WebSocket Hub Concurrent Map Panic** (hub.go:345-347, 376-378)
- **Problem**: Initial state goroutine reads h.clients map without lock
- Line 345: `if _, ok := h.clients[client]` (NO LOCK)
- Main loop writes to h.clients with lock (line 326, 394)
- **Impact**: "fatal error: concurrent map read and write" crashes hub
- **Fix**: Acquire RLock before all client map reads in goroutine
**P0-3: WebSocket Send on Closed Channel Panic** (hub.go:348, 380)
- **Problem**: Check client exists, then send - channel can close between
- **Impact**: "send on closed channel" panic crashes hub
- **Fix**: Hold RLock during both check and send (defensive select already present)
**P0-4: CSRF Store Shutdown Data Corruption** (csrf_store.go:189-196)
- **Problem**: Stop() calls save() after signaling worker. Both hold only RLock
- Worker's final save writes to csrf_tokens.json.tmp
- Stop()'s save writes to same file concurrently
- **Impact**: Corrupted/truncated csrf_tokens.json on shutdown
- **Fix**: Added saveMu mutex to serialize all disk writes
**P0-5: CSRF Store Deadlock on Double-Stop** (csrf_store.go:103-108)
- **Problem**: stopChan unbuffered, no sync.Once guard, uses send not close
- **Impact**: Second Stop() call blocks forever waiting for receiver
- **Fix**:
- Added sync.Once field stopOnce
- Changed to close(stopChan) within stopOnce.Do()
- Prevents double-close panic and deadlock
All fixes maintain backwards compatibility. The recovery token fix is particularly critical as it closes a security vulnerability allowing replay attacks on password reset flows.
Replace string(rune(i)) with strconv.Itoa(i) in hub_concurrency_test.go
for generating client IDs. While this is test code and not a production bug,
it uses the same incorrect pattern that caused the PR #575 bug.
This ensures consistent best practices across the codebase and avoids
confusion for developers who might copy this pattern.
Related: #575
